Not sure which json feed file errored out

[subratom]

I have 80K records which are in several files (20 records/ file). I have created FeedMe url like this http://www.example.com/1.json

I have written a shell script which copies the file from a folder puts in the root of the site, lets FeedMe process the file and the moves that file to a different folder. The issue is sometimes I see in TaskManager plugin (tasks table in craft database), few files failed and I have no idea which failed because every file that is being processed is 1.json.

Is there a way to modify the logging so I know which file failed?

[engram-design]

It really depends on what the error being thrown is, but I suppose it could record the node that its trying to parse. Would that be helpful? Not being familiar with your setup, and whether they contain the same nodes, etc.

Are there any errors recorded in feedme.log, or phperrors.log?

[subratom]

I see there are about 5 feedme.log files in runtime folder. I opened up first file and the first line in that file is

<date/ time> [info] [plugin] [Forced] FeedMeError - .

This is followed by content blocks. I don't know if these content blocks are the ones that failed or not.

[engram-design]

I recon they are. Currently (depending on the error), it'll either provide the error message returned (entry couldn't be saved due to required missing fields, etc), or report the entry it failed on. This all depends on what duplication handling you're doing.

Whats one of the content blocks in the error for example?

[subratom]

[ { "ContentID": 110924, "Id": 73518, "Title": "Despite being technically impossible, underwater breathing ‘device’ receives $830,000 in crowdsourced funding", "CorrectedTitle": "Despite being technically impossible, underwater breathing ‘device’ receives $830,000 in crowdsourced funding", "Slug": "despite-being-technically-impossible-underwater-breathing-device-receives-830-000-in-crowdsourced-funding", "Parent": "", "Ancestors": "", "Author": "bauschje01", "Post Date": "2016-03-30 09:30:00", "Expiry Date": null, "Enabled": "Yes", "Status": "live", "Summary": "Despite mounting evidence that this device is not feasible, its online funding campaign is closing in on the one million-dollar mark", "Sub Title": "Gadget is being lauded by some, criticized by many", "Contents": "<p>An underwater breathing device, widely criticized by experts in both the scientific and technical communities, has just surpassed the $830,000 benchmark in contributions made via crowdsourcing site Indiegogo.<br></br><br></br></p><p><img src=\"http://www.electronicproducts.dev/assets/ektronimages/fajb_triton_01_mar2016.jpg\" alt=\"Triton\" /><figure contenteditable=\"false\" title=\"Triton\" data-ektron-image-src=\"/uploadedImages/Biotech/Wearables/fajb_triton_01_mar2016.jpg\" data-media=\"/uploadedImages/Biotech/Wearables/fajb_triton_01_mar2016.jpg?targetTypeId=SmartPhone\" data-media481=\"/uploadedImages/Biotech/Wearables/fajb_triton_01_mar2016.jpg\"><img src=\"http://www.electronicproducts.dev/assets/ektronimages/fajb_triton_01_mar2016.jpg\" alt=\"Triton\" /></figure> <br></br>Referred to as “Triton”, the mouth-based device is marketed as having artificial gills capable of providing its user with enough oxygen for up to 45 minutes of underwater swim time, at a maximum dive depth of 15 feet. It is able to accomplish this by way of a microporous filter that has tiny holes in it, which are small enough to stop water from getting in, but large enough to allow for the dissolved, “free-floating” oxygen to enter.<br></br><br></br>The oxygen is next compressed and transferred over to an on-board storage tank, a process powered by a supercharged lithium-ion battery. <br></br><br></br><img src=\"http://www.electronicproducts.dev/assets/ektronimages/fajb_triton_02_mar2016.jpg\" alt=\"Triton in use\" /><figure contenteditable=\"false\" title=\"Triton in use\" data-ektron-image-src=\"/uploadedImages/Biotech/Wearables/fajb_triton_02_mar2016.jpg\" data-media=\"/uploadedImages/Biotech/Wearables/fajb_triton_02_mar2016.jpg?targetTypeId=SmartPhone\" data-media481=\"/uploadedImages/Biotech/Wearables/fajb_triton_02_mar2016.jpg\"><img src=\"http://www.electronicproducts.dev/assets/ektronimages/fajb_triton_02_mar2016.jpg\" alt=\"Triton in use\" /></figure> <br></br>Now for the reality: this device, and all of its claims, is impossible for several reasons. For one, it’s small — Triton measures just 11.5 inches in length. If it’s capable of filtering out oxygen from the water — enough for a diver to not drown — it would have to be filtering approximately 24 gallons of water every minute. What’s more, it would need to be operating at 100% efficiency to avoid any risk of accidental choking and / or shortness of breath. <br></br><br></br>Additionally, this much water coming through the filter system is equivalent to, more or less, six garden hoses’ worth of pressure. To sustain this constant level of flow, the swimmer would have to be moving through the water at exceptionally fast speeds. <br></br><br></br>So, that’s just the filtering process. In terms of the actual technology, being able to compress oxygen requires a lot of energy. Andrew David Thaler, a deep-sea ecologist, spoke to Business Insider about it and explained that a battery capable of supporting this type of system would need to be so powerful, it isn’t actually available on the market yet. <br></br><br></br>Also, breathing in pure, pressurized oxygen is hazardous to one’s health — too much of it can cause severe damage to the user’s respiratory and central nervous systems.<br></br><br></br>There has been a theory floated out there that perhaps the device is actually electrolyzing the water to produce usable oxygen. This technology already exists, so it’s plausible — but only as a theory. You see, to support a system that electrolyzes that much water to produce enough oxygen to support a diver underwater for 45 minutes would require a ton of electrical energy — about a nuclear reactor’s worth (or so). <br></br><br></br>All of this is nothing new to the Triton crew — they’ve been hearing these criticisms for the last two years since the technology was first introduced. None of it has deterred them, though, as they went ahead and built a prototype and quickly put it on Indiegogo to see if they could win the public’s support. <br></br><br></br>And you know what? They got it, having already reached the team’s funding goal of $50,000, and in fact, already surpassing the $830,000 mark at the time this article’s being written. <br></br><br></br>That’s 2,200+ people who’ve gotten behind this idea, all scientific and technical evidence aside. <br></br><br></br></p><p> </p><p>To check out the campaign for yourself, head to the <a href=\"https://www.indiegogo.com/projects/triton-world-s-first-artificial-gills-re-breather#/\" title=\"Triton page\">Triton page</a> .</p><p>Via <a href=\"http://www.iflscience.com/technology/artificial-gills-underwater-breathing-device-has-820000-funding-despite-being\" title=\"iflscience.com\">iflscience.com</a> </p>", "Part Numbers": "", "Print": "0", "Print Issue Month Year": "03/2016", "Meta Tags": "biotechnology-research,triton-underwater-breathing-device", "ArticleType": "Featured Articles", "Primary Article Classification": "Wearables", "Companies": "Electronic Products Magazine", "Primary End Markets": "Science/ Research", "Image Thumbnail": "fajb triton 02 mar2016", "HotTopic": "0", "DownloadList": [ { "DownloadType": "", "DownloadValue": "" } ] }

This is one from feedme.log file

[engram-design]

Although that shows the node that an error was thrown on, its quite odd that I can't find the code that outputs that. I wonder if it was generated by an older version of Feed Me?

[subratom]

What version of FeedMe would that be? Currently I am using 1.4.11. I still haven't updated it to the latest one

[engram-design]

Yeah, thats pretty much the latest. Anyway - maybe you could add an item to your feed thats a unique identifier (number, date, etc) that you could map to keep track of what file its processing on?

[subratom]

I could add a field, just file number would do fine as all the files are unique anyway.

[engram-design]

Closing for the moment due to inactivity. Re-open if you have more issues.