USA-MA20483-X getting empty schedules for several xmltvids

[karog]

Over the last 24 hours, I have been getting empty /schedules data for multiple xmltvids. By empty I mean 0 length string. Last night it was only 60947 which is working today but today 14766, 59303, 58625, 59296, 58574, 57581, 34941, 21868, and 64492 are all returning 0 length strings for /schedules.

Here is what /schedules/md5 shows for one of them: { "64492": [ { "days": 13, "md5": "", "modified": "" } ] }

The same happens for days: 4.

I would appreciate this being investigated.

[rkulagowski]

I think there was a glitch; when I look at 64492 on the server I can see that it's got a valid schedule. Can you retry and let me know if you're still seeing issues with empty schedules?

[karog]

Well it seems to have repaired itself. All of my channels are now working. I didn't report it right after the single instance yesterday. I wanted to wait to see if it was a glitch. And to my surprise today the one channel was ok but the long list in my prior message were bad in the same way. I provided the one instance of the /schedules/md5 response also clearly wrong. Apparently between the time I wrote and the time you inspected, things cleared up.

Thanks for checking and responding. If it turns up again, I will note it here.

For the most part, things have been remarkably good. My grabber updates 72 channels with a new day of data (I currently don't do credits) in about 60-90 seconds plus about 30-45 seconds just to clear and then reset firsts and lasts. If I run it when there are no new updates, it completes in about 6 seconds. And this is on very underpowered pogoplugs in javascript within nodejs.

[rkulagowski]

Excellent, glad it got cleared up. The next version of the API (under development now) will give per-day granularity for the schedules, so each day on each stationID from "day 0" to "day 13" will have its own MD5, which will allow clients to only pull the schedule for "day 8" if that's the only one that's changed.

[karog]

Good to know. I want to rewrite my grabber as I was in learning mode when I wrote it. It took a bit to learn to use Promises for async I/O calls effectively. I will wait until the new API is available. Any rough estimate as to when?

[karog]

I am running into this problem again, 16 channels failing trying several times over a 45 minutes period at 2015-01-18 23:07:43 with Last data update: 2015-01-18 15:41:43. Failing xmltvids:

14886, 14765, 17665, 14988, 16833, 19634, 19596, 20362, 32633, 56905, 58574, 60150, 45507, 57394, 34941, 66379

[rkulagowski]

OK, I can see that the issue is upstream data corruption. Let me investigate.

[karog]

Well I am glad you can see the problem. I am up to 36 bad channels now.

I hope the problem is not too serious and fixable.

[rkulagowski]

Can you see if a refresh gets you data again?

[karog]

Per your request, I just updated again and all channels loaded without error with14 days of data.

Thanks.

Was this a newly introduced problem or something that has been lurking and only now manifest?

[rkulagowski]

It was always there; I didn't have code to properly handle an upstream issue which could have happened, but never had.

[rkulagowski]

The code should now properly handle between 14 and 30 days of upstream data in API 20141201; closing.