search

SchedulesDirect / mfdb-json

mythfilldatabase grabber for the Schedules Direct JSON service
14 stars 3 forks source link

Second run of mfdb-json leads to several channels with NO DATA with API 20141201 #7

Closed yasij closed 9 years ago

yasij commented 9 years ago

I just upgraded to API 20141201 from 20140530, and I'm getting data loss on an incremental schedule update the next day. The initial import of schedule data filled out the schedule. However, when mfdb-json.php ran the next day, several channels are marked "NO DATA". This is reproducible. If I clear the schedulesdirect database and start over, the full schedule is loaded. The next day, several channels get marked as "NO DATA". This does not happen with API 20140530.

yasij commented 9 years ago

2015-05-14.debug.log: 2015-05-14 02:46:37:Processing /tmp/mfdb_programs_HV1yLo/programs.00.json 2015-05-14 02:46:41:Processing /tmp/mfdb_programs_HV1yLo/programs.01.json 2015-05-14 02:46:43:Processing /tmp/mfdb_programs_HV1yLo/programs.02.json 2015-05-14 02:46:46:Processing /tmp/mfdb_programs_HV1yLo/programs.03.json 2015-05-14 02:46:49:Processing /tmp/mfdb_programs_HV1yLo/programs.04.json 2015-05-14 02:46:52:Processing /tmp/mfdb_programs_HV1yLo/programs.05.json 2015-05-14 02:46:54:Processing /tmp/mfdb_programs_HV1yLo/programs.06.json 2015-05-14 02:46:57:Processing /tmp/mfdb_programs_HV1yLo/programs.07.json 2015-05-14 02:46:59:Processing /tmp/mfdb_programs_HV1yLo/programs.08.json

2015-05-15.debug.log: 2015-05-15 14:18:12:Processing /tmp/mfdb_programs_7a0r0D/programs.00.json

rkulagowski commented 9 years ago

OK - the mfdb-json for 20141201 is still getting the bugs worked out; most of my effort has been in stabilizing the server side.

Can you switch back to 20140530 in the interim until I have things ready?

yasij commented 9 years ago

Sure. I saw the announcement that 20140530 is EOL at the end of the month. Will this be delayed if the client isn't ready?

rkulagowski commented 9 years ago

mfdb-json will be done by then.

rkulagowski commented 9 years ago

I've updated some code; can you git pull and then let me know if you're still seeing this issue?

yasij commented 9 years ago

I will try it tonight.

yasij commented 9 years ago

Nope, the second run after there is an updated schedule still causes a ton of channels to show NO DATA.

rkulagowski commented 9 years ago

Is it possible to do something like a TeamViewer session so that I can get access to your machine and see what's happening? I can't reproduce it here, and with TeamViewer you'll be able to see what I'm doing on the host.

rkulagowski commented 9 years ago

This bug has been fixed.

Please do the following:

git pull mfdb-json.php --purge=schedules

(this will clear out the old / bad schedules - but only do it once!)

On your next run (after the above purge) you should see the large swaths of NO DATA should now be gone.

The last batch update on the server side was 2015-06-15@ ~ 12P CDT. If you can do the above reset "soon" then there will be another batch run in a few hours. Please update this ticket when you've done the above purge and then I will update the ticket once the next run has occurred after that so that you can confirm that the bug has been fixed.

yasij commented 9 years ago

Okay, purge complete.

rkulagowski commented 9 years ago

OK, the server batch process just finished.

Can you run the grabber again? Please make sure that you don't use the --purge function this time.

yasij commented 9 years ago

Looks like it's fixed. It did an incremental update of 27 programs, and there are no longer lots of holes with NO DATA. Thanks.

rkulagowski commented 9 years ago

Excellent. Thanks for being patient.