kgroeneveld / tv_grab_sd_json

XMLTV grabber for the Schedules Direct JSON API
8 stars 6 forks source link

Not a HASH reference at tv_grab_sd_json line 527. #3

Open kyl416 opened 9 years ago

kyl416 commented 9 years ago

Whenever I try to download data from one of the generic satellite lineups for Europe, I get the following:

Updating schedules... \ POST https://json.schedulesdirect.org/20141201/schedules/md5 ==> 200 OK (4s) Not a HASH reference at ./tv_grab_sd_json line 52

To reproduce it, add ZZZ-19.2E-DEFAULT as a lineup, which is listed for any German postal code.

kyl416 commented 9 years ago

Small update, it works if I manually change the config file to channels mode and manually add channels, but it fails when I do lineups mode to get all the channels. I found another lineup it occurs with: DEU-0001050-X.

kgroeneveld commented 9 years ago

Thanks for the bug report. I just pushed a change which seems to fix the issue for the ZZZ-19.2E-DEFAULT lineup. I couldn't seem to add the DEU-0001050-X lineup to test with. Hopefully the same fix works for both.

kyl416 commented 8 years ago

I got further this time, but now there's another error once it gets to programs: Updating schedules... * POST https://json.schedulesdirect.org/20141201/schedules/md5 ==> 200 OK (2s) Empty/invalid schedule for channel 74767 * POST https://json.schedulesdirect.org/20141201/schedules ==> 200 OK (12s) Updating programs... \ POST https://json.schedulesdirect.org/20141201/programs ==> 200 OK (3s) Removing old cache entires... Not a HASH reference at ./tv_grab_sd_json line 643

kgroeneveld commented 8 years ago

I am not seeing that same error. I suspect there is unexpected data in your cache file either from a lineup I have never tested before or it somehow became corrupted. Perhaps you could send me a copy of your cache file (default location is $HOME/.xmltv/tv_grab_sd_json.cache)?

I pushed another fix to check for and skip unexpected data near line 643.

If it still does not work for you then you could try deleting the cache file and running it again. Perhaps you could also remove all other lineups from your account except for ZZZ-19.2E-DEFAULT. You may be able to narrow down which lineup is triggering the condition.

Unfortunately I think there are probably many other possible spots in the code that could abort on unexpected data. I really need to go through the whole thing and add better handling for this.

kyl416 commented 8 years ago

The cache file is too big to post. I had another lineup there that I deleted and it worked. I readded the problem lineup afterwards (ZZZ-28.2E-DEFAULT) and now it stops here: "Not a HASH reference at tv_grab_sd_json line 675"

I don't remember which region I found that lineup since I added it a while back, but you can manually add it using curl after you fecth a token: curl https://json.schedulesdirect.org/20141201/lineups/ZZZ-28.2E-DEFAULT -H "token: xxx" -X PUT

kyl416 commented 8 years ago

Some more details that might help you figure it out, the ZZZ-28.2E-DEFAULT lineup doesn't have channel numbers, just satellite coordinates, I put it in channels mode and manually added channels and it works that way, however the following appears: "Invalid channel in lineup HASH(0x4266dd0)" This is what the map and stations section looks like for one of the channels I tested with in the raw json file:

{
    "stationID": "77777",
    "frequencyHz": ​11343000000,
    "polarization": "V",
    "deliverySystem": "DVB-S",
    "modulationSystem": "QPSK",
    "symbolrate": ​27500,
    "fec": "2/3",
    "serviceID": ​55209,
    "networkID": ​2,
    "transportID": ​2304
}
{
    "stationID": "77777",
    "name": "Rishtey",
    "callsign": "RISHTEY",
    "broadcastLanguage": 
[
    "en-GB"
],
"descriptionLanguage": 
    [
        "en-GB"
    ]
}

this is what one of the outputs look like in the final xmltv file for the channel:

  <channel id="I77777.labs.zap2it.com">
    <display-name>Rishtey</display-name>
    <display-name>RISHTEY</display-name>
    <display-name>unknown number</display-name>
  </channel>
kgroeneveld commented 8 years ago

I just pushed up a few more changes which includes more validation of data and error handling. There is a lot more error handling that should still be added but the current version now seems to work okay with the ZZZ-28.2E-DEFAULT lineup. The returned JSON for this lineup seems to include some invalid channel data and some empty schedules which was causing problems.

I also added a change which allows you to add lineups such as ZZZ-28.2E-DEFAULT without having to do the country/postal code search.