VPRO.nl source not working

[kyl416]

I'm getting no data with the recently added vpro.nl source.

[hikavdh]

Have you (with the first beta) recreated your config or ran it with --configure. Else the new chanids won't be in your config. Check that the channel lines contain 13 tags. Name, group, chanid, 8 sourceids and the icon sourceid and name. The last sourceid is for vpro.nl

[kyl416]

Yes I reran configure, but when I download I get errors like "No data on vpro.nl for day=0" "No data on vpro.nl for day=1" "No data on vpro.nl for day=2" "No data on vpro.nl for day=3" "No data on vpro.nl for day=4" "No Data from vpro.nl for channel: Sky Radio" "No Data from vpro.nl for channel: Radio 538" "No Data from vpro.nl for channel: NPO 1" "0 base page(s) fetched from vpro.nl" "5 failure(s) on vpro.nl"

And for channels that are only available from the vpro source have no listings in the final output

[hikavdh]

Weird, I just now see it coming in. Try in your browser: http://www.vpro.nl/epg-embeddable.html That's the link for today. For other days it is: http://www.vpro.nl/epg-embeddable/content/0.html?day=yyyy-mm-dd up to five days or so. Do you perchance have a not dutch IP address?

[kyl416]

I just did another run, this time I got more days.

[hikavdh]

OK, keep me informed. I'm off now, I check in later today. If you get it running I can advise you to set your preferred description to this source, they are good! I'm at present busy extracting all the extra data from there.

[hikavdh]

Se the latest probably final beta: https://github.com/tvgrabbers/tvgrabnlpy/releases/tag/beta-2.2.4-p20151106

[kyl416]

It's weird, on one try I got some pages from vpro.nl data, but the next try it's just failures. I see similar things with npo.nl where some pages fail:

3 base page(s) fetched from npo.nl 4 failure(s) on npo.nl 0 base page(s) fetched from vpro.nl 5 failure(s) on vpro.nl

Any chance there's a cookie warning problem?

[hikavdh]

Does the log say anything about why it fails, other then no data? I also get 1 or 2 failures on npo.nl every run, but that is normally corrected by redundancy (all failures are tried a second time on a second run). It could be that the path through your provider to them is not so good. So then you would see time-outs. It anyhow is logical that npo.nl and vpro.nl share the problem as I assume they are hosted in the same place! I test through ziggo and collect through kpn and they normally both work OK. I just did a ping test. npo.nl is blocking it, vpro.nl gives around 10,3 ms through ziggo and around 18,2 ms through kpn, 10 is normal, 18 is quite high..

Around line 433 in the script there is a line: self.global_timeout = 10. You can try raising it to 15 or even 20.

Do you see problems accessing the earlier mentioned pages in a normal browser?

I now suddenly realize I didn't add the redundancy to vpro.nl. I'll add it.

[kyl416]

I didn't have any problems accessing the pages through a browser. Right now it's in the weird overnight hours where NPO's gids are incomplete, so I'll try again in a few hours

The log doesn't show anything besides multiple entries for each day saying "No data on vpro.nl"

Looking at the ping/traceroute results, vpro.nl is using cloudvps as a host while NPO.nl is self hosted.

[hikavdh]

Yes I saw and what is your ping time?

[hikavdh]

Oh and while some sources are unpredictable between 12 and 4+, with npo and vpro I only saw a very short time of bad accessibility just after midnight..

[kyl416]

Mine is about 100ms for vpro.nl, although I'm in the USA, so a ping like that is normal for a server located across the Atlantic. I use tv_grab_nl.py to get listings for the channels that do online streaming like BVN, the regional omroep's and radio stations.

[hikavdh]

I earlier was thinking about geoblocking being the cause, but you say you can access the pages? I however know that npo does practice it. You should first try raising the timeout as your access time is ten times mine. Else you can, if you can arrange that, try working through a Europe based proxy. That would defeat any geoblocking.

[kyl416]

NPO's guides and websites load fine via a browser here too. The only geoblocking is with some of the livestreams.

Any chance you can add a configurable option to increase the timeout? It would be a lot easier than needing to patch the file each time a new update comes out.

[hikavdh]

I might, but not right now. I never did because you rarely need to fiddle with it. As long as you make sure it is saved in utf-8 and with unix linebreaks ... Wich only is a problem in a simple Windows editor. If in Windows use notepad+ https://notepad-plus-plus.org/ In Linux any text editor will do.

[kyl416]

The default 10 seconds timeout should be fine even with a 100ms ping, I haven't seen any timeout errors in the log. The pages load quickly, it's not like some really poor sites where pages fail to load or quit halfway through.

I wonder though, you have IE6 as the user agent, there's several sites that now render differently when they see that user agent. Maybe it's time to bump that up to IE11 with windows 7?

[hikavdh]

Could be that hasn't changed in 5 years. But that doesn't explain why only you see this problem. And the only really different thing is you being outside Europe. I know Belgium or Germany don't give problems. And the two things different because of that is your US ip-address and the access time.

[hikavdh]

Or maybe any blacklisting problem with your provider???

[kyl416]

As a test I just switched my browser's user-agent to IE6, I got a bunch of errors and VPRO's epg takes a lot longer to load. I do see a lot of "if lt" code on their site that checks for anything older than IE9

[hikavdh]

Oh and as I said I do get some 5 or 10% failures on npo.nl especially on some more busy hours. So the time-out is simple and easy to try. I also finished adding the retry cycle. Another small thing and I'll post it.

[hikavdh]

Ok, I'll look at it!

[hikavdh]

It though will take a little longer, as it's one of the few parts of the program I haven't really toughed, since I took over in January.

[hikavdh]

I'm thinking, maybe this does slowdown so much, that with your access-time the time-out gets reached. So until I change this, maybe raising the time-out temporarily solves the problem.

[hikavdh]

OK with some updated user agent strings and some other refinements. I haven't tested it jet, so for now only as tag: https://github.com/tvgrabbers/tvgrabnlpy/releases/tag/beta-2.2.4-p20151107

[kyl416]

This time 0 NPO failures, while vpro.nl worked on the retry

[hikavdh]

While I do not hear anything anymore I assume it now works!

[kyl416]

Yes, it's working now, thanks