Closed qdacsvx closed 5 years ago
honir
commented
5 years ago Thanks for the info.
So there's two issues in there:
I'll have to remind myself how the caching in XMLTV works...
honir
commented
5 years ago I've fixed the warning message on Unable to retrieve web page for 381
It's not practical to do anything about the 'broken' page being cached. Pages are only cached for 1 hour so if you re-try after that period then the cache will no longer be used.
qdacsvx
commented
5 years ago Thanks for updating. If I understand correctly, a warning will cause the grabber to return 1. That's sufficient for my purposes. It would be helpful to have a more informative warning message like "Unable to get programme data for [channel]."
honir
commented
5 years ago Yes, the grabber will return exit code 1 on warnings.
I can't really see any difference between Unable to retrieve web page for 381 and Unable to get programme data for 381
This error should be infrequent as it essentially means there's either an error in the format of the URL being requested (e.g. invalid station or invalid date), or there's a server error at TVGuide. Both of these should be exceptional.
honir
commented
5 years ago p.s. If you want to display a more meaningful channel name than "381" see the "man tv_grab_uk_tvguide" page for how to use the .map.conf file
Thanks for taking the time to report an issue. Please take a moment to review our open/closed issues above, in case your issue has already been reported.
If you are reporting a new issue, please give your issue a descriptive title and fill out the blanks below, providing as much information as possible.
XMLTV Version?
… git
XMLTV Component?
… $ tv_grab_uk_tvguide --version XMLTV module version 0.6.1 This is tv_grab_uk_tvguide version 0.6.1
What happened?
While doing grabs (around 100 channels) 1-2 random channels will fail to download all their programmes. The grabber prints a warning "Unable to retrieve web page for [channel id]" for each failing channel but still may output an incomplete XML. Rerunning the grabber doesn't download any of the missing programmes. In a multi-day grab e.g. --days 2, sometimes one day downloads but not the second.
What did you expect to happen?
If some of the programmes for a channel can't be downloaded, the grabber should halt with an error. The grabber should report if there are warnings "Exiting without warnings." and return an error code if the warning is serious. Losing programmes is serious.
Did you see any warnings/errors?
(Please paste any warnings/errors, if available)
Unable to retrieve web page for 381
What steps are needed to reproduce this issue?
Grab 100+ channels and search for the warning. Log of grabbing the channel with the warning. The previous day (12-6AM for this grabber) did not download any programmes.
93.conf.gz
tv_grab_uk_tvguide --quiet --debug --config-file conf/93.conf --days 1 --output xml/381 && mv xml/381 X381 Fetching https://www.tvguide.co.uk/channellisting.asp?ch=381&cTime=07%2F08%2F2019%2000%3A00%3A00 from server. Fetching https://www.tvguide.co.uk/channellistings.asp?ch=381&cTime=07/08/2019%2000:00:00&thisTime=&thisDay= from server. Unable to retrieve web page for 381 Fetching https://www.tvguide.co.uk/channellisting.asp?ch=381&cTime=07%2F09%2F2019%2000%3A00%3A00 from server. Fetching https://www.tvguide.co.uk/channellistings.asp?ch=381&cTime=07/09/2019%2000:00:00&thisTime=&thisDay= from server. Fetching https://www.tvguide.co.uk/detail/2831277/8812747/counting-on from server. Fetching https://www.tvguide.co.uk/detail/1563465/8812753/dogs-101 from server. Fetching https://www.tvguide.co.uk/detail/1948337/8812732/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2504166/8812752/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/2510687/8812740/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/3080423/8812741/unexpected from server. Fetching https://www.tvguide.co.uk/detail/3086202/8812744/unexpected from server. Fetching https://www.tvguide.co.uk/detail/2517822/8812734/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/2528369/8812748/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/2923738/8812743/counting-on from server. Fetching https://www.tvguide.co.uk/detail/1565339/8812746/dogs-101 from server. Fetching https://www.tvguide.co.uk/detail/1948537/8812742/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2062672/8812745/my-five-wives from server. Fetching https://www.tvguide.co.uk/detail/2841681/8812735/my-giant-life-haleighs-journey-so-far from server. Fetching https://www.tvguide.co.uk/detail/3022345/8812754/lost-in-transition from server. Fetching https://www.tvguide.co.uk/detail/2859619/8812736/untold-stories-of-the-er from server. Fetching https://www.tvguide.co.uk/detail/1474344/8812733/monsters-inside-me from server. Fetching https://www.tvguide.co.uk/detail/2471785/8812738/my-600lb-life-where-are-they-now from server. Fetching https://www.tvguide.co.uk/detail/2062672/8827884/my-five-wives from server. Fetching https://www.tvguide.co.uk/detail/2841681/8827882/my-giant-life-haleighs-journey-so-far from server. Fetching https://www.tvguide.co.uk/detail/1563465/8827896/dogs-101 from server. Fetching https://www.tvguide.co.uk/detail/1948337/8827895/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2517822/8827885/playhouse-masters from server. Exiting without warnings.
X381.gz
$ tv_to_text X381 07-09 (Tuesday)
06:00--08:00 Counting On // Joy and Austin Tie the Knot Discovery Home & Health 08:00--09:00 Dogs 101 // Keeshond/Borzoi/Dingo Discovery Home & Health 09:00--10:00 Toddlers & Tiaras // Cheetalicious Pageant Discovery Home & Health 10:00--11:00 Playhouse Masters // Playhouse to the Stars Discovery Home & Health 11:00--12:00 Playhouse Masters // Ultimate Birdhouse Treehouse Discovery Home & Health 12:00--13:00 Unexpected // Step Up and Be a Dad Discovery Home & Health 13:00--14:00 Unexpected // When the Water Breaks Discovery Home & Health 14:00--15:00 Playhouse Masters // Lake Cabin Playhouse Discovery Home & Health 15:00--16:00 Playhouse Masters // Ghost Train Playhouse Discovery Home & Health 16:00--17:00 Counting On // In Love In Switzerland Discovery Home & Health 17:00--18:00 Dogs 101 // Bluetick Coonhound/Petit Bassett Griffon Discovery Home & Health 18:00--19:00 Toddlers & Tiaras // Carolina Queens Pageant Discovery Home & Health 19:00--20:00 My Five Wives // The Big Decision Discovery Home & Health 20:00--21:00 My Giant Life: Haleigh's Journey So Far Discovery Home & Health 21:00--22:00 Lost in Transition Discovery Home & Health 22:00--23:00 Untold Stories of the ER // Raging Reunion Discovery Home & Health 23:00--00:00 Monsters Inside Me // Shape Shifters Discovery Home & Health
Generated from TV Guide UK by tv_grab_uk_tvguide.
Log of regrabbing. Note that the warning persists; the grabber still cannot download any programmes for the previous day (12-6AM).
$tv_grab_uk_tvguide --quiet --debug --config-file conf/93.conf --days 1 --output xml/381 && mv xml/381 X381 Fetching https://www.tvguide.co.uk/channellisting.asp?ch=381&cTime=07%2F08%2F2019%2000%3A00%3A00 from server. Fetching https://www.tvguide.co.uk/channellistings.asp?ch=381&cTime=07/08/2019%2000:00:00&thisTime=&thisDay= from cache without checking with server. Unable to retrieve web page for 381 Fetching https://www.tvguide.co.uk/channellisting.asp?ch=381&cTime=07%2F09%2F2019%2000%3A00%3A00 from server. Fetching https://www.tvguide.co.uk/channellistings.asp?ch=381&cTime=07/09/2019%2000:00:00&thisTime=&thisDay= from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2831277/8812747/counting-on from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1563465/8812753/dogs-101 from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1948337/8812732/toddlers-tiaras from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2504166/8812752/playhouse-masters from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2510687/8812740/playhouse-masters from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/3080423/8812741/unexpected from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/3086202/8812744/unexpected from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2517822/8812734/playhouse-masters from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2528369/8812748/playhouse-masters from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2923738/8812743/counting-on from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1565339/8812746/dogs-101 from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1948537/8812742/toddlers-tiaras from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2062672/8812745/my-five-wives from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2841681/8812735/my-giant-life-haleighs-journey-so-far from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/3022345/8812754/lost-in-transition from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2859619/8812736/untold-stories-of-the-er from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1474344/8812733/monsters-inside-me from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2471785/8812738/my-600lb-life-where-are-they-now from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2062672/8827884/my-five-wives from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2841681/8827882/my-giant-life-haleighs-journey-so-far from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1563465/8827896/dogs-101 from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/1948337/8827895/toddlers-tiaras from cache without checking with server. Fetching https://www.tvguide.co.uk/detail/2517822/8827885/playhouse-masters from cache without checking with server. Exiting without warnings.
$ tv_to_text X381 07-09 (Tuesday)
06:00--08:00 Counting On // Joy and Austin Tie the Knot Discovery Home & Health 08:00--09:00 Dogs 101 // Keeshond/Borzoi/Dingo Discovery Home & Health 09:00--10:00 Toddlers & Tiaras // Cheetalicious Pageant Discovery Home & Health 10:00--11:00 Playhouse Masters // Playhouse to the Stars Discovery Home & Health 11:00--12:00 Playhouse Masters // Ultimate Birdhouse Treehouse Discovery Home & Health 12:00--13:00 Unexpected // Step Up and Be a Dad Discovery Home & Health 13:00--14:00 Unexpected // When the Water Breaks Discovery Home & Health 14:00--15:00 Playhouse Masters // Lake Cabin Playhouse Discovery Home & Health 15:00--16:00 Playhouse Masters // Ghost Train Playhouse Discovery Home & Health 16:00--17:00 Counting On // In Love In Switzerland Discovery Home & Health 17:00--18:00 Dogs 101 // Bluetick Coonhound/Petit Bassett Griffon Discovery Home & Health 18:00--19:00 Toddlers & Tiaras // Carolina Queens Pageant Discovery Home & Health 19:00--20:00 My Five Wives // The Big Decision Discovery Home & Health 20:00--21:00 My Giant Life: Haleigh's Journey So Far Discovery Home & Health 21:00--22:00 Lost in Transition Discovery Home & Health 22:00--23:00 Untold Stories of the ER // Raging Reunion Discovery Home & Health 23:00--00:00 Monsters Inside Me // Shape Shifters Discovery Home & Health
Generated from TV Guide UK by tv_grab_uk_tvguide.
Any other information?
(For example, is this a new or intermittent issue?) Usually a few random channels out of 100 fail to download.
Regrabbing fails as above if the cache is still present. If the cache is manually (or automatically) deleted, regrabing succeeds. cache381.tar.gz cache381-new.tar.gz
Successful regrabbing after manually deleting the cache:
cache381complete.tar.gz
tv_grab_uk_tvguide --quiet --debug --config-file conf/93.conf --days 1 --output xml/381 && mv xml/381 X381 Fetching https://www.tvguide.co.uk/channellisting.asp?ch=381&cTime=07%2F08%2F2019%2000%3A00%3A00 from server. Fetching https://www.tvguide.co.uk/channellistings.asp?ch=381&cTime=07/08/2019%2000:00:00&thisTime=&thisDay= from server. Fetching https://www.tvguide.co.uk/detail/1931038/8723892/my-600lb-life from server. Fetching https://www.tvguide.co.uk/detail/2062671/8812737/my-five-wives from server. Fetching https://www.tvguide.co.uk/detail/2841668/8812755/my-giant-life-cocos-journey-so-far from server. Fetching https://www.tvguide.co.uk/detail/1622946/8812750/extreme-couponing from server. Fetching https://www.tvguide.co.uk/detail/1622947/8812739/extreme-couponing from server. Fetching https://www.tvguide.co.uk/detail/1946370/8812749/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2504166/8812751/playhouse-masters from server. Fetching https://www.tvguide.co.uk/channellisting.asp?ch=381&cTime=07%2F09%2F2019%2000%3A00%3A00 from server. Fetching https://www.tvguide.co.uk/channellistings.asp?ch=381&cTime=07/09/2019%2000:00:00&thisTime=&thisDay= from server. Fetching https://www.tvguide.co.uk/detail/2831277/8812747/counting-on from server. Fetching https://www.tvguide.co.uk/detail/1563465/8812753/dogs-101 from server. Fetching https://www.tvguide.co.uk/detail/1948337/8812732/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2504166/8812752/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/2510687/8812740/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/3080423/8812741/unexpected from server. Fetching https://www.tvguide.co.uk/detail/3086202/8812744/unexpected from server. Fetching https://www.tvguide.co.uk/detail/2517822/8812734/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/2528369/8812748/playhouse-masters from server. Fetching https://www.tvguide.co.uk/detail/2923738/8812743/counting-on from server. Fetching https://www.tvguide.co.uk/detail/1565339/8812746/dogs-101 from server. Fetching https://www.tvguide.co.uk/detail/1948537/8812742/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2062672/8812745/my-five-wives from server. Fetching https://www.tvguide.co.uk/detail/2841681/8812735/my-giant-life-haleighs-journey-so-far from server. Fetching https://www.tvguide.co.uk/detail/3022345/8812754/lost-in-transition from server. Fetching https://www.tvguide.co.uk/detail/2859619/8812736/untold-stories-of-the-er from server. Fetching https://www.tvguide.co.uk/detail/1474344/8812733/monsters-inside-me from server. Fetching https://www.tvguide.co.uk/detail/2471785/8812738/my-600lb-life-where-are-they-now from server. Fetching https://www.tvguide.co.uk/detail/2062672/8827884/my-five-wives from server. Fetching https://www.tvguide.co.uk/detail/2841681/8827882/my-giant-life-haleighs-journey-so-far from server. Fetching https://www.tvguide.co.uk/detail/1563465/8827896/dogs-101 from server. Fetching https://www.tvguide.co.uk/detail/1948337/8827895/toddlers-tiaras from server. Fetching https://www.tvguide.co.uk/detail/2517822/8827885/playhouse-masters from server. Exiting without warnings.
X381complete.gz
$tv_to_text X381
07-09 (Tuesday)
00:00--01:00 My 600lb Life // Tara's Story Discovery Home & Health 01:00--02:00 My Five Wives // Mending a Marriage Discovery Home & Health 02:00--03:00 My Giant Life: Coco's Journey So Far Discovery Home & Health 03:00--03:30 Extreme Couponing // Callie & Kelly Discovery Home & Health 03:30--04:00 Extreme Couponing // Perry & Melissa Discovery Home & Health 04:00--05:00 Toddlers & Tiaras // California Tropic: Under the Sea Pageant Discovery Home & Health 05:00--06:00 Playhouse Masters // Playhouse to the Stars Discovery Home & Health 06:00--08:00 Counting On // Joy and Austin Tie the Knot Discovery Home & Health 08:00--09:00 Dogs 101 // Keeshond/Borzoi/Dingo Discovery Home & Health 09:00--10:00 Toddlers & Tiaras // Cheetalicious Pageant Discovery Home & Health 10:00--11:00 Playhouse Masters // Playhouse to the Stars Discovery Home & Health 11:00--12:00 Playhouse Masters // Ultimate Birdhouse Treehouse Discovery Home & Health 12:00--13:00 Unexpected // Step Up and Be a Dad Discovery Home & Health 13:00--14:00 Unexpected // When the Water Breaks Discovery Home & Health 14:00--15:00 Playhouse Masters // Lake Cabin Playhouse Discovery Home & Health 15:00--16:00 Playhouse Masters // Ghost Train Playhouse Discovery Home & Health 16:00--17:00 Counting On // In Love In Switzerland Discovery Home & Health 17:00--18:00 Dogs 101 // Bluetick Coonhound/Petit Bassett Griffon Discovery Home & Health 18:00--19:00 Toddlers & Tiaras // Carolina Queens Pageant Discovery Home & Health 19:00--20:00 My Five Wives // The Big Decision Discovery Home & Health 20:00--21:00 My Giant Life: Haleigh's Journey So Far Discovery Home & Health 21:00--22:00 Lost in Transition Discovery Home & Health 22:00--23:00 Untold Stories of the ER // Raging Reunion Discovery Home & Health 23:00--00:00 Monsters Inside Me // Shape Shifters Discovery Home & Health
Generated from TV Guide UK by tv_grab_uk_tvguide.
What other software are you using?
Operating System: $ uname -a Linux mx1 4.19.0-5-amd64 #1 SMP Debian 4.19.37-2~mx17+1 (2019-05-15) x86_64 GNU/Linux
Perl Version: $ perl --version
This is perl 5, version 24, subversion 1 (v5.24.1) built for x86_64-linux-gnu-thread-multi (with 85 registered patches, see perl -V for more detail)