tv_grab_uk_tvguide Cannot fetch channels

[benwallis]

XMLTV Version?

0.6.1

XMLTV Component?

tv_grab_uk_tvguide version 0.6.1

What happened?

Failed to fetch channels during configuration.

What did you expect to happen?

Fetch channels and configure.

Did you see any warnings/errors?

[Could not fetch http://my.tvguide.co.uk/channellisting.asp?ch=86&cTime=, error: 500 Can't connect to my.tvguide.co.uk:443, aborting] Fetching channels: 0%

What steps are needed to reproduce this issue?

1. xmltv tv_grab_uk_tvguide --configure
2. (enter) to set default cache directory

Any other information?

First time trying to use xmltv 0.6.1

What other software are you using?

Windows 10 x64 Home This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-msys-thread-multi

[honir]

tv_grab_uk_tvguide is now broken in that version of the Windows executable.

Please try the latest version : http://xmltv.org/alpha/xmltv.exe

[benwallis]

Thanks,

The URL changed but still getting the same error:

could not fetch https://www.tvguide.co.uk/channellisting.asp?ch=74&cTime=, error: 500 Can't connect to www.tvguide.co.uk:443, aborting

[honir]

Have you tried that url in a browser? (hint: it works for me)

Check your firewall (and router) is allowing the xmltv program access to the internet.

[benwallis]

I allowed the xmltv.exe in windows defender firewall but no difference.

I can get to the URL in the browser.

And I can get the HTML from that URL using cURL in PHP (run locally). But only if I disable SSL certificate checking. Otherwise curl returns this error:

SSL certificate problem: unable to get local issuer certificate

Could that be related?

I'm on Windows build 1903.

EDIT: I can fetch the https page via PHP curl after downloading the latest cacert.pem from curl

[rmeden]

Many SSL certs are having a problem right now.

Comodo a major inexpensive SSL provider had an intermediary certificate expire 2 days ago.

Schedulesdirect.org and PercData.com were both affected.  It was strange... Firefox didn't complain, but CURL did. (I've also seen reports that IE complained).

It may take a while for sites to update their CA certs.

Robert

[garybuhrmaster]

had an intermediary certificate expire 2 days ago.

Actually, it was the root cert expire which had signed the intermediary cert. The intermediary has been signed by two different root certs for some time now to support the transition.

It was strange... Firefox didn't complain, but CURL did. (I've also seen reports that IE complained). It may take a while for sites to update their CA certs.

The expiration of the (legacy) root cert was well known about, and due to cross-trust of the intermediary should not cause problems with recent versions of applications that had integrated the alternative signing root cert into their cert store and properly handled the alternative cert chain (from the intermediary).

Unfortunately some older apps may not have updated their cert store, or improperly handled the alternative chain (or users have not been updating to newer versions of the software). Note that Firefox ships its own set of trusted root certs (for good/bad/ugly reasons), so its working differently is not totally unexpected.

[rmeden]

On 5/31/2020 4:31 PM, Gary Buhrmaster wrote:

Unfortunately some older apps may not have updated their cert store, or improperly handled the alternative chain (or users have not been updating to newer versions of the software).

The SchedulesDirect data server that failed is running AWS 2018-03 release with pretty recent patches.

It uses LWP::UserAgent supplied by OS. It also has Curl 7.61.1 which also failed.

So, even if your OS was current, things will still have failed... .including Many/Most XMLTV packages   So this was a lot more wide spread than "some older apps".  I'd be surprised if XMLTV.EXE didn't fail.

Robert

[garybuhrmaster]

It also has Curl 7.61.1 which also failed.

Curl 7.61.1 is ancient (in Internet terms). It has 10 documented CVEs against it. Any system still running it should have been updated to something more recent long ago. It is also possible that your curl is linked against an openssl < 1.1.1 (also ancient) which does not properly traverse the cert chain (openssl 1.1.1 was released in 2018, plenty of time for updates for any distribution which cares about security).

[knowledgejunkie]

And I can get the HTML from that URL using cURL in PHP (run locally). But only if I disable SSL certificate checking. Otherwise curl returns this error:

SSL certificate problem: unable to get local issuer certificate

@benwallis Are you still experiencing problems retrieving listings?

[benwallis]

Yes, still same error message

[knowledgejunkie]

Wondering how to proceed with this issue.

The grabber itself seems to be fine; the issue seems to be a result of the system's SSL/Perl installation.

[garybuhrmaster]

Wondering how to proceed with this issue.

The grabber itself seems to be fine; the issue seems to be a result of the system's SSL/Perl installation.

As I recall, the xmltv.exe on Windows includes it own perl interpreter. And I recall the latest strawberry perl has been updated with a new(er) openssl. It is possible that a full rebuild using the latest perl and libraries would resolve this issue (note that a 64-bit build would probably also indirectly resolve #63), but that would require someone with a windows system to build a new exe to test with.

[thedave42]

I get this error with the latest builds of the exe. I am also willing to help build a new exe on a Windows machine but, ActiveState EOL'd the perl dev kit that contains the perlapp application used to build the xmltv.exe. I can't get ahold of the dev kit anymore.

However, I did use Strawberry perl to generate the Makefile, and I was able to use gmake to build, and install the app. I am able to use the batch files that are installed as part of the install process to successfully call this grabber using Windows 10 with all the latest security updates on it, and I no longer get this error when I use this method.

[rmeden]

I've normally been the one to build the EXE, but I had to update my machine and can't get a clean install working for all programs, and can't even get my old VM used to make the buidls work! We really need a new EXE solution :(

[simonrb2000]

I get the same error on Linux

[RealMurphy]

I get the same error on Linux

sorry for probably only adding noise, on Linux (Debian testing with 0.6.3) I get no error at all:

tv_grab_uk_tvguide --configure    
tv_grab_uk_tvguide uses a cache with files that it has already downloaded. Please specify where the cache shall be stored.
Directory to store the cache in: [$HOME/.xmltv/cache] 
Reformatting: 100% [============================================]
Select the channels that you want to receive data for.
TVC News [yes,no,all,none (default=no)] all
Sony MAX yes
Vice yes
[...]
$  tv_grab_uk_tvguide --list-channels|wc   
Reformatting: 100% [============================================]   2997    5789  120596

thus at least on Linux, upgrading to 0.6.3 ought to help.

[simonrb2000]

thus at least on Linux, upgrading to 0.6.3 ought to help.

Maybe I’m doing something wrong. This is what I get:-

pi@raspberrypi:~ tv_grab_uk_tvguide --configure tv_grab_uk_tvguide uses a cache with files that it has already downloaded. Please specify where the cache shall be stored. Directory to store the cache in: [/home/pi/.xmltv/cache] Fetching channels: 0% [ ]Can't call method "look_down" on an undefined value at /usr/bin/tv_grab_uk_tvguide line 758. pi@raspberrypi:~$

[mkbloke]

@simonrb2000, are you definitely using version 0.6.3 of tv_grab_uk_tvguide? As far as I can see, that call to look_down doesn't happen in the 0.6.3 version until line 766.

[simonrb2000]

I’ll check, I only just installed it so I’d assume it’s the latest version.

[mkbloke]

Unfortunately, dependent on how/from where you installed, you often cannot assume that. :-(

If you're using Raspberry Pi OS with the Debian 10/stable/Buster main repo, then it would appear the most recent version is 0.6.1.

A quick fix, if you don't want to get into pulling from elsewhere via apt pinning or something else is to grab the most recent version of tv_grab_uk_tvguide from master on here and use that version directly with the 0.6.1 back end. In theory, if the devs here on the xmltv project are following semantic version guidelines, that shouldn't break until a major version change, or at least, I think that's the case anyway! I am doing exactly that right now and it's been working fine. I just check here for new versions of the grabber in master when something breaks or just occasionally anyway and replace my local copy with the updated one every now and then.

[simonrb2000]

Unfortunately, dependent on how/from where you installed, you often cannot assume that. :-(

If you're using Raspberry Pi OS with the Debian 10/stable/Buster main repo, then it would appear the most recent version is 0.6.1.

A quick fix, if you don't want to get into pulling from elsewhere via apt pinning or something else is to grab the most recent version of tv_grab_uk_tvguide from master on here and use that version directly with the 0.6.1 back end. In theory, if the devs here on the xmltv project are following semantic version guidelines, that shouldn't break until a major version change, or at least, I think that's the case anyway! I am doing exactly that right now and it's been working fine. I just check here for new versions of the grabber in master when something breaks or just occasionally anyway and replace my local copy with the updated one every now and then.

You are correct, I’m using Version 0.61. I’ll have to figure out how to proceed with what you suggested :) Thank you

[simonrb2000]

All working on 0.6.3; Thank you

[knowledgejunkie]

@rmeden Given the "PDK gone EOL" news from Active State that @thedave42 linked to above, and your response about building the exes, do we have a plan going forward for Windows builds? Are they still possible at all?

In their PDK EOL blog post Active State recommend providing the bare Perl code sans wrapper. With Windows now being able to run a Linux shell, and/or Perl being able to be installed locally with users installing dependencies, is this the best route now?

[rmeden]

@knowledgejunkie I don't have a solution. :( I had to replace my laptop and can't get the old VM with PDK to work. I've tried setting it up on a new computer and so far no luck there getting all the dependanices to compile, much less build an easily distributable package.

[rmeden]

I just took a look at my Strawberry test environment... looks like everything works except the TK packages for my program, tv_check :) I guess that's pretty good for pretty much all users. I've been playing with PAR::Packer and it looks like I can make it work. I'll probably have a new xmltv.exe this weekend!

[knowledgejunkie]

@rmeden This looks promising!

[rmeden]

I posted on xmltv-devel mailing list.. the new xmltv.exe has been pushed http://alpha-exe.xmltv.org give it a shot and let me know.

[knowledgejunkie]

@benwallis Have you been able to test with our latest XMLTV builds for Windows? Can we close this issue?

[rmeden]

FYI... I updated the alpha-exe last night http://alpha-exe.xmltv.org

[honir]

Closed - stale thread. If the problem with SSL certificates still exists please open a new issue.