tv_grab_fi cannot be configured

[zweq]

XMLTV Version?

1.1.2

XMLTV Component?

tv_grab_fi

Perl Version

5.32.1

Operating System

Debian 5.10.113

What happened?

getting list of channels: 80% [===================================================================== ]fetchRaw(): could not fetch https://areena.yle.fi/tv/opas, error: 403 Forbidden, aborting at /usr/local/bin/tv_grab_fi line 1634.

The same error also occurs when trying to fetch information from areena.yle.fi.

What did you expect to happen?

Configuration process to finish without an error.

Did you see any warnings/errors?

Yes, see above.

What steps are needed to reproduce this issue?

/usr/local/bin/tv_grab_fi --configure --config-file /tmp/foo

Please attach your config file below:

-- coding: utf-8 --

(This is all that was created.)

Any other information?

I noticed the following issue from 2018 is somewhat similar: https://github.com/XMLTV/xmltv/issues/2 Therefore, to answer the questions posed there:

• It's not a network connectivity issue. The page loads fine in a browser.
• I believe I now only have files of the current version left. I did the perl Makefile.PL && make && make test && make install routine. When running from the installation directory, the result is the same.
• I have packages liblwp-protocol-https-perl (6.10-1) and liblwp-useragent-determined-perl (1.07-1.1) installed.

[stefanb2]

I just noticed the same thing. It can also be reproduced with curl, while on the other hand the URLs work fine when entered into a browser.

As far as I can tell it boils down to the fact that the site now rejects any request without a a Accept-Language header. I'm currently testing a rough fix to grab/fi/fi/source/yle.pm by inserting the following lines before the call to fetchTree():

  # TEMP FIX: without this header the request is rejected with 403
  $XMLTV::Get_nice::ua->default_header('Accept-Language' => 'en');

If you are not comfortable to change the source code, then I suggest to replace the YLE source with another source in the config file. This of course won't work for the --configure run.

[zweq]

Yeah, the fix works! Thanks!

[stefanb2]

Before committing a fix I have sent a query to YLE why they changed the site. Maybe they will be nice and fix the problem for us :-)

[stefanb2]

According to their response the web site experienced a DDoS attack over the weekend. Requiring that header was one of the changes they introduced to handle it.

So I guess it will stay and we have to adapt XMLTV for it.