Missing EPG Data for UK Live channels

[kevwag]

Describe the bug

Nearly all EPG data is missing for UK Live channels. I started investigating this issue in the hope of being able to fix it myself but I have discovered that the root of the problem is in the xmltv data the addon pulls from https://github.com/Catch-up-TV-and-More/xmltv/ and therefore it's not something I can fix myself by editing local .py files - the data is automatically updated by a timed script.

PLEASE NOTE: I'm aware that technically, I should raise the issue in the xmltv repo section but I'm not sure if there's anyone actively monitoring it now - other than the automatic script, there doesn't appear to have been any activity for over 12 months. I have also copy / pasted this entire post as an issue (Issue #17) under the xmltv repo, just in case someone is still watching in there.

As this problem is server-based, platform info and Kodi log are not relevant - the plugin IS grabbing what little data is available so the log won't show any errors. I will delete those sections from this template and instead, I will post all the info I think is relevant to the issue and urls of the related files.

To Reproduce

Steps to reproduce the behaviour:

1. Configure UK Live channels with IPTV Manager as per the instructions
2. In Kodi settings Select PVR >> Guide >> Clear Data (forces refresh)
3. Select "TV" page from Kodi home screen
4. Look for Program listing for relevant UK channels

Expected behaviour

Programme schedule data should tell you what's on the channel.

Actual behaviour

Nearly all channels show "No information available", however, scrolling several days ahead sometimes reveals a random program listing.

Update:

I have subscribed to SchedulesDirect since the settings menu suggests that SD is now supported. I've entered my account login and password in Settings >> Accounts >> TV Guides, along with Lineup ID "GBR-1000193-DEFAULT" but this doesn't work either (I know the account and lineup ID work fine as I tested them on a NextPVR server).

My Debug Notes

My first observation is that in Catch-up-TV-and-More/xmltv/scripts/ there are two folders relevant to the UK: tv_grab_uk_bleb and tv_grab_uk_tvguide but the Bleb Website is broken and no longer contains any data. The TVguide Website IS working and it does contain all the relevant EPG data we need. The channel mapping and URL structure does not appear to have changed in comparison to the URLs the script attempts to scrape and the fact that it does occasionally manage to scrape a tiny bit of data, I'm inclined to think that the script is timing out while trying to scrape the data.

Looking at the files in the script's raw output folder, the scraping for some days fails completely and no xml file exists, the ones that do exist contain very little data and the log files reveal the failures:

Example:

If we look at one specific channel; "Yesterday" which has the EPG ID "320.tvguide.co.uk" From the latest update (12th April 2022), the Kodi EPG shows nothing at all until 00:00 on the 17th April where one single show ("Scouting for toys") is listed. The XML file for the 17th April does indeed contain that show, it also contains a couple more but for some reason they're not showing in the EPG. If we check The log file for the same day we see multiple complete failures until it gets to channel 320 where it fails on its first try, before managing to scrape 2% of the channel's data on a second attempt.

[Tue Apr 12 12:23:40 2022] Fetching listings: 0% Unable to retrieve web page for 140 Unable to retrieve web page for 140 Unable to retrieve web page for 178 Unable to retrieve web page for 178 ..... Unable to retrieve web page for 279 Unable to retrieve web page for 320 [Tue Apr 12 12:24:54 2022] Fetching listings: 2% Unable to retrieve web page for 422 Unable to retrieve web page for 422 etc....

There are 15 attempts to scrape between the two timestamps shown, (a period of 1 minute 14 seconds), which tells us that each URL attempt is only given 4.93 seconds to respond before the script calls it a timeout failure and moves on.
I'm guessing that the one partial success failed at 2% due to equally unrealistic timeout limits.

[joaopa00]

Does the problem still occur?

[kevwag]

@joaopa00 Yes, this has been broken forever! I did have a look to see if there was anything I could fix (with my albeit limited coding knowledge) but when I traced the problem, it's technically being caused outside of the addon (by the data in the separate Catch-up-TV-and-More/xmltv repo.) so there's nothing I could change locally in the addon that would make it work.

Although the problem is not technically with this addon, I don't think anyone's keeping an eye on Catch-up-TV-and-More/xmltv, that's why I opened an error report in here as well as in the xmltv repo, since you guys have the access to be able to fix it. Since the xmltv grabber runs on the github server and not locally, technically speaking, the addon is actually working ok. It's getting the xmltv lists from the server as it should, the problem is that the lists on the server are virtually empty.

If you look at the uk xmltv files, the data for most of the channels is missing. Some of the daily scrapes come up completely empty, whereas others sporadically contain occasional bits of data for one or two channels. The scraper logs seem to suggest that it's timing out when waiting for a response from the tvguide.co.uk server. The fact that it does sometimes successfully manage to scrape a bit of data suggests to me that it's just the timeout delay that's causing the issue, it does occasionally scrape a bit of data when the server responds fast enough.

As a workaround I've been using Takealug EPG Grabber from the Kodinerds repo. It's quite good and easy to understand but it's mainly focused on German and mainland European services so currently, it doesn't have all the UK channels in one place. You can find most of them by browsing the built-in lists for European cable services. However; not all of the lists are in English, even for English language channels... from England. A couple of my channel's guide listings came up in German on my first attempt!

It would be awesome if you could fix the addon's built-in epg grabber and I'm pretty sure it's as simple as increasing the timeout delay while it's scraping the tvguide.co.uk server.

[joaopa00]

The formely used UK grabber is broken completely (see issue with UK grapper). I tried to update to the newest version. Unfortunately, on the virtual machines of Github, there are problems of dependencies. So, don't your breath, it will take a very long moment to fix this problem, If ever it's fixed...