search

jnylen / xmltv.se

Channel requests, issues, discussions etc.
9 stars 0 forks source link

Channels only having data for 2 days #48

Closed FritzHerbers closed 3 years ago

FritzHerbers commented 3 years ago

As per https://github.com/pixelmonster-ou/xmltv.se/issues/21 the free data is limited to 3 days.

However, only 2 days are available (list made on 2020-10-20 after update and showing the last program per channel):

RTL Crime HD                   crimehd.rtl.de                 2020-10-22 00:00:00
Nick Jr. HD                    jr.nick.de                     2020-10-22 00:00:00
RTL plus                       rtlplus.de                     2020-10-22 00:05:00
MTV HD                         mtv.ch                         2020-10-22 00:05:00
3sat HD                        hd.3sat.de                     2020-10-22 00:10:00
Das Erste HD                   hd.daserste.de                 2020-10-22 00:13:00
hr HD                          hrhd.daserste.de               2020-10-22 00:15:00
NDR HD                         ndrhd.daserste.de              2020-10-22 00:15:00
PHOENIX HD                     phoenixhd.daserste.de          2020-10-22 00:15:00
Comedy Central                 comedycentral.ch               2020-10-22 00:15:00
Universal TV                   universalchannel.de            2020-10-22 00:15:00
ProSieben                      prosieben.de                   2020-10-22 00:20:00
SAT.1 HD                       hd.sat1.de                     2020-10-22 00:20:00
ProSieben HD                   hd.prosieben.de                2020-10-22 00:20:00
MDR HD                         mdrhd.daserste.de              2020-10-22 00:23:00
Motorvision TV HD              motorvision.de                 2020-10-22 00:25:00
Nat Geo Wild HD                wildhd.natgeo.de               2020-10-22 00:30:00
Servus TV HD                   hd.servustv.de                 2020-10-22 00:30:00
WDR HD                         wdrhd.daserste.de              2020-10-22 00:30:00
RTL 2 HD                       rtl2.de                        2020-10-22 00:35:00
ARD-alpha HD                   br-alpha.daserste.de           2020-10-22 00:40:00
arte HD                        hd.arte.de                     2020-10-22 00:45:00
sportdigital HD                sportdigital.tv                2020-10-22 00:45:00
Eurosport 2 HD                 2.eurosport.de                 2020-10-22 01:00:00
kabel eins Doku HD             doku.kabel1.de                 2020-10-22 01:00:00
Stingray Classica              classica.de                    2020-10-22 01:15:00
kabel eins HD                  hd.kabel1.de                   2020-10-22 01:20:00
kabel eins                     kabel1.de                      2020-10-22 01:20:00
Nitro HD                       nitrohd.rtl.de                 2020-10-22 01:50:00
SAT.1 Gold HD                  goldhd.sat1.de                 2020-10-22 04:00:00
Super RTL                      superhd.rtl.de                 2020-10-22 04:00:00
ProSieben MAXX HD              maxxhd.prosieben.de            2020-10-22 04:00:00
RTL HD                         hd.rtl.de                      2020-10-22 04:00:00
TOGGO plus                     plus.toggo.de                  2020-10-22 04:00:00
Nat Geo HD                     hd.natgeo.de                   2020-10-22 04:20:00
ProSieben FUN HD               fun.prosieben.de               2020-10-22 04:35:00
BBC World News                 bbcworldnews.com               2020-10-23 00:00:00
Auto Motor + Sport H           auto-motor-und-sport.tv        2020-10-23 00:20:00
Extreme Sports                 extremesports.com              2020-10-23 01:25:00
MTV Brand New                  brandnew.mtv.de                2020-10-23 04:00:00
KiKA HD                        kikahd.daserste.de             2020-10-30 07:20:00

Can you please make the data available for 3 days.

On the channel status page, the non available dates for a channel should be set to "E means data is missing for that date".

As the three days limit is a "feature", it would be nice to find the information in the projects readme. Project relevant decisions/features/setups/architecture/user relevant changes/future changes/etc. should be summed up in the readme as they are hard to find working through the issues.

As you call it "free data", is/will there be a paid/registered download. Where is the project heading too? Time frame?

jnylen commented 3 years ago

Hi @FritzHerbers,

Can you please make the data available for 3 days.

It is available for 3 days. Today + 2 days.

On the channel status page, the non available dates for a channel should be set to "E means data is missing for that date".

Low priority thing. It won't get changed to E as we have data for it. Its just not exported.

As the three days limit is a "feature", it would be nice to find the information in the projects readme. Project relevant decisions/features/setups/architecture/user relevant changes/future changes/etc. should be summed up in the readme as they are hard to find working through the issues.

Low priority thing. From what I have learned maintaining it is that no one checks the README anyway.

As you call it "free data", is/will there be a paid/registered download. Where is the project heading too? Time frame?

If you read the README you could find a link to it. It's Honeybee.it.

FritzHerbers commented 3 years ago

Can you please make the data available for 3 days.

It is available for 3 days. Today + 2 days.

I checked it once again, it seems that loading at 05:30 CET is too early to receive 3 days, and it might be that the export files are not updated yet. At 08:30 CET I received the third day.

At which time are the export files finished updated/created?

As you call it "free data", is/will there be a paid/registered download. Where is the project heading too? Time frame?

If you read the README you could find a link to it. It's Honeybee.it.

A friend of mine already tried two times in the paste to setup and tried it out myself today. Sign-in is working, but no idea how payment goes (Icon->Billing). Therefor we thought the service is not yet activated. At https://www.honeybee.it/xmltv/edit, channels can be selected, at the switch at the top enabled. No file URL was ever created. Also no information if this URL (when it works) is automated downloadable (curl/basic authentication). Generated API key and tried the REST API example: 403 Forbidden.

FritzHerbers commented 3 years ago

There seem to be occasions that there are only 2 days available. I rerun 3 times today. As per 05.11.2020 12:30:

arte HD                        hd.arte.de                     2020-11-07 04:00:00
NDR HD                         ndrhd.daserste.de              2020-11-07 04:15:00
...

There is data available for the 05.11 and 06.11: 2 days

FritzHerbers commented 3 years ago

Since 2 days I haven't received an update. I am using tv_grab_eu_xmltvse, which uses a HTTP cache. One file (2020-11-14) in the cache was corrupt, therefor the update process failed.

As this is about the third time I had a corrupt file in the cache, I hope that xmltv.se only shows finished data files for download (no growing file during processing) and that the issue is solely on my side.

xmltv.se should process the data file creation in a temporary file and move the file to the download area.

@jnylen How are the data files created (temp-file and moved, or a growing file during processing) ?

jnylen commented 3 years ago

@fritzherbers

There seem to be occasions that there are only 2 days available. I rerun 3 times today. As per 05.11.2020 12:30:

arte HD                        hd.arte.de                     2020-11-07 04:00:00
NDR HD                         ndrhd.daserste.de              2020-11-07 04:15:00
...

There is data available for the 05.11 and 06.11: 2 days

These should get done per auto as those are built on the data from the main channel. And they both have data now.

Since 2 days I haven't received an update. I am using tv_grab_eu_xmltvse, which uses a HTTP cache. One file (2020-11-14) in the cache was corrupt, therefor the update process failed.

As this is about the third time I had a corrupt file in the cache, I hope that xmltv.se only shows finished data files for download (no growing file during processing) and that the issue is solely on my side.

xmltv.se should process the data file creation in a temporary file and move the file to the download area.

@jnylen How are the data files created (temp-file and moved, or a growing file during processing) ?

They are created directly with the data so no growing file. The gzipped file is built from the content of the xml file. Do these issues disappear after 3 hours?

FritzHerbers commented 3 years ago

@jnylen

Do these issues disappear after 3 hours?

As it is a cache, when the file is needed once more, it is taken from the cache, also when corrupt/incomplete. Only when I notice there is an incomplete file, I delete the cache. With the sliding 3 days data availability window, when a file is incomplete after 3 days it will not be sourced in, and local processing is working fine again (when no later date has corrupt files).

They are created directly with the data so no growing file.

I don't understand what you mean with "created directly". As there are users downloading files at any time. The file (should be sane, mine was partial and missing the part up to ) should be moved into this download area in an atomic way. It should be created (or downloaded) in a temporary file and when all processing is done (gzip) on the file, moved into the download area.

The only atomic way is a move (not a copy of the file). Sorry to ask again, do you move the file into the download area?

When you can assure that you move the data (gzip) file, transfer errors can happen anytime. tv_grab_eu_xmltvse (or the xmltv caching library) has to take care, that incomplete files are invalidated (and used for processing) and not written to the cache.

jnylen commented 3 years ago

@jnylen

Do these issues disappear after 3 hours?

As it is a cache, when the file is needed once more, it is taken from the cache, also when corrupt/incomplete. Only when I notice there is an incomplete file, I delete the cache. With the sliding 3 days data availability window, when a file is incomplete after 3 days it will not be sourced in, and local processing is working fine again (when no later date has corrupt files).

They are created directly with the data so no growing file.

I don't understand what you mean with "created directly". As there are users downloading files at any time. The file (should be sane, mine was partial and missing the part up to ) should be moved into this download area in an atomic way. It should be created (or downloaded) in a temporary file and when all processing is done (gzip) on the file, moved into the download area.

The only atomic way is a move (not a copy of the file). Sorry to ask again, do you move the file into the download area?

When you can assure that you move the data (gzip) file, transfer errors can happen anytime. tv_grab_eu_xmltvse (or the xmltv caching library) has to take care, that incomplete files are invalidated (and used for processing) and not written to the cache.

The files are updated and created directly. Meaning we write to the exact same files you fetch.

I'm full aware what you talk about. This would mean a bit of changes in the exporter to handle this. And this has been on the to do list for a while.

I don't provide any guarantees that the files are working as XMLTV.se is provided as-is.

But I sin to have this fixed but when I can't say I'm busy with other stuff up until December 17. So can't promise it will be fixed before then.

FritzHerbers commented 3 years ago

@jnylen

No valid data for

PHOENIX HD                     phoenixhd.daserste.de          2020-12-21 23:00:00
NDR HD                         ndrhd.daserste.de              2020-12-22 00:00:00
MDR HD                         mdrhd.daserste.de              2020-12-22 00:00:00
jnylen commented 3 years ago

@FritzHerbers Fixed. They have been queued for a reimport. Issue was that they change their filenames so the filename regex broke.

Can you next time create a new issue rather than posting it in here? Otherwise it falls between the chairs as this issue is lower prioritized.

jnylen commented 3 years ago

Closing this now. It will be kept as is for now.

Files have changed to be provided by our cloud provider so it shouldn't have the same issue as before (with non-complete files). This sync will run on a schedule every 10-15 min.