[Question] Current status of EPG in XMLTV format

[rchovan]

Hi, what is current status for XMLTV ? I saw issue #975 and merge #821, but this feature is not documented in API, and of course, does not work on my set-top-box.

Do I miss something ?

This is my box:

"imagedistro": "openatv", 
  "mem1": "1154780 kB", 
  "mem3": "482628 kB vo\u013en\u00e9 / 1154780 kB celkom", 
  "mem2": "482628 kB", 
  "oever": "OE-Alliance 4.4", 
  "transcoding": true, 
  "fp_version": 20038, 
  "kernelver": "3.14.2", 
  "uptime": "10d 13:06", 
  "enigmaver": "2020-05-02", 
  "driverdate": "20160120", 
  "imagever": "6.4.20200502", 
  "friendlychipsetdescription": "\u010cipset (Verzia frontprocesora)", 
  "shares": [], 
  "friendlyimagedistro": "OpenATV", 
  "brand": "GI", 
  "boxtype": "xpeedlx3", 
  "machinebuild": "inihdp", 
  "lcd": 0, 
  "EX": "", 
  "webifver": "OWIF 1.3.9", 
  "grabpip": 0, 

[rchovan]

Do you have example of usage ? on request http:///web/epgxmltv I get response: Err500 - Processing Failed

[jbleyel]

http://<ip_address>/web/epgxmltv?bRef=<bouquet ref>&lang=<iso language code>

[rchovan]

Already tried this approach, but still I have error:

request bouquets

url: 
http://192.168.2.7/api/bouquets

response:
{
 "bouquets": [
  [
   "1:7:1:0:0:0:0:0:0:0:FROM BOUQUET \"userbouquet.SkylinkSlovakRepublic.tv\" ORDER BY bouquet", 
   "Skylink Slovak Republic"
  ], 
  [
   "1:7:1:0:0:0:0:0:0:0:FROM BOUQUET \"userbouquet.favourites.tv\" ORDER BY bouquet", 
   "Skylink CZ/SK"
  ], 
  [
   "1:7:1:0:0:0:0:0:0:0:FROM BOUQUET \"userbouquet.iptv.tv\" ORDER BY bouquet", 
   "IPTV"
  ], 
  [
   "1:7:1:0:0:0:0:0:0:0:FROM BOUQUET \"userbouquet.iptv_webcams.tv\" ORDER BY bouquet", 
   "IPTV Webcams"
  ], 
  [
   "1:7:1:0:0:0:0:0:0:0:FROM BOUQUET \"userbouquet.LastScanned.tv\" ORDER BY bouquet", 
   "Last Scanned"
  ]
 ]
}

request epgxmltv

url: 
http://192.168.2.7/web/epgxmltv?bRef=%221:7:1:0:0:0:0:0:0:0:FROM%20BOUQUET%20%22userbouquet.SkylinkSlovakRepublic.tv%22%20ORDER%20BY%20bouquet%22&lang=en

response:
GET http://192.168.2.7/web/epgxmltv?bRef=%221:7:1:0:0:0:0:0:0:0:FROM%20BOUQUET%20%22userbouquet.SkylinkSlovakRepublic.tv%22%20ORDER%20BY%20bouquet%22&lang=en 500 (Internal Server Error)

[jbleyel]

Please try this:

http://192.168.2.7/web/epgxmltv?bRef=1%3A7%3A1%3A0%3A0%3A0%3A0%3A0%3A0%3A0%3AFROM%20BOUQUET%20%22userbouquet.SkylinkSlovakRepublic.tv%22%20ORDER%20BY%20bouquet&lang=en

[rchovan]

Hi, sorry for long delay. Yes, this url is working, but it is ok almost 3 min for response ?

[jbleyel]

This is not an error. It depends on the amount of channels in your bouquet and the amount of epg entries.

[jbleyel]

Can we close this issue?

You can add &endTime=XXX to limit the epg data. XXX = max. minutes

Please do not use a value greater 10000 which means 7 days.

[rchovan]

Hi, report is generated, but I'm not sure if it is ok, Thank for endTime, it is useful. But I think there is another error. With endTime=60 , it looks OK,

but when I use for example 120min, then there is error:

This page contains the following errors:
error on line 3473 at column 308: PCDATA invalid Char value 26
Below is a rendering of the page up to the first error.

and here is response: epgxmltv.zip

[jbleyel]

There is an encoding problem.

<desc lang="sk">The Svetlanov Symphony Orchestra of Russia and pianist Boris Berezovsky are led by conductor Vladimir Verbitsky in a performance of works by Aram Khachaturyan and Alexander Borodin. The concert was recorded in December of 2016 at the Tchaikovsky Concert Hall in Moscow and features Borodins In the Steppes of Central Asia, and Khachaturyan's Piano Concerto in D-flat Major and "Adagio from Spartacus and Phrygia" (transcription for piano and orchestra). The programme is produced by the Moscow Philharmonic Society, which Saint Petersburg-born composer Dmitri Shostakovich himself once described as playing a significant role "in the development of musical life [in Russia]. It is a kind of university which is attended by millions of music lovers and thousands of musicians." The Moscow Philharmonic Society was founded in 1922 by then-Commissar for Culture, Anatoly Lunacharsky, and has over the years come to be Russia's leading concert organizing institution.</desc>

I don't know how to fix because the value should be correct with this convert.

return unicode(val, 'utf_8', errors='ignore').encode('utf_8', 'ignore')

This can be:

A) The channel "Stingray Classica HD" sends not utf8 valid strings. B) The epg data on your box is not full valid utf8. C) Your image is old or have some bugs. D) anything else

I can't fix this because i need this channel, your box and your image to reproduce this error.

[jbleyel]

@ims21 : Maybe you can help here.

[jbleyel]

Please put this file:

https://gist.github.com/jbleyel/abeccd1addd8f76e71a0b43586a8969b

on your Box as

/usr/lib/enigma2/python/Plugins/Extensions/OpenWebif/controllers/models/services.py

Then reboot your box.

[rchovan]

Hi, I think, it is working, and it is much much faster. Request without end time, which lasted almost 3 mins, take now only 650ms.

btw, I have OpenATV 6.4, updated to latest version:

  "imagedistro": "openatv", 
  "mem1": "1154780 kB", 
  "mem3": "961816 kB vo\u013en\u00e9 / 1154780 kB celkom", 
  "mem2": "961816 kB", 
  "oever": "OE-Alliance 4.4", 
  "transcoding": true, 
  "fp_version": 20038, 
  "kernelver": "3.14.2", 
  "uptime": "0:12", 
  "enigmaver": "2020-05-21", 
  "driverdate": "20160120", 
  "imagever": "6.4.20200521", 
  "friendlychipsetdescription": "\u010cipset (Verzia frontprocesora)", 
  "shares": [], 
  "timerpipzap": false, 
  "friendlyimagedistro": "OpenATV", 
  "timerautoadjust": false, 
  "brand": "GI", 
  "boxtype": "xpeedlx3", 
  "machinebuild": "inihdp", 
  "lcd": 0, 
  "EX": "", 
  "webifver": "OWIF 1.3.9", 

[jbleyel]

I think 650 ms vs. 3 min can't be the same result. You may have not so many epg entries after reboot and then you will have a faster result. Please zap to "Stingray Classica HD" and wait a few seconds. Then zap to a few other channels and wait a few seconds.

Then try the request again.

[rchovan]

You are right, even after EPGrefresh I have for full EPG only 137kB big response. Previously it was 400k for 2hours of EPG. I've checked it manually, for service "Stingray Classica HD (1:0:1:17D5:C91:3:EB0000:0:0:0:)" I have only 2 results in response. Same it is for other services. So in this changed code, request without endTime does not return all EPG entries, it is limited.

I saw this error again once, but currently I can't simulate it. When I try 1440 min for one day, it is ok, EPG for 2 days end with connection reset.

I'll check it over weekend, because EPG for Stingray is only till Saturday.

I have checked sRef for service, but probably it is not implemented:

http://192.168.2.7/web/epgxmltv?sRef=1%3A0%3A1%3A17D5%3AC91%3A3%3AEB0000%3A0%3A0%3A0%3A

[jbleyel]

The epgxmltv is only for bouquets and not for a single service. There are many other requests especially for services. -> https://github.com/E2OpenPlugins/e2openplugin-OpenWebif/wiki/OpenWebif-API-documentation

[rchovan]

Hi, because of some other bugs in OpenATV 6.4, I need to downgrade to OpenATV 6.3. STB was unstable, and it freezes a lot. Anyway, I have uninstalled OpenWebIf provided with image, and installed latest version from git (https://github.com/E2OpenPlugins/e2openplugin-OpenWebif/raw/gh-pages/enigma2-plugin-extensions-openwebif_1.3.9-e2openpluginsgit20200523_all.ipk). Then I have replaces services.pyo with file provided gist from you.

Here is error again: response:

This page contains the following errors:
error on line 18720 at column 23: PCDATA invalid Char value 26
Below is a rendering of the page up to the first error.

line in xml file:

    <programme start="20200524220000 +0200" stop="20200525003228 +0200" channel="1:0:1:17D5:C91:3:EB0000:0:0:0:">
        <title lang="sk">Dvoák - Rusalka</title>
        <sub-title lang="sk">Dvoák - Rusalka</sub-title>
        <desc lang="sk">Ádám Fischer conducts the orchestra and choir of La Monnaie in a performance of Antonín Dvoák's (1841-1904) Rusalka, a fairytale opera in three acts. Rusalka is the composer's ninth opera and still one of the cornerstones of Czech opera houses. The main roles are played by Myrtò Papatanasiu (Rusalka), Pavel ernoch (Prince), Annalena Persson (Foreign Princess), Willard White (Vodnik) and Renée Morloc (Je¸ibaba). A rusalka is a water sprite from Slavic mythology, living in a lake or river. In this story, Rusalka wishes to become human in order to be loved by a prince. For her wish to be fulfilled, Rusalka must give up her voice and face eternal damnation if her love fails. Stage director Stefan Herheim masterfully presents this lyrical fairytale, recorded at La Monnaie, Brussels, in 2016. In this widely acclaimed interpretation, the fairytale elements sometime take on frighteningly realistic dimensions so that one might see this enchanting production as a psychoanalytical study of male fantasies and female archetypes.</desc>
    </programme>

and full xml response: epg.zip

[rchovan]

@jbleyel Hi, any news on this ?

[jbleyel]

Hi @ims21 , can you help here? Because i have no channel with this encoding problem.

[ims21]

All my EPG is broadcasted ... I think, he using imported epg ... then let solve it who creating imported EPG. He must convert to true cp ...

[jbleyel]

Hi @ims21 : Do you have this channel? "Stingray Classica HD (1:0:1:17D5:C91:3:EB0000:0:0:0:)" Maybe the EPG data from the provider is not utf-8.

[ims21]

• Stingray Classica HD is German service => some smart boys creating EPG in M$ Word and not in some utf-8 prof. program. Lat1 has not "ř". It is wrong converted to UTF-8 on provider side. In utf-8 there must be "ř" as 2chars: 0xC5 0x99

I have in E2 there Dvoák instead Dvořák. I don't care.

Similar is when using lower and upper quotation marks in Word and then put it to broadcasting...

[ims21]

In CP1250 is "ř" as 0xE0 => you can see, that in epg is wrong trancoded ... in UTF-8 it must be "0xC5 0x99". Let he call to provider :)

[rchovan]

@ims21, I think, that force provider to fix this on his side is almost impossible. Could you please add any error correction to avoid this error ? Even If that wrong character will be missing, is still better than crash during request.

[jbleyel]

That's what i wanna do but i need the "buggy" epg.dat for testing and the sref of the channel. I have only Astra 19,2.

[rchovan]

Hi, @jbleyel If it helps, I'm attaching zipped epg.dat, epg.xml and error line: epg.zip

This page contains the following errors: error on line 23316 at column 304: PCDATA invalid Char value 26 Below is a rendering of the page up to the first error.

[jbleyel]

The xml is empty. I need only the channel name.

[rchovan]

sorry, uploading correct XML: epgxmltv.zip

[ims21]

It has no sence add any of 'correction' (workarounds) to Owif or Enigma. Enigma or Owif, both know live with it well. I have no crash on this.

If you have crash , this crash must do your EPG imported. Solve it in this importer, that it will drop this chars...

[jbleyel]

Hi @rchovan , what's the source of your EPG Data? Do you get the EPG from SAT only?

[rchovan]

Yes, only from SAT, and refreshed with epgrefresh service every day.

[ims21]

from sat in xml format ???

[jbleyel]

No! The xml is from openwebif the epg.dat is not from an external source. The wrong encoding comes from the sat provider.

[ims21]

@jbleyel - do not add any workaround to Owif. I am using same EPG from same program with same wrong char code and E2 or Owif are stable !

Let they fixing his e2... @rchovan what EPG type do you have set in enigma ? OpenTV too ? Or some import ?

[jbleyel]

I have fixed the issue but i have also fixed some other problems. There is only a single line in owf now. This line can be removed if e2 is fixed.

[rchovan]

I'm not sure. AFAIK I have configured only epgrefresh to scan for EPG on this bouquete. And the error is not 100% reproducible, it depends on the description. As you can see in this thread, there were some days without error.

Sorry, can't switch to english language. EPG settings:

[ims21]

Hmm ... workaround for something, on what not crashing enigma or owif ... Should be this optionaly? I want see, what is broadcasted.

@rchovan - try turn off any EPG and only EIT using. Then tell, if it crashing...

[ims21]

Yes - whole epgcache + EPGRefresh + saving EPG is one of randomize self crashing operation ... do not look problems, where arent. It not crashed on this chars.

[ims21]

turn of saving data EPG ... why do you needed it ?

[rchovan]

I have saving EPG for auto timers and restore after reboot. I have disabled saving, and other functions, deleted epg.dat and restarted E2 (init 4 > 3).

root@xpeedlx3:~# ls  /media/hdd/epg*
ls: /media/hdd/epg*: No such file or directory

After restart I have same error, even with few EPG data:

[ims21]

[ims21]

No crash ... this char is ignored.

[rchovan]

@ims21 could you please generate EPG XMLTV with /web/epgxmltv service ? I'm not talking that it is wrong displayed in browser, I'm talking, that this service can't create XMLTV file, request end with error so I can't use this service.

[ims21]

XMLTV is OpenATV command or is it standard dmm command ?

[rchovan]

It is OpenWebif service to generate XMLTV (http://wiki.xmltv.org/index.php/Main_Page) compatible file. Please look at comments from this: https://github.com/E2OpenPlugins/e2openplugin-OpenWebif/issues/1012#issuecomment-628648882 to this: https://github.com/E2OpenPlugins/e2openplugin-OpenWebif/issues/1012#issuecomment-628786314

[jbleyel]

Please try the latest version i have made several changes.

[rchovan]

Can't build, will wait for autobuilds. I'll test it tomorrow. Thanks.

[rchovan]

can't install this new build

 opkg install enigma2-plugin-extensions-openwebif_1.4.0-latest_all.ipk
Upgrading enigma2-plugin-extensions-openwebif from 1.3.9-e2openpluginsgit20200523-r0 to 1.4.0-e2openpluginsgit20200615-r0 on root.
Collected errors:
 * calculate_dependencies_for: Cannot satisfy the following dependencies for enigma2-plugin-extensions-openwebif:
 *      python-six (>= 1.15) *
 * opkg_solver_install: Cannot install package enigma2-plugin-extensions-openwebif.

this is my version

root@xpeedlx3:/hdd# opkg list-installed | grep six
python-six - 1.14.0-r0

[jbleyel]

I will change this to 1.14. You can use opkg install --nodeps

[rchovan]

Hi, reinstalled without deps. So far XMLTV is generated without error, but all description for selected provider, are in English language. I'll see how it will be with Slovak/Czech description, but probably need to wait.

I want to check EPG manually on that channel, but realized, that I can't see EPG on any chanel in OWIF,

[rchovan]

Hi @jbleyel I can confirm, that this issue is fixed with latest version,

OWIF 1.3:

OWIF 1.4:

but there is new one mentioned in previous post. Should I open new issue for new bug ?