Python3 th0ma7 updates - Githubissues

th0ma7 commented 3 years ago

Now working on the proper branch...

In order for the title & description to be fully compliant in both TVH & Kodi all escape characters needs to be converted to legacy style HTML such as the following:

        data = data.replace('&','&amp;')
        data = data.replace('"','&quot;')
        data = data.replace("'",'&apos;')
        data = data.replace('<','&lt;')
        data = data.replace('>','&gt;')

Other improvements

[x] When CC tag is found then <subtitles type="teletext" /> can be added
[x] Add minimal categories when xdetails=false
[x] Add <length units="minutes">60</length> tag
[x] Manage premiere and last-chance flags
[x] Add full channel name
[x] Only delete irrelevant ShowCache files
[x] Converted Cast, Premiere, New, Finale to using xmltv tags instead of extending the description.
[x] Remove non-essential english wording in description to be more generic and "bilingual"
[x] Follow XMLTV Capabilities for baseline

Notes

Patch related to "full channel name": the proposed variable exception = "..." may be better fitted with an xml config file.

Added more detailed log information such as:

2021/06/12 17:19:50 Running zap2epg-3.0
2021/06/12 17:19:50 Kodi settings version: 2
2021/06/12 17:19:50     Country: Canada [CAN]
2021/06/12 17:19:50     Postal code: J3B1M4
2021/06/12 17:19:50     TV Guide duration: 13 days
2021/06/12 17:19:50     TV Guide Start: 1623862800 [offset: 4 days]
2021/06/12 17:19:50     Lineup: Local Over the Air Broadcast
2021/06/12 17:19:50     Configuration file: /home/spksrc/script.module.zap2epg/epggrab/conf/zap2epg.xml
2021/06/12 17:19:50     Caching directory path: /home/spksrc/script.module.zap2epg/epggrab/cache

th0ma7 commented 3 years ago

@edit4ever I believe this PR is good for a review as you see fit. Honestly I never been able to get such a good result until I found your EPG grabber! And now with my proposed additions I get all I ever hoped from any TV guide.

Hopefully you'll consider merging the various patches proposed part of this PR. I can provide additional info as you see fit further explaining the changes I've done.

Ran such as:

$ python3 zap2epg.py

I was able to achieve the following results (see attached images) kodi-guide-movie-info-fr_CA kodi-guide-movie-info-detailed-fr_CA kodi-guide-movie-example-fr_CA tvheadend-movie-example-fr_CA

Using the following configuration:

<settings version="2">
  <setting id="zipcode">***-CAD-Postal-Code-***</setting>
  <setting id="lineupcode">lineupId</setting>
  <setting id="lineup">Local Over the Air Broadcast</setting>
  <setting id="device">-</setting>
  <setting id="days">14</setting>
  <setting id="redays">1</setting>
  <setting id="slist"></setting>
  <setting id="stitle">false</setting>
  <setting id="xdetails">true</setting>
  <setting id="xdesc">true</setting>
  <setting id="epgenre">3</setting>
  <setting id="epicon">1</setting>
  <setting id="tvhoff">false</setting>
  <setting id="usern">admin</setting>
  <setting id="passw">*********</setting>
  <setting id="tvhurl">192.168.***.***</setting>
  <setting id="tvhport">9981</setting>
  <setting id="tvhmatch">false</setting>
  <setting id="chmatch">true</setting>
  <setting id="desc01">10</setting>
  <setting id="desc02">18</setting>
  <setting id="desc03">6</setting>
  <setting id="desc04">13</setting>
  <setting id="desc05">6</setting>
  <setting id="desc06">15</setting>
  <setting id="desc07">2</setting>
  <setting id="desc08">9</setting>
  <setting id="desc09">2</setting>
  <setting id="desc10">14</setting>
  <setting id="desc11">6</setting>
  <setting id="desc12">11</setting>
  <setting id="desc13">6</setting>
  <setting id="desc14">12</setting>
  <setting id="desc15">6</setting>
  <setting id="desc16">20</setting>
  <setting id="desc17">6</setting>
  <setting id="desc18">19</setting>
  <setting id="desc19">0</setting>
  <setting id="desc20">0</setting>
</settings>

Where movie from guide used as example is as follow in xmltv:

        <programme start="20210603010000 0000" stop="20210603030600 0000" channel="63040.zap2epg">
                <episode-num system="dd_progid">MV00047402.0000</episode-num>
                <title lang="en">Témoin sous surveillance</title>
                <desc lang="en">Un policier se cache parmi la communauté amish afin de protéger un enfant qui a été témoin d&apos;un meurtre.
13+ | 1985 </desc>
                <length units="minutes">126</length>
                <date>1985</date>
                <icon src="https://zap2it.tmsimg.com/assets/p8671_b_h9_at.jpg" />
                <previously-shown  />
                <rating>
                        <value>13+</value>
                </rating>
                <credits>
                        <actor role="Det. Capt. John Book" src="https://zap2it.tmsimg.com/assets/25704_v9_bb">Harrison Ford</actor>
                        <actor role="Rachel Lapp" src="https://zap2it.tmsimg.com/assets/1133_v9_bb">Kelly McGillis</actor>
                        <actor role="Daniel Hochleitner" src="https://zap2it.tmsimg.com/assets/3384_v9_ba">Alexander Godunov</actor>
                        <actor role="Chief Paul Schaeffer" src="https://zap2it.tmsimg.com/assets/34191_v9_ba">Josef Sommer</actor>
                        <actor role="Samuel Lapp" src="https://zap2it.tmsimg.com/assets/81259_v9_bb">Lukas Haas</actor>
                        <actor role="Eli Lapp" src="https://zap2it.tmsimg.com/assets/29513_v9_ba">Jan Rubes</actor>
                        <actor role="Det. Lt. James McFee" src="https://zap2it.tmsimg.com/assets/650_v9_ba">Danny Glover</actor>
                        <actor role="Elaine" src="https://zap2it.tmsimg.com/assets/79320_v9_bb">Patti LuPone</actor>
                        <actor role="Moses Hochleitner" src="https://zap2it.tmsimg.com/assets/71706_v9_bb">Viggo Mortensen</actor>
                </credits>
                <category lang="en">Movie</category>
                <category lang="en">Crime drama</category>
        </programme>

th0ma7 commented 3 years ago

@edit4ever while trying to figure the easiest way to integrate your project I ended looking into the option of an embed-python script. This allowed merging zap2epg.py into tv_grab_zap2epg creating a unique TV grabber script. I've been testing it since a few days and it runs really well directly through TVH.

From there I used TVH default epggrab directory tree and moved the configuration file (now also renamed to zap2epg.xml for clarity), cache and logs under it such as:

$ tree epggrab
epggrab
├── cache
│   ├── ...
│   ├── SH03890063.json
│   ├── SH03891010.json
│   └── xmltv.xml
├── conf
│   └── zap2epg.xml
└── log
    └── zap2epg.log

What's in theory left to do would be:

to add a bit of code to manage the BaseDir variable if ran on raspi or synology which should be relatively trivial to do using uname -a output or similar.
Adjust the Kodi part to use the new config file name and directory structure
On my end, simply add the tv_grab_zap2epg and zap2epg.xml file to the SynoCommunity tvheadend package.

Again, trying to be as transparent as possible and documenting the changes I'm proposing. I will happily work things through with you if you are interested in order to get this merged. Otherwise just let me know as I can simply fork & rename on my side and resync as needed. Let me know what you see fit.

Again, thnx for you great project, cheers!

edit4ever commented 3 years ago

Thank you for these updates - I have been buried in a big project and still have some work to do on it. As soon as I get a chance, I will test and merge these PRs.

I appreciate the effort as I don't have much time to support this project lately.

th0ma7 commented 3 years ago

@edit4ever this is great! Let me know if I can do anything to help or if you'd like to chat to see what's may need additional adjustments.

Note that I can't test the Kodi part and haven't touched it at all. So it may be currently broken due to path and filename change originally pointing to settings.xml

th0ma7 commented 3 years ago

Honestly the code base is really good as it allowed me to easily add multiple functionalities. It already achieve above and beyond all of my expectations and much more than any other solution I tried previously. Also, with the latest additions it now makes the grabber to follow xmltv capabilities guidelines: http://wiki.xmltv.org/index.php/XmltvCapabilities

I still have a few ideas but not sure how much more mileage I'm willing to do for now. My focus is now more towards getting this merged in a form or another so I can start looking into integrating in the SynoCommunity tvheadend package and update the package repository.

Ideas for a later time:

[ ] Add a configuration setting for "credits"
[ ] Rework the code so it can run without any configuration file
[ ] Migrate the channel name exception variable to the configuration file
[ ] Add french channels exception variable in configuration channel in order to tag 'lang=fr' to title and descriptions as needed
[ ] Add grabber extra capabilities such as --extradetails, --extradescription, --credits.

Here is an example of the TVH configuration using extra arguments field: TVH-InternalEPG-Grabbers

th0ma7 commented 3 years ago

I was able to address http connection to TVH with commit https://github.com/edit4ever/script.module.zap2epg/pull/37/commits/da57f1d168cd6ead8f5a356b081235765f3b8d77 solving issue #36

It works both anonymously and with username/password:

spksrc@spksrc-debian10:~/script.module.zap2epg$ head -20 epggrab/log/zap2epg.log 
2021/06/13 13:18:45 Running zap2epg-3.0
2021/06/13 13:18:45 Kodi settings version: 2
2021/06/13 13:18:45     Country: Canada [CAN]
2021/06/13 13:18:45     Postal code: J3B1M4
2021/06/13 13:18:45     TV Guide duration: 14 days
2021/06/13 13:18:45     TV Guide Start: 1623589200
2021/06/13 13:18:45     Lineup: Local Over the Air Broadcast
2021/06/13 13:18:45     Configuration file: /home/spksrc/script.module.zap2epg/epggrab/conf/zap2epg.xml
2021/06/13 13:18:45     Caching directory path: /home/spksrc/script.module.zap2epg/epggrab/cache
2021/06/13 13:18:45 No channel list found - adding all stations!
2021/06/13 13:18:45 Tvheadend anonymous access...
2021/06/13 13:18:45 Starting new HTTP connection (1): 192.168.80.10:9981
2021/06/13 13:18:45 http://192.168.80.10:9981 "GET /api/channel/grid?all=1&limit=999999999&sort=name&filter=%5B%7B%22type%22:%22boolean%22,%22value%22:true,%22field%22:%22enabled%22%7D%5D HTTP/1.1" 200 2257
2021/06/13 13:18:45 Accessing Tvheadend channel list from: http://192.168.80.10:9981
2021/06/13 13:18:45 14 Tvheadend channels found...
2021/06/13 13:18:45 Checking for old cache files...
2021/06/13 13:18:45 Deleting old cache: 1623600000.json.gz
2021/06/13 13:18:45 Deleting old cache: 1623654000.json.gz
2021/06/13 13:18:45 Deleting old cache: 1623643200.json.gz
2021/06/13 13:18:45 Deleting old cache: 1623621600.json.gz

th0ma7 commented 3 years ago

I've completed the integration of the code into the "next" tvheadend package for Synology NAS. Again, your consideration to get this work merged would be much appreciated. My synocommunity PR is now pretty much pending on this to confirm final download URL and version to use from your github source (currently pointing to mine through this PR temporarily).

Also, I noticed the following error when ran anonymously on 127.0.0.1. The error was cause by restricting access to local LAN only for * user (thus loopback wasn't authorized). The JSON query failed such as:

2021/06/17 07:17:15 Tvheadend anonymous access...
2021/06/17 07:17:15 Starting new HTTP connection (1): 127.0.0.1:9981
2021/06/17 07:17:15 http://127.0.0.1:9981 "GET /api/channel/grid?all=1&limit=999999999&sort=name&filter=%5B%7B%22type%22:%22boolean%22,%22value%22:true,%22field%22:%22enabled%22%7D%5D HTTP/1.1" 401 250
2021/06/17 07:17:15 Accessing Tvheadend channel list from: http://127.0.0.1:9981
2021/06/17 07:17:15 Exception: main
Traceback (most recent call last):
  File "<string>", line 794, in mainRun
  File "<string>", line 130, in tvhMatchGet
  File "/volume1/@appstore/tvheadend/env/lib/python3.8/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/var/packages/python38/target/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/var/packages/python38/target/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/var/packages/python38/target/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm guessing that a better error capturing would be helpful to provide user with appropriate message such as Unable to access URL or similar.

edit4ever commented 3 years ago

merging zap2epg.py and the grabber breaks the kodi addon.

If you want to go down this route - then it should be it's own branch or fork.

Some of your updates/fixes would be good to integrate into the kodi version - but that merge isn't one of them.

Might I suggest you spin it off as a standalone grabber for TVH - my version was really designed for a beginner with a kodi setup and interface for simplicity. I can see value in your version as a standalone - but it really is a different animal because it assumes the user is knowledgeable and working with the command line for setup.

th0ma7 commented 3 years ago

@edit4ever

merging zap2epg.py and the grabber breaks the kodi addon.

It made so much sense to me to merge them to ease the integration on other platforms and allow supporting greater xmltv capabilities. But sadly I have no way to test it out on RaspberryPi. There must be a qemu/vm that can do the trick to which I could then help out as you see fit.

If you want to go down this route - then it should be it's own branch or fork.

Might I suggest you spin it off as a standalone grabber for TVH - my version was really designed for a beginner with a kodi setup and interface for simplicity. I can see value in your version as a standalone - but it really is a different animal because it assumes the user is knowledgeable and working with the command line for setup.

Fine by me but it was really much hoping not to. To me its just a matter of well describing the howto and that should do it.

Some of your updates/fixes would be good to integrate into the kodi version - but that merge isn't one of them.

Sad of having a duplicate project sharing 99% of the code and now diverting. I would much rather giving you a hand on fixing any remaining issues instead but I totally understand your disinterest of doing so.

th0ma7 commented 3 years ago

Closing the pull request. Let me know if you change your mind. In the meantime closing and created a new fork at https://github.com/th0ma7/tv_grab_zap2epg

edit4ever commented 3 years ago

I'm certainly not against merging - just not sure I have the time t sort it out right now. Also I want to make sure the Kodi addon works as a basic setup with GUI for people who don't work in the linux backend.

If you're interested in trying to see how both can work together - the easiest test system is using VMware and running LibreELEC. You can download the latest ova file from the repo below to install as a VM:

https://test.libreelec.tv/

The biggest challenge I see is that Kodi addons tend to work as python modules - and your merged grabber is no longer seen as a python file. The default.py file is what the zap2epg addon runs when selected in kodi - so it may just be adapting that to work with the merged file.

Thanks again for your work - it's definitely worth merging a lot of your tweaks even if we can't sort the merged file idea.

edit4ever / script.module.zap2epg

Python3 th0ma7 updates #37

Other improvements

Notes