Offline Import Not Working

[thawtes]

offline import not working

image: openatv 6.4

Traceback (most recent call last): File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 13, in <module> File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py", line 12, in <module> ValueError: Attempted relative import in non-package

[TwolDE2]

OfflineImport import EPGConfig import EPGImport ————->>> change to Import . EPGConfig import . EPGImport. ——-> check it then works

[thawtes]

import EPGConfig

i try and result import . EPGConfig ^ SyntaxError: invalid syntax

not work

[TwolDE2]

sorry try: from . import EPGConfig from . import EPGImport

[thawtes]

from . import EPGConfig

I tried again but it still doesn't work

Traceback (most recent call last): File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 12, in <module> from . import EPGConfig ValueError: Attempted relative import in non-package

[TwolDE2]

so have now realised this is run basically as a script and those imports in py3 are therefore much harder to handle ... when I have time to concentrate will look, unless someone else has a quick solution you might try with python execute and the -m option

[thawtes]

so have now realised this is run basically as a script and those imports in py3 are therefore much harder to handle ... when I have time to concentrate will look, unless someone else has a quick solution you might try with python execute and the -m option

is there any progress? were you able to do something?

( -m option not working )

thx

[TwolDE2]

Afraid it was summerged by other issues - Now you have reminded me will look again, but will be next week

[thawtes]

Afraid it was summerged by other issues - Now you have reminded me will look again, but will be next week

thanks, good luck...

[TwolDE2]

@thawtes - so by adding to the sys.path I was able to get so far, but still big issues . So how & on what do you normally run this import

[Huevos]

@thawtes can you tell us the exact command you uses on the command line and where the files being processed are located.

[Huevos]

@thawtes please see this commit... and please give feedback. https://github.com/oe-alliance/XMLTV-Import/commit/5714422eebdb3ff4a89bc0294672dc25172ae141

[thawtes]

@thawtes - so by adding to the sys.path I was able to get so far, but still big issues . So how & on what do you normally run this import

python /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py /etc/epgimport/mycustom.sources.xml (or rytec.sources.xml) When I give this command, the import of EPG data starts at that moment and the result is successful. A great feature for me.

[thawtes]

@thawtes can you tell us the exact command you uses on the command line and where the files being processed are located.

python /usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py /etc/epgimport/rytec.sources.xml When I give this command, the import of EPG data starts at that moment and the result is successful. A great feature for me. everything was working great before this commit https://github.com/oe-alliance/XMLTV-Import/commit/a848f6ea018f43a90b6924f5240e4c632992e4cf

[thawtes]

@thawtes please see this commit... and please give feedback. 5714422

now i tried again as you said, so like this commit https://github.com/oe-alliance/XMLTV-Import/commit/5714422eebdb3ff4a89bc0294672dc25172ae141

command cd /usr/lib/enigma2/python/Plugins/Extensions

command python -m EPGImport.OfflineImport it seems to be working, so it doesn't give an error message

command python -m EPGImport.OfflineImport /etc/epgimport/rytec.sources.xml unfortunately it doesn't work :(

result

[EPGImport] oudeis patch not detected, using epg.dat instead.
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 76, in <module>
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 59, in importFrom
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py", line 237, in beginImport
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat_importer.py", line 3, in <module>
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat.py", line 13, in <module>
ModuleNotFoundError: No module named 'boxbranding'

[TwolDE2]

which image are you running that on????

[Huevos]

Ok so try this... hopefully there will not be more...

In epgdat.py, lines 12/13:

from boxbranding import getImageDistro
EpgDatV8 = getImageDistro() in ("openvix",)

Change to:

try:
    from boxbranding import getImageDistro
    EpgDatV8 = getImageDistro() in ("openvix",)
except:
    EpgDatV8 = os.path.isfile("/etc/image-version") and "distro=openvix" in open("/etc/image-version").read()

[thawtes]

which image are you running that on???? openATV 7.0

[thawtes]

Ok so try this... hopefully there will not be more...

In epgdat.py, lines 12/13:

from boxbranding import getImageDistro
EpgDatV8 = getImageDistro() in ("openvix",)

Change to:

try:
  from boxbranding import getImageDistro
  EpgDatV8 = getImageDistro() in ("openvix",)
except:
  EpgDatV8 = os.path.isfile("/etc/image-version") and "distro=openvix" in open("/etc/image-version").read()

command now it works! thanks @Huevos @TwolDE2

python -m EPGImport.OfflineImport /etc/epgimport/rytec.sources.xml
[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= XYZ_SERVICES
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### thread is ready ### Events: 8
[EPGImport] imported 8 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

but it doesn't create /media/hdd/epg_new.dat file therefore, the import could not be performed ??

and is this an error message?

[EPGImport] failed to load C implementation, sorry
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object

[Huevos]

Can you add some debug in EPGImport.py?

    def doThreadRead(self, filename):
        """This is used on PLi with threading"""
        for data in self.createIterator(filename):
            if data is not None:
                print("data", data)
                print("type(data)", type(data))
                self.eventCount += 1
                try:
                    r, d = data
                    print("r", r)
                    print("type(r)", type(r))
                    print("d", d)
                    print("type(d)", type(d))
                    print("d[0]", d[0])
                    print("type(d[0])", type(d[0]))
                    print("self.longDescUntil", self.longDescUntil)
                    print("type(self.longDescUntil)", type(self.longDescUntil))
                    print("d[0] > self.longDescUntil", d[0] > self.longDescUntil)
                    if d[0] > self.longDescUntil:
                        # Remove long description (save RAM memory)
                        d = d[:4] + ('',) + d[5:]
                    self.storage.importEvents(r, (d,))
                except Exception as e:
                    import traceback
                    traceback.print_exc()
                    print("[EPGImport] ### importEvents exception:", e, file=log)
        print("[EPGImport] ### thread is ready ### Events:", self.eventCount, file=log)
        if filename:
            try:
                os.unlink(filename)
            except Exception as e:
                print("[EPGImport] warning: Could not remove '%s' intermediate" % filename, e, file=log)

        return

[thawtes]

@Huevos

[EPGImport] oudeis patch not detected, using epg.dat instead.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= MY SERVICES
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA59:7B0E:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:19:CA01:A926:42E:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:19:CA01:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:19:CA01:A926:42E:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:19:CA01:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:'], (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649019600, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649019600
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:'], (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0))
type(data) <class 'tuple'>
r ['1:0:1:EA67:4C5:4C5:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649062800, 43200, 'my service program info', '', 'my service descriptions', 0)
type(d) <class 'tuple'>
d[0] 1649062800
type(d[0]) <class 'int'>
self.longDescUntil 1649454729.7295923
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[EPGImport] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport] ### thread is ready ### Events: 8
[EPGImport] imported 8 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[Huevos]

You seem to be missing part of the debug from your output:

                    import traceback
                    traceback.print_exc()

[thawtes]

You seem to be missing part of the debug from your output:

                    import traceback
                    traceback.print_exc()

there are these lines

EPGImport.py https://github.com/thawtes/XMLTV-Import/commit/28eee358e2073f18411451ba6284510b379160f8

[Huevos]

Your log is showing [EPGImport] ### importEvents exception but not the output from the traceback.

[TwolDE2]

@thawtes looking at the log, I believe that you are not using epgcache to import events but epgdat_importer and epgdat-------> [EPGImport] oudeis patch not detected, using epg.dat instead. [EPGImport] failed to load C implementation, sorry

so can you use the 3 modules updated in my last git update(titled offline changes) - I have added Huevos debug plus a bit more to see if which modules we hit. see ---> https://github.com/TwolDE2/XMLTV-Import/commit/01f1510cc51f621a65b6c2ca0b06c0b4dee387a4

[thawtes]

@TwolDE2 @Huevos

after this commit https://github.com/TwolDE2/XMLTV-Import/commit/01f1510cc51f621a65b6c2ca0b06c0b4dee387a4

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649106000, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649106000, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649106000
type(d[0]) <class 'int'>
self.longDescUntil 1649539498.63188
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649106000, 43200, 'program_name', '', 'program_desc', 0)
[EPGImport][doThreadRead] ### importEvents exception: name 'event_starttime_unix_gmt' is not defined
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649149200, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649149200, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649149200
type(d[0]) <class 'int'>
self.longDescUntil 1649539498.63188
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649149200, 43200, 'program_name', '', 'program_desc', 0)
[EPGImport][doThreadRead] ### importEvents exception: name 'event_starttime_unix_gmt' is not defined
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[TwolDE2]

@thawtes My bad on print statement, but at least we know where we are...... epgdat.py!

1. can you copy dreamcrc.c from your XMLTV git into /usr/lib/enigma2/python/Plugins/Extensions/EPGImport and lets see if we can import the C++ routine.
2. You said you are using ATV, so I am guessing they are using the V8 format for the epg, so have forced that variable to True.
3. added the import change, more debug, the EpgDatV8 change, etc to epgdat.py Hopefully all correct this time, and hopefully we will see where it breaks . could be the pack routines or the fancy crc routine as the C++ code was being simulated in python(possibly for the 1st time). Anyway here: https://github.com/TwolDE2/XMLTV-Import/commit/7c72478537af21ddfb4c1b900f3c6e3b0c36138a

[thawtes]

@thawtes My bad on print statement, but at least we know where we are...... epgdat.py!

1. can you copy dreamcrc.c from your XMLTV git into /usr/lib/enigma2/python/Plugins/Extensions/EPGImport and lets see if we can import the C++ routine.
2. You said you are using ATV, so I am guessing they are using the V8 format for the epg, so have forced that variable to True.
3. added the import change, more debug, the EpgDatV8 change, etc to epgdat.py Hopefully all correct this time, and hopefully we will see where it breaks . could be the pack routines or the fancy crc routine as the C++ code was being simulated in python(possibly for the 1st time). Anyway here: TwolDE2@7c72478

after this commit TwolDE2@7c72478

root@vuduo2:/usr/lib/enigma2/python/Plugins/Extensions# python -m EPGImport.OfflineImport /etc/epgimport/myServices.sources.xml
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 76, in <module>
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/OfflineImport.py", line 59, in importFrom
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/EPGImport.py", line 245, in beginImport
    from . import epgdat_importer
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat_importer.py", line 3, in <module>
    from . import epgdat
  File "/usr/lib/enigma2/python/Plugins/Extensions/EPGImport/epgdat.py", line 16
    import . dreamcrc
           ^
SyntaxError: invalid syntax
[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.

import . dreamcrc change to import dreamcrc

result ...

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] failed to load C implementation, sorry
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServivces.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649192400, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649192400, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649192400
type(d[0]) <class 'int'>
self.longDescUntil 1649625082.8190384
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649192400, 43200, 'program_name', '', 'program_desc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649192400 43200 program_name program_desc
[epgdat][short_desc] s :  program_name
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649235600, 43200, 'program_name', '', 'program_desc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649235600, 43200, 'program_name', '', 'program_desc', 0)
type(d) <class 'tuple'>
d[0] 1649235600
type(d[0]) <class 'int'>
self.longDescUntil 1649625082.8190384
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649235600, 43200, 'program_name', '', 'program_desc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649235600 43200 program_name program_desc
[epgdat][short_desc] s :  program_name
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[TwolDE2]

Thanks- for your patience! so made a couple of changes 1) fixed hopefully import, so if you copied the module over perhaps it will find it. 2) I encode s before its used in epgdat.py, so lets see if that gets through the struct.pack routines see https://github.com/TwolDE2/XMLTV-Import/blob/python3/src/EPGImport/epgdat.py there are a couple of fields in the struct.pack, I am not sure about, but perhaps its just my lack if understanding.

[thawtes]

you're welcome, I really thank you..

New LOG

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1648069200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1648069200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1648069200
type(d[0]) <class 'int'>
self.longDescUntil 1649796613.7646136
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1648069200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1648069200 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1648112400, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:', '1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1648112400, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1648112400
type(d[0]) <class 'int'>
self.longDescUntil 1649796613.7646136
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1648112400, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1648112400 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

it keeps giving this error: [EPGImport][doThreadRead] ### importEvents exception: argument for 's' must be a bytes object

and, I still don't understand why /media/hdd/epg_new.dat file is not created

[TwolDE]

So import now works for C++ Talked with Huevos and as a result try change on my git to epgdat.py https://github.com/TwolDE2/XMLTV-Import/commit/57e7b5cb733ce3b8bb679cb6d8762934039fd179

[thawtes]

thanks for your patience!

new LOG :

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:'], (1649451600, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649451600, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649451600
type(d[0]) <class 'int'>
self.longDescUntil 1649884937.8729842
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649451600, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649451600 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: can't concat str to bytes
data (['1:0:1:CA02:A926:42E:1A40000:0:0:0:'], (1649494800, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:1:CA02:A926:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649494800, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649494800
type(d[0]) <class 'int'>
self.longDescUntil 1649884937.8729842
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649494800, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649494800 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] s :  MyProgramName
[EPGImport][doThreadRead] ### importEvents exception: can't concat str to bytes
[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:1:CA02:A926:42E:1A40000:0:0:0:
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[EPGImport][doThreadRead] ### importEvents exception: can't concat str to bytes

note: /media/hdd/epg_new.dat file still cannot be created.

Could it be a problem with the epgdat_importer.py file?

thx

[Huevos]

First we need to get rid of the errors. Then it should work.

[TwolDE]

So import now works for C++ Talked with Huevos and as a result try change on my git to epgdat.py https://github.com/TwolDE2/XMLTV-Import/commit/57e7b5cb733ce3b8bb679cb6d8762934039fd179

[TwolDE]

so new update to epgdat - changes variable name "s" to sd & ld to avoid confusion on debug. also changes in short & long description routines so that hopefully we get binary string(s) in both routines (issue was string added to bytes) ... this is python 3 only, need to modify to support both https://github.com/TwolDE2/XMLTV-Import/commit/af6f69f3ffe4977359dad5c735fe65a9af5975b8

[thawtes]

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'

[XMLTVConverter] Enumerating event information

data (['1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649538000
type(d[0]) <class 'int'>
self.longDescUntil 1649995104.6941166
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649538000 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] sd :  MyProgramName
[epgdat][short_desc] res :  b'eng\x0e\x15MyProgramName\x00'
[epgdat][long_desc] ld :  MyProgramDesc
[epgdat][long_desc] num_tot_desc :  1

data (['1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649581200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649581200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649581200
type(d[0]) <class 'int'>
self.longDescUntil 1649995104.6941166
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649581200, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649581200 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] sd :  MyProgramName
[epgdat][short_desc] res :  b'eng\x0e\x15MyProgramName\x00'
[epgdat][long_desc] ld :  MyProgramDesc
[epgdat][long_desc] num_tot_desc :  1

[EPGImport][doThreadRead] ### thread is ready ### Events: 2
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] event :  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])
[epgdat][preprocess_events_channel] packing data for event:  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])

[EPGImport] Failure in epg_done
[EPGImport] imported 2 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

[EPGImport] Failure in epg_done > this error msg in epgdat_importer.py

[TwolDE]

so better, 90% there? ... hopefully added some more corrections, debug on final packs - all in epgdat.py https://github.com/TwolDE2/XMLTV-Import/commit/f7bc48157d0c1c08fb269293a2b5f027e272e2a4

[thawtes]

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] using C module, yay
[EPGImport] nextImport, source= MY_CHANNELS
[EPGImport] afterDownload /etc/epgimport/myServices.xml
[EPGImport] afterChannelDownload None
[EPGImport] Using twisted thread
[EPGImport] Parsing channels from '/etc/epgimport/rytec.channels.xml'
[XMLTVConverter] Enumerating event information
data (['1:0:16:CB24:A02D:42E:1A40000:0:0:0:'], (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0))
type(data) <class 'tuple'>
r ['1:0:16:CB24:A02D:42E:1A40000:0:0:0:']
type(r) <class 'list'>
d (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
type(d) <class 'tuple'>
d[0] 1649538000
type(d[0]) <class 'int'>
self.longDescUntil 1650044574.933827
type(self.longDescUntil) <class 'float'>
d[0] > self.longDescUntil False
[epgdat_importer][epgdatclass] program :  (1649538000, 43200, 'MyProgramName', '', 'MyProgramDesc', 0)
[epgdat][add_event]add event:- starttime, duration, title, description 1649538000 43200 MyProgramName MyProgramDesc
[epgdat][short_desc] sd :  MyProgramName
[epgdat][short_desc] res :  b'eng\x0e\x15MyProgramName\x00'
[epgdat][long_desc] ld :  MyProgramDesc
[epgdat][long_desc] num_tot_desc :  1
[EPGImport][doThreadRead] ### thread is ready ### Events: 1
[epgdat][preprocess_events_channel] service :  1:0:16:CB24:A02D:42E:1A40000:0:0:0:
[epgdat][preprocess_events_channel] event :  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])
[epgdat][preprocess_events_channel] packing data for event:  (1649538000, 43200, (1729232839, b'eng\x0e\x15MyProgramName\x00'), [(2728014673, b'\x00eng\x00\x00\x0e\x15MyProgramDesc')])
[epgdat][preprocess_events_channel] through pack_1 2nd  b'\x00\x01\xe9\x1e'
[epgdat][preprocess_events_channel] through pack_2 b'!\x00\x00'
[epgdat][preprocess_events_channel] through pack_3 b'\x12\x00\x00'
[epgdat][preprocess_events_channel] through pack_4 b'\xc7\xff\x11g'
[epgdat][preprocess_events_channel] through pack_4 - 2nd b'\xc7\xff\x11gQ3\x9a\xa2'
[epgdat][final_process] ENTRY : 
[epgdat][final_process] through pack_1 b'2Tv\x98ENIGMA_EPG_V8\x01\x00\x00\x00'
[epgdat][final_process] through pack_1 - 2nd b'\x02\x00\x00\x00'
[epgdat][final_process] through final pack_1 pack_2 b'\xc7\xff\x11g\x01\x00\x00\x00' [b'M\x13eng\x0e\x15MyProgramName\x00', 1]
[epgdat][final_process] through final pack_1 pack_2 b'Q3\x9a\xa2\x01\x00\x00\x00' [b'N\x15\x00eng\x00\x00\x0e\x15MyProgramDesc', 1]
[EPGImport] imported 1 events
[EPGImport] no Oudeis patch, load(/media/hdd/epg_new.dat) required
Done, data is in /media/hdd/epg_new.dat
[EPGImport] #### Finished ####

**that's it :) it works now. @TwolDE @Huevos thank you so much

now created /media/hdd/epg_new.dat**

[TwolDE2]

Great - will strip out debug and issue pull after some cleanup Also need to see if we can get the C++ routine automatically added into the epgimport directory have to look at ipk build

[thawtes]

@TwolDE @TwolDE2 @Huevos

there is a problem.

/media/hdd/epg_new.dat file is created (after change to epg.dat) but data is not visible in EPG list +infobar EPG

When I restart the device, the epg.dat file remains as epg.dat.loading and enigma2 does not start, the system becomes unusable.

maybe this LOG can be useful start enigma2 with debug log

[eEPGCache] time updated.. start EPG Mainloop
[eEPGCache] /media/hdd/epg.dat not found, try /epg.dat
[eEPGCache] /epg.dat not found, giving up
[eEPGTransponderDataReader] start Mainloop
[eEPGChannelData] next update in 2 sec
[SEC] set dynamic current limiting
[eDVBServicePlay] timeshift
[eEPGChannelData] start reading events(1649710479)
[eDVBSectionReader] DMX_SET_FILTER pid=18
[eDVBSectionReader] DMX_SET_FILTER pid=18
[eDVBSectionReader] DMX_SET_FILTER pid=18
[eEPGChannelData] abort non avail nownext reading
[eEPGChannelData] abort non avail schedule reading
[eEPGChannelData] abort non avail schedule other reading

thanks

[TwolDE]

so having stopped the image, you have renamed the old dat file and copied /media/hdd/epg_new.dat to /media/hdd/epg.dat and then restarted the image?

[TwolDE]

... you could also rename/disable the C++ routine and see if you have same issue with the embedded software routine in epgdat.py when you copy dat file

[thawtes]

so having stopped the image, you have renamed the old dat file and copied /media/hdd/epg_new.dat to /media/hdd/epg.dat and then restarted the image?

yes i know, i did it this way

[thawtes]

... you could also rename/disable the C++ routine and see if you have same issue with the embedded software routine in epgdat.py when you copy dat file

i didn't fully understand it

[TwolDE2]

If the C++ routine (dreamcrc) doesn‘t load, then it uses a “ software simulation“ in python in epgdat.py. I just wondered if that might work better (although probably slower). So perhaps rename dreamcrc to something else, so that the import fails and then it will use the python code instead.

the fact that epg.dat doesn‘t load implies the data is not correct, something I experienced initially with Crossepg

[thawtes]

ok, i understand

without dreamcrc :

[EPGImport][beginImport] oudeis patch not detected, using epg.dat instead.
[EPGImport][beginImport] using epgdat_importer.epgdatclass.
[EPGImport] failed to load C implementation, sorry
....
[EPGImport][doThreadRead] ### importEvents exception: ord() expected string of length 1, but int found
....

/media/hdd/epg_new.dat file is not created

with dreamcrc https://github.com/oe-alliance/XMLTV-Import/issues/64#issuecomment-1095554289

Could the problem be due to the characters being this way (b'eng\x0c\x15 ....etc)?

[TwolDE]

@thawtes So I have cleaned up a bit more and everything now runs clean on both C++ crc module or via ,py version. But epg.dat is corrupt on loading. So thought I would try python2 version. runs and produces same number of records. But epg.dat is corrupt on loading.

So I cannot get either py2 or py3 Ofline_importer versions to produce a valid epg.dat I am using rytec UK files as import source and running on OpenViX with V8 epgcache (py2 & py3).

[TwolDE]

@thawtes - fixed will upload tomorrrow

[AbuBaniaz]

@thawtes Is it OK to close this issue? I have been advised it has been fixed.

[thawtes]

@thawtes Is it OK to close this issue? I have been advised it has been fixed.

I will check and let you know