autobouquetsmaker on nextp3 branch

[LandRocket]

Hope this is the right place to post this...... apologies if not.

On my own self-compile, when trying to run AutoBouquetsMaker, is crashing.

Test box is VU Ultimo 4K, attempted with compiled images of ATV and OBH.

Obviously the crashlog tells me it's something to do with the writing of the lamedb but I'm not quite proficient enough in python to attempt to fix :(

Crashlog shows:

07:24:32.8758 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-main][checkTunerLock] ACQUIRING TSID/ONID 07:24:32.9782 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][read] Reading cable_uk_virgin (hd_40971_dvbc_uk)... 07:24:32.9784 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Tools][customtransponder] Transponder provider name cable_uk_virgin 07:24:33.0167 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Adapter 0 07:24:33.0169 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Demuxer 0 07:24:33.0170 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Frontend 0 07:24:33.0171 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] DVBType dvbc 07:24:33.0172 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] NIT pid: 0x10 07:24:33.0173 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] NIT current table id: 0x40 07:24:33.0174 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] NIT other table id: 0x41 07:24:33.0175 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Ignore visible service flag: 0 07:24:33.0176 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] SDT pid: 0x11 07:24:33.0177 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] SDT current table id: 0x42 07:24:33.0178 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] SDT other table id: 0x46 07:24:33.0179 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Reading transponders... 07:24:36.8903 [ ] gdi/grc.cpp:211 thread [gRC] main thread is non-idle! display spinner! 07:24:40.7579 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Scan complete, netid: 40971 07:24:40.7643 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Added/Updated 48 transponders with network_id = 0xa00b 07:24:40.7645 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Reading services (LCN)... 07:24:44.8054 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Read 462 services 07:24:45.4386 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] 427 valid services 07:24:45.4396 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][read] cable_uk_virgin, Done 07:24:45.5428 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Saving... 07:24:45.5469 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Bouquets to hide: {} 07:24:45.5470 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] TV bouquets to keep: [] 07:24:45.5471 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Radio bouquets to keep: [] 07:24:45.5472 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Add provider prefix to bouqets: False 07:24:45.5473 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-BouquetsWriter] Writing lamedb... 07:24:46.2043 { D } Traceback (most recent call last): 07:24:46.2044 { D } File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/main.py", line 517, in doBuildIndex 07:24:46.2046 { D } File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/manager.py", line 102, in save 07:24:46.2048 { D } File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/bouquetswriter.py", line 145, in writeLamedb 07:24:46.2050 { D } File "/usr/lib/python2.7/codecs.py", line 706, in write 07:24:46.2069 { D } return self.writer.write(data) 07:24:46.2070 { D } File "/usr/lib/python2.7/codecs.py", line 369, in write 07:24:46.2073 { D } data, consumed = self.encode(object, self.errors) 07:24:46.2074 { D } UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 33626: ordinal not in range(128) 07:24:46.2074 [ E ] python/python.cpp:209 call [ePyObject] (PyObject_CallObject(<bound method AutoBouquetsMaker.doBuildIndex of <class 'Plugins.SystemPlugins.AutoBouquetsMaker.scanner.main.AutoBouquetsMaker'>>,()) failed)

[nickersk]

what you get without reloading sys?

[atvcaptain]

root@mutant51:~# python Python 2.7.14 (default, Jun 20 2018, 22:11:25) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sys sys.getdefaultencoding() 'utf-8'

[nickersk]

why the hell i always get ascii after reboot? what LC_ setting do you use on build machine? looks so, it catches encoding setting from build machine env or what...will do clean build. thx anyhow...

[atvcaptain]

have you see the sitecustomize.py you found here: /usr/lib/python2.7 and not /usr/lib/python2.7/site-packages

[atvcaptain]

looks new version change search order and search first here: /usr/lib/python2.7 and ignore the version /usr/lib/python2.7/site-packages

old version search first in /usr/lib/python2.7/site-packages and ignore the file in /usr/lib/python2.7

[nickersk]

yep, you're right. i thought we have to deal with this: https://github.com/openembedded/openembedded-core/commit/6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280#diff-0d0b8f6e8239d68c95dd8cc4227c96c3

look at this sitecustomize.py: -def __enableDefaultEncoding():

• import sys
• try:
• sys.setdefaultencoding( "utf8" )
• except LookupError:
• pass

[nickersk]

everything ok after image built from scratch.

[AbuBaniaz]

Tried AutoBouquetsMaker with an image built overnight, worked fine too.

[LandRocket]

Just thought I'd chip in and say, same here! :) Sat and Cable working fine together again! Appreciate all the hard work you all put in. Thank you again..... 👍

[nickersk]

i am really glad i could help to make us all happier :-) and as the icing on the cake - i am testing gcc 8 image on my vusolose v2 and it seems to run flawlessly 👍 we still have to do something with samba not working ... on my box at least

[LandRocket]

Recompiled today, and this issue seems to have popped it's head back up, but of course the LOCALE errors are gone from the console.... just wanted to check this is known about....

[arn354]

Did you trash your build folder?

[LandRocket]

I believe I rm'd and re-cloned 2 days ago, but you've got me thinking now.....

[LandRocket]

Still crashing on fresh clone of git today (20180627)

enigma2_crash_1530106530.log Enigma2-debug-20180627_14-33-51.log Enigma2-debug-20180627_14-35-42.log

[nickersk]

i smell build env LC isue...what LC do you use on build machine?

[LandRocket]

Definitely haven't changed the build environment since the working copy but, on the buildserver:

LANG=en_US.UTF-8 LANGUAGE= LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=

Admittedly this is different from the STB locale: LC_ALL= LC_MEASUREMENT=en_GB.UTF-8 LC_PAPER=en_GB.UTF-8 LC_MONETARY=en_GB.UTF-8 LC_NAME=en_GB.UTF-8 LC_COLLATE=C.UTF-8 LC_CTYPE=C.UTF-8 LC_ADDRESS=en_GB.UTF-8 LC_NUMERIC=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 LC_TELEPHONE=en_GB.UTF-8 LC_IDENTIFICATION=en_GB.UTF-8 LC_DATE=en_GB.UTF-8 LC_TIME=en_GB.UTF-8

[nickersk]

this looks ok. what you get from stb: python import sys sys.getdefaultencoding()

?

[LandRocket]

And there it is... :(

import sys sys.getdefaultencoding() 'ascii'

[atvcaptain]

can you test us its same ?

[LandRocket]

after changing lang to US/restart, on STB: LC_ALL= LC_MEASUREMENT=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_COLLATE=C.UTF-8 LC_CTYPE=C.UTF-8 LC_ADDRESS=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_DATE=en_US.UTF-8 LC_TIME=en_US.UTF-8

Running ABM, still crashing the same...

sys.getdefaultencoding() still returns 'ascii'

[betacentauri]

Please check the sitecustomize.py. Is it the OE one which don’t set the default encoding?

[nickersk]

python ignores it i think, maybe try to patch site.py ? https://stackoverflow.com/questions/36623471/persist-utf-8-as-default-encoding else we have to do some kind of reload(sys) in sitecustomize.py which is not the correct way IMHO.

[LandRocket]

Agreed.. python appears to be ignoring meta-oe-alliance/meta-oe/recipes-devtools/python/python/sitecustomize.py or I certainly can't see it having any effect on my build attempts.

Grabbing the bouquetswriter.py from @Huevos (that sets UTF-8 in the plugin) gets ABM working again.

[nickersk]

just created PR for testing...i would leave python to decide for utf8 for himself based on env :-) this is one of the ways how this should be done. rebuilding now, if this will be not enough, i will hardcode replace ascii with utf-8

[betacentauri]

Ah, there is a problem that the last changes move the sitecustomize.py to the python-core-src package. And build don't create sitecustomize.pyo. So sitecustomize.pyo is missing in the image. I hope will be fixed soon.

[betacentauri]

This should fix the problem: https://github.com/oe-alliance/oe-alliance-core/commit/ba9fda003b39fbf493a2f1c6ffbd7fea0fc68d0e

[LandRocket]

Again, appreciated.... can't test ATM but I have faith 👍