autobouquetsmaker on nextp3 branch

[LandRocket]

Hope this is the right place to post this...... apologies if not.

On my own self-compile, when trying to run AutoBouquetsMaker, is crashing.

Test box is VU Ultimo 4K, attempted with compiled images of ATV and OBH.

Obviously the crashlog tells me it's something to do with the writing of the lamedb but I'm not quite proficient enough in python to attempt to fix :(

Crashlog shows:

07:24:32.8758 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-main][checkTunerLock] ACQUIRING TSID/ONID 07:24:32.9782 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][read] Reading cable_uk_virgin (hd_40971_dvbc_uk)... 07:24:32.9784 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Tools][customtransponder] Transponder provider name cable_uk_virgin 07:24:33.0167 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Adapter 0 07:24:33.0169 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Demuxer 0 07:24:33.0170 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Frontend 0 07:24:33.0171 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] DVBType dvbc 07:24:33.0172 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] NIT pid: 0x10 07:24:33.0173 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] NIT current table id: 0x40 07:24:33.0174 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] NIT other table id: 0x41 07:24:33.0175 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Ignore visible service flag: 0 07:24:33.0176 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] SDT pid: 0x11 07:24:33.0177 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] SDT current table id: 0x42 07:24:33.0178 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] SDT other table id: 0x46 07:24:33.0179 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Reading transponders... 07:24:36.8903 [ ] gdi/grc.cpp:211 thread [gRC] main thread is non-idle! display spinner! 07:24:40.7579 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Scan complete, netid: 40971 07:24:40.7643 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Added/Updated 48 transponders with network_id = 0xa00b 07:24:40.7645 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Reading services (LCN)... 07:24:44.8054 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] Read 462 services 07:24:45.4386 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-DvbScanner] 427 valid services 07:24:45.4396 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][read] cable_uk_virgin, Done 07:24:45.5428 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Saving... 07:24:45.5469 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Bouquets to hide: {} 07:24:45.5470 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] TV bouquets to keep: [] 07:24:45.5471 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Radio bouquets to keep: [] 07:24:45.5472 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-Manager][save] Add provider prefix to bouqets: False 07:24:45.5473 { } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-BouquetsWriter] Writing lamedb... 07:24:46.2043 { D } Traceback (most recent call last): 07:24:46.2044 { D } File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/main.py", line 517, in doBuildIndex 07:24:46.2046 { D } File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/manager.py", line 102, in save 07:24:46.2048 { D } File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/bouquetswriter.py", line 145, in writeLamedb 07:24:46.2050 { D } File "/usr/lib/python2.7/codecs.py", line 706, in write 07:24:46.2069 { D } return self.writer.write(data) 07:24:46.2070 { D } File "/usr/lib/python2.7/codecs.py", line 369, in write 07:24:46.2073 { D } data, consumed = self.encode(object, self.errors) 07:24:46.2074 { D } UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 33626: ordinal not in range(128) 07:24:46.2074 [ E ] python/python.cpp:209 call [ePyObject] (PyObject_CallObject(<bound method AutoBouquetsMaker.doBuildIndex of <class 'Plugins.SystemPlugins.AutoBouquetsMaker.scanner.main.AutoBouquetsMaker'>>,()) failed)

[nickersk]

I think this may have something to do with console/terminal/shell locale.

[LandRocket]

Interesting you should say that... there is a message on console when logging in....

-bash: warning: setlocale: LC_TIME: cannot change locale (C.UTF-8): No such file or directory -bash: warning: setlocale: LC_TIME: cannot change locale (en_GB.UTF-8): No such file or directory

[atvcaptain]

you can search and fix the issue, at the moment i have limit time, many todo at work

[atvcaptain]

and we have one other big issue in nextp3 when rebuild a image 2-3-4 times you get strange issue like this

ERROR: enigma2-plugin-extensions-openwebif-1.3.0+gitAUTOINC+007c9bb527-r0 do_package_write_ipk: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: 0001: 0002:do_package_ipk(d) 0003: File: '/home/oe1/atv63/build-enviroment/openembedded-core/meta/classes/package_ipk.bbclass', lineno: 87, function: do_package_ipk 0083: 0084: os.chdir(oldcwd) 0085: 0086: if error: 0087: raise error 0088:} 0089:do_package_ipk[vardeps] += "ipk_write_pkg" 0090:do_package_ipk[vardepsexclude] = "BB_NUMBER_THREADS" 0091: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/oe1/atv63/build-enviroment/builds/openatv/release/h9/tmp/work/all-oe-linux-gnueabi/enigma2-plugin-extensions-openwebif/1.3.0+gitAUTOINC+007c9bb527-r0/packages-split/enigma2-plugin-extensions-openwebif'

sumo dont extract the source again and i have no idea, oe-a bug or sumo bug ?

[Schimmelreiter]

07:24:46.2074 { D } UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 33626: ordinal not in range(128) is unlikely to be related to the console glitch -bash: warning: setlocale: LC_TIME: cannot change locale (en_GB.UTF-8): No such file or directory

It simply means that there are non-ASCII characters inside a string that is going to be converted from ASCII to Unicode. ASCII is standardized for the first 128 characters (0-127), 128 is outside of that range.

The glitch is pretty interesting howerver, in so far as there are only complaints about LC_TIME, not LC_DATE and so on. It can probably be fixed by simply taking the corresponding files from the glibc built by that branch.

[Huevos]

What provider is causing the crash? Have you deleted your lamedb and just done an ABM scan. Most likely cause of this crash is junk in lamedb before running ABM.

[AbuBaniaz]

Can you attach your lamedb file please?

[LandRocket]

Will have a look later today, reverted to the 6.2 branch so that I could see the footy yesterday.

I was able to get this to crash from a clean install (build sent to usb, flashed, wizard followed, configuring nothing more than timezone and tuners, installed plugin from feeds, configure provider/s, attempt scan)

I believe that when it does crash, it doesn't touch the lamedb at all, but I will do a total clean install again and backup the lamedb before running so I have a complete comparison....

[LandRocket]

OK, yes it does touch the lamedb... it zeros it when it crashes :( So I can't send you the lamedb post-crash

-rw-r--r-- 1 root root 0 Jun 18 06:57 lamedb -rw-r--r-- 1 root root 191449 Jun 18 06:56 lamedb.original

That's from a clean install

[Huevos]

We need a copy of lamedb for testing. Are you saying this crash doesn't happen if you loaded 6.2.

If not we need a copy of the lamedb file that was active on the receiver before the scan occurs.

Why does the lamedb have 191449 bytes in it if no scan has occured yet? What does that content correspond to?

[LandRocket]

The crash certainly doesn't happen if I build 6.2 on the same Ubuntu box.

The lamedb of 191449 bytes is what was installed once OpenATV boots when compiled from nextp3.

[LandRocket]

lamedb.zip

Attached for your perusal.....

[Huevos]

Can you post your debug log so we can see your exact ABM config.

[LandRocket]

Just to try and eliminate gremlins in the system, I've re-cloned the branch and recompiled (ATV). As before, done initial setup etc etc...

I'm enabling the "Extra Debug" in ABM, but not seeing any additional logs, just the crash log... which I upload here in case it's inside there. If I need to do anything specific/grab from a different location, please let me know.

enigma2_crash_1529331608.log

How I configured ABM.....

[AbuBaniaz]

Please attach your debug log too. This is the debug log of Enigma2.

15:20:07.8308 {   } Plugins/SystemPlugins/AutoBouquetsMaker/log.py:23 write [ABM-BouquetsWriter] Writing lamedb...
15:20:08.0142 { D } Traceback (most recent call last):
15:20:08.0142 { D }   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/main.py", line 517, in doBuildIndex
15:20:08.0144 { D }   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/manager.py", line 102, in save
15:20:08.0145 { D }   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/bouquetswriter.py", line 145, in writeLamedb
15:20:08.0147 { D }   File "/usr/lib/python2.7/codecs.py", line 706, in write
15:20:08.0152 { D }     return self.writer.write(data)
15:20:08.0153 { D }   File "/usr/lib/python2.7/codecs.py", line 369, in write
15:20:08.0156 { D }     data, consumed = self.encode(object, self.errors)
15:20:08.0157 { D } UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 33110: ordinal not in range(128)
15:20:08.0157 [ E ] python/python.cpp:209 call [ePyObject] (PyObject_CallObject(<bound method AutoBouquetsMaker.doBuildIndex of <class 'Plugins.SystemPlugins.AutoBouquetsMaker.scanner.main.AutoBouquetsMaker'>>,()) failed)

[LandRocket]

Gotya.... this'll be what you want then :)

Enigma2-debug-20180618_17-21-35.log Enigma2-debug-20180618_17-23-07.log

[AbuBaniaz]

Can you execute this command and then run ABM again? Please report back if it crashes.

init 4 && sleep 10 && rm -rf /etc/defaultsat.tar.gz && rm -rf /etc/enigma2/lamedb && rm -rf /etc/enigma2/.tv && rm -rf /etc/enigma2/.radio && rm -rf /etc/enigma2/.del && init 3

Would you mind attaching the ABM plugin that was compiled please? Sorry.

[LandRocket]

20180619_0710-ABM_debug.zip

I have included the ABM plugin that my system compiled, but the nice folks at ATV have actually got their feeds online, at first I did wonder if I'd installed an older ABM plugin by mistake...

I therefore grabbed their IPK: -rw-r--r-- 1 root root 197176 Jun 18 14:23 atv63.ipk -rw-r--r-- 1 root root 197176 Jun 18 14:23 landrocket63.ipk root@vuultimo4k:/tmp# diff atv.ipk landrocket.ipk root@vuultimo4k:/tmp#

IPKs original name for both copies was enigma2-plugin-systemplugins-autobouquetsmaker_3.0+git466+f805e3b-r0_cortexa15hf-neon-vfpv4.ipk

I did a "before" and "after" again, as I do tend to totally wipe the box between attempts, as I don't want it to suddenly start working and then not be able to reproduce what I last did that made it work.

All yours for your perusal :)

[LandRocket]

Attaching another set of logs for you... to prove I do try and read the logs lol

line 1571 onwards of Enigma2-debug-20180619_07-00-46 was reading: 07:00:48.8792 [ ] dvb/db.cpp:1038 loadBouquet [eDVBDB] loading bouquet... /etc/enigma2//bouquets.tv

And I wondered why it found any bouquets, as you specifically asked me to run a terminal command to take them out, then I realised in your terminal command:

rm -rf /etc/enigma2/.tv && rm -rf /etc/enigma2/.radio

The comment system here removes the asterisks lol

So I added the necessary asterisks (asterisk.tv, asterisk.radio) and re-ran the ABM scan. Unfortunately it didn't make any difference, but I thought I'd attach them anyway :)

enigma2_crash_1529389600.log Enigma2-debug-20180619_07-25-05.log Enigma2-debug-20180619_07-26-51.log

[atvcaptain]

python use wrong locate config, the issue comes from linux not from plugin, we must first fixt the locate issue in new branch

[LandRocket]

so the ABM issue is related to the locale issue then? I think I was getting confused since FreeSat and Sky are both located on the same satellite, FreeSat scans, but Sky doesn't....

[Huevos]

Let's wait for atvcaptain to update Linux and try again. The problem must be a character in extended ASCII i.e. 128-256. Probably one of the channel names, e.g. "RTÉ One".

[arn354]

maybe try to compile a image without "bash". openatv and more distros include bash in the image. I think something changed in bash and locales upstream

[LandRocket]

RE: Channel names and extended ASCII.. Yep I'll go with that. Makes sense, but I couldn't extract enough out of the logs to be 100% sure. If there's anything I can do within my limited ability to contribute, hope I can.

[Huevos]

@arn354, I don't understand. The error says outside range(128). The char causing the problem (in this case) is 195. Why is Python correctly using range(256) in other branches but not here?

[LandRocket]

@atvcaptain said it was down to the locales that are erroring even at shell stage. Now I'm not by any means an expert but in any language/locale - 65 = a. so lets call that the "root" ascii locales...

as soon as you start trying to produce accents/chars that are not in everyones charset, I believe they go into the "extended" set so in french 165 might be an e with a top accent, in german it might be the the B with the dots..

so if there was something in the bouquet that didn't use the standard ascii set (ie sub 128) - then if the locales were broke, then trying to put them into the database would be a cause for a crash. imagine trying to type japanese on a US keyboard... thats what the plugin is trying to do... when the locales work it can "pull" them from each locale - when you press the red button on the initial boot of ATV you get some wonderful languages I have no idea of understanding or writing, so if the root OS doesn't know about them either how the heck is it supposed to write a file with them in.....

That's my comprehension of what's happened...

[Huevos]

Locale is not relevant. It is reporting ASCII out of range(128). The char is 195. Charset of DVB is Latin1 by default and that is what ABM works in.

[Huevos]

Here is the fault...

This is from branch 4.1: vusolo4k login: root root@vusolo4k:~# python Python 2.7.13 (default, Oct 30 2017, 20:35:52) [GCC 6.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sys reload(sys) <module 'sys' (built-in)> sys.getdefaultencoding() 'utf-8'

This is from branch "nextp3": root@mutant51:~# python Python 2.7.14 (default, Jun 18 2018, 20:32:57) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sys reload(sys) <module 'sys' (built-in)> sys.getdefaultencoding() 'ascii'

So default encoding is wrong in "nextp3". Currently "ascii". Should be "utf-8".

[Huevos]

Setting this value correctly in bouquetwriter.py avoids the crash, but this should be fixed where it is broken.

bouquetwriter.py.zip

[Huevos]

https://github.com/oe-alliance/AutoBouquetsMaker/commit/c53bb406efc444593d70f8a07335af98f5769181

[LandRocket]

Awesome! Big thank you to everyone who worked on this.. I wish I was more proficient to be able to actually correct some of this stuff - but I thank you all for bearing with me :)

[atvcaptain]

i have found the issue fakelocale/minilocale was old, the package must create new with files from new glibc 2.2.7 locale

[nickersk]

glibc2.27 generated locales looks the same arent they? now i see they are not ... and its not generating c.utf-8 or am i missing something ?

[Schimmelreiter]

Exactly what I said ... C.UTF-8 is a semi-official extension to locale. It's missing in yocto and was taken from Debian. But as nothing is complaining about anything but LC_TIME, it should be ok to simply keep the C.UTF-8.

[Schimmelreiter]

Setting this value correctly in bouquetwriter.py avoids the crash, but this should be fixed where it is broken.

It's broken in Python: Python defaults to ASCII and ASCII is in fact only the lower 7 bits of 1 byte chars. Everything beyond are local codepages (CP850, CP437, ...)

The Python or base-files or so includes a hack to make Python default to UTF-8 system-wide, which is actually not recommended (but has worked for us so far anyways). Maybe the Python devs changed something so that this hack does no longer work? Or maybe the hack is simply missing now ...

[nickersk]

it will take some time, but i'll find out how to bring it back to life.

[atvcaptain]

i have change some here local too, and LC_* works now, i have see you patch, i check this too again, need some time 100% rebuild :)

[betacentauri]

Until now it seems this file fixed the problem: https://github.com/oe-alliance/oe-alliance-core/blob/nextp3/meta-oe/recipes-devtools/python/python-utf8-hack/sitecustomize.py Question is now why it don't work any longer.

[betacentauri]

Ah. The installed sitecustomize.py is not the OE-A one. It's the default openembedded one: # OpenEmbedded sitecustomize.py (C) 2002-2008 Michael 'Mickey' Lauer <mlauer@vanille-media.de>

[nickersk]

besides that it is totally not recommended to use this hack (many discussions on net), there must exist much better solution or we create ours new :-D

[atvcaptain]

yes in python-core is the wrong file

[nickersk]

we need to fix outdirs variable in oe-alliance-brandig-remotes, make errors on references itself ...on rebuild

[atvcaptain]

we have more strange rebuild issue , no idea at the moment

[nickersk]

rebuilt 6 or 7 times without errors, except oe-alliance-branding-remote (fixed temp locally) and 1 time pam-smbpass. next i will try to find way to get rid of obsolete pam-smbpass because i think we can use libpam.

[atvcaptain]

some time Collected errors:

• calculate_dependencies_for: Cannot satisfy the following dependencies for openatv-base:
• oe-alliance-base *
• opkg_solver_install: Cannot install package openatv-base. ring libpcre1.

or ERROR: enigma2-6.3+gitAUTOINC+b1d726e582-r0 do_package_write_ipk: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: 0001: 0002:do_package_ipk(d) 0003: File: '/home/oe1/atv63/build-enviroment/openembedded-core/meta/classes/package_ipk.bbclass', lineno: 87, function: do_package_ipk 0083: 0084: os.chdir(oldcwd) 0085: 0086: if error: 0087: raise error 0088:} 0089:do_package_ipk[vardeps] += "ipk_write_pkg" 0090:do_package_ipk[vardepsexclude] = "BB_NUMBER_THREADS" 0091: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/oe1/atv63/build-enviroment/builds/openatv/release/hd51/tmp/work/mutant51-oe-linux-gnueabi/enigma2/6.3+gitAUTOINC+b1d726e582-r0/packages-split/enigma2-meta'

ERROR: enigma2-6.3+gitAUTOINC+b1d726e582-r0 do_package_write_ipk: Function failed: do_package_ipk ERROR: Logfile of failure stored in: /home/oe1/atv63/build-enviroment/builds/openatv/release/hd51/tmp/work/mutant51-oe-linux-gnueabi/enigma2/6.3+gitAUTOINC+b1d726e582-r0/temp/log.do_package_write_ipk.7136 NOTE: recipe enigma2-6.3+gitAUTOINC+b1d726e582-r0: task do_package_write_ipk: Failed ERROR: Task (/home/oe1/atv63/build-enviroment/meta-oe-alliance/meta-oe/recipes-oe-alliance/enigma2/enigma2.bb:do_package_write_ipk) failed with exit code '1' NOTE: Tasks Summary: Attempted 9267 tasks of which 9066 didn't need to be rerun and 1 failed.

allways different bb no pattern to recognize

[nickersk]

really strange ...i dont face these errors anymore i think it throws errors always on different package, because it fails at packages split. ahm i forgot almost...i have remarked rm_work in site conf

[atvcaptain]

i check the folder and no files, or my hardware have big issue, i check build next days with different hardware

[nickersk]

maybe that some funny script deletes files before they get packaged by opkg or something. try disabling inherit rm_work i got powerfull feeling that last change around setscene in rm_work class messed up things.

[atvcaptain]

Welcome to openATV for mutant51 openatv 6.3 mutant51

mutant51 login: root root@mutant51:~# export declare -x EDITOR="/bin/editor" declare -x HOME="/home/root" declare -x HUSHLOGIN="FALSE" declare -x LANG="de_DE.UTF-8" declare -x LC_ADDRESS="de_DE.UTF-8" declare -x LC_ALL="" declare -x LC_COLLATE="C.UTF-8" declare -x LC_CTYPE="C.UTF-8" declare -x LC_DATE="de_DE.UTF-8" declare -x LC_IDENTIFICATION="de_DE.UTF-8" declare -x LC_MEASUREMENT="de_DE.UTF-8" declare -x LC_MESSAGES="de_DE.UTF-8" declare -x LC_MONETARY="de_DE.UTF-8" declare -x LC_NAME="de_DE.UTF-8" declare -x LC_NUMERIC="de_DE.UTF-8" declare -x LC_PAPER="de_DE.UTF-8" declare -x LC_TELEPHONE="de_DE.UTF-8" declare -x LC_TIME="de_DE.UTF-8" declare -x LOGNAME="root" declare -x MAIL="/var/spool/mail/root" declare -x OLDPWD declare -x OPIEDIR declare -x PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin" declare -x PS1="\u@\h:\w\\$ " declare -x PWD="/home/root" declare -x QPEDIR declare -x QTDIR declare -x SHELL="/bin/bash" declare -x SHLVL="1" declare -x TERM="xterm-color" declare -x TERMINFO="/etc/terminfo" declare -x USER="root" root@mutant51:~# python Python 2.7.14 (default, Jun 20 2018, 22:11:25) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sys reload(sys) <module 'sys' (built-in)> sys.getdefaultencoding() 'utf-8'

[atvcaptain]

https://github.com/oe-alliance/oe-alliance-core/commit/b48f518777e8de7a322f750240781d53da796b1c