Access to favorites not working on python2 due to some unicode character

[pitsi]

Prerequisites

• [X] I have searched the issue tracker to avoid opening a duplicate issue.
• [X] I have read the documentation on how to create a debug log file.

Description

Today I noticed that the "favorite videos" entry of the main menu is not working. The error it produces is mentioned on the first two lines of the log. Then I tried the "online favorites" option of chaturbate, which popped up the error in the remaining lines. The offending unicode character on this log is refered to as "kiss mark", but in my previous attempt it was "black heart". Unfortunately, I can't retrieve anything from that log, because I have launched kodi more than twice since and even kodi.old.log shows "kiss mark" is to blame.

I am 99.9% sure that ALL the entries in my favorites come from chaturbate, so such a character is almost impossible to exist in a name. Moreover, the favorites were imported straight from uwc when the import function was added to the addon and, although I have not opened them for months now, I am sure I had opened them in the past.

p.s. The pasted log will be deleted once the issue is resolved.

Kodi version

18.9

Cumination version

1.1.110

Operating System and version

Libreelec 9.2.6 x64

URL to debug log file

https://notebin.de/?f8cab329a8375f0b#CRVC7fg9tg3sGN3VrMNw1M1fTiyp3BTz3vKrJWSzgmqE

[Gujal00]

looks like that particular favourite is using unicode encoding instead of utf-8 encoding and failing in Py2.7 due to that so the favorite seems to have been stored in the favorites.db incorrectly as unicode encoded rather than utf-8 encoded by UWC and the issue got carried over to Cumination as you imported from UWC

[pitsi]

I agree with your suggestion, but why wasn't there a problem before? As I said, I am 99.9% certain that all the favorites in there are ONLY chaturbate usernames, so such a character should not exist between them. I am also certain that both those menu entries have worked properly in the past.

Today, the error comes up with a different unicode character, u+1f607, which is "smiley with a halo". I can grab the .kodi/userdata/addon_data/plugin.video.cumination/favorites.db file, open it with sqlite browser and check, but it will contain hundreds of lines because I really have hundreds of bookmarks in it. Also, I know that this is the file which is being parsed by the "online favorites" option of chaturbate and not the one kodi itself uses (.kodi/userdata/favourites.xml).

Can I delete the paste now?

[pitsi]

This happened just now. The error is the same, but no character is mentioned this time

2024-06-20 14:23:48.504 T:139679625316096   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.UnicodeEncodeError'>
                                            Error Contents: 'ascii' codec can't encode characters in position 17-18: ordinal not in range(128)
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 177, in <module>
                                                sys.exit(main())
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 170, in main
                                                process_queries(argv)
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 161, in process_queries
                                                url_dispatcher.dispatch(mode, queries)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/url_dispatcher.py", line 130, in dispatch
                                                cls.func_registry[mode](*args, **kwargs)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/sites/chaturbate.py", line 337, in onlineFav
                                                + "[COLOR deeppink]Watching: [/COLOR]" + str(model["num_users"]) + " viewers"
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/utils.py", line 854, in cleantext
                                                text = h.unescape(text.decode('utf8')).encode('utf8')
                                              File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
                                            UnicodeEncodeError: 'ascii' codec can't encode characters in position 17-18: ordinal not in range(128)
                                            -->End of Python script error report<--

[pitsi]

Today's... "lucky utf character" is "bitting lip", with utf code 1fae6. I know I can remove the db file and start over, but I have like hundreds of entries in there. I can verify that such a move works because I backed up the file somewhere and on next launch the addon opened the menu as usual.

Also, can I remove the paste now and add it as a code block?

[12asdfg12]

You can try to export the favorites in uncompressed mode, open the file in Notepad++, and convert it to UTF8 format. Then delete the favorites from cumination and reload the modified backup. If the problem is not solved, you can edit the backup file and delete/modify the names (the file can be formatted with the JSTool plugin from Notepad++).

[pitsi]

First things first. I checked the addon's code and the file in question is favorites.db which is an sqlite3 database. Proof

$ file favorites.db 
favorites.db: SQLite 3.x database, last written using SQLite version 3028000, page size 1024, file counter 3688, database pages 632, cookie 0x8, schema 4, UTF-8, version-valid-for 3688

How do I export it in uncompressed mode? I am on linux, but I do have windows on the side and notepad++ installed there.

I opened it with sqlitebrowser and it has ~3500 entries, ALL of which are from chaturbate and NONE containing a problematic character. Basically it is this set of columns: rowid - name - url - mode - image - duration - quality - domain and a) the name column is the "username" each cam has b) the url column is always like so: https://chaturbate.com/username c) the mode column is either 222 orchaturbate.Playvid. I think the ones with 222 are the ones which were imported from uwc d) the image column is either https://roomimg.stream.highwebmedia.com/riw/username.jpg?timestamp or https://thumb.live.mmcdn.com/riw/username.jpg?timestamp e) the domain column is always chaturbate.com

[12asdfg12]

Addon setting/ Favorites and keywords/ uncheck Compress backup files then Backup favorites

[pitsi]

Ok I just did it. The "compress backup" option was already unchecked. It made a json file, with .bak extention and ~3500 entries that look like so

    {
      "name": "username [COLOR deeppink][age][/COLOR]public",
      "img": "(whatever the image column above had)",
      "url": "(whatever the url column had)",
      "mode": (whatever the mode column had),
      "duration": null,
      "quality": null
    },

It is basically an export of the sqlite3 .db file in json. And NO weird characters anywhere. All those entries are under a section named data (there is also a section named meta above it).

[pitsi]

I just noticed that you said above to export the favorites and then delete them. I did it via the addon's settings and when I tried importing the backed up file I got this, which seems worse.

2024-06-21 18:19:25.926 T:140619140937472   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.AttributeError'>
                                            Error Contents: 'int' object has no attribute 'startswith'
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 177, in <module>
                                                sys.exit(main())
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 170, in main
                                                process_queries(argv)
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 161, in process_queries
                                                url_dispatcher.dispatch(mode, queries)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/url_dispatcher.py", line 130, in dispatch
                                                cls.func_registry[mode](*args, **kwargs)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/favorites.py", line 503, in restore_fav
                                                elif favorite["mode"].startswith('custom_') and favorite["mode"].split('.')[0] not in custom_sites:
                                            AttributeError: 'int' object has no attribute 'startswith'
                                            -->End of Python script error report<--

[12asdfg12]

I tested with Kodi 18.9 and backup/delete/restore worked fine. The problem is somewhere in your file. If you want to share your favorites.db file or the backup file, I can take a look.

[pitsi]

If I can find a way to clear ALL the usernames from all the fields they are mentioned, I will.

[pitsi]

Here is the backup file with all the usernames removed and in a human readable json format. cumination-favorites_2024-06-21_18-00-53.json

Moreover, in order to prove that none of the files has any special characters, file says that they are just ascii text.

$ file -k cumination-favorites_2024-06-21_18-00-53.*
cumination-favorites_2024-06-21_18-00-53.bak:  JSON text data\012- , ASCII text, with very long lines (65536), with no line terminators
cumination-favorites_2024-06-21_18-00-53.json: JSON text data\012- , ASCII text

p.s. I will be afk the following days.

[12asdfg12]

The import error is caused by the "mode": 222, lines. "mode" was a number in UWC but in Cumination it is a string (in the case of Chaturbate it is: "chaturbate.Playvid").

You can try to replace the lines in the json file: "modes": 222, with "mode": "chaturbate.Playvid", And then do the import.

However, this does not explain the Unicode errors. These problems were caused because you copied favorites.db from UWC to Cumination, you had to export the favorites in UWC and import them in Cumination.

[pitsi]

I will try it and report back tomorrow. I can delete all uwc entries if needed... provided that I won't break it more.

As for the copied favorites, you are right. I had not explored all of the addons options back then.

[pitsi]

That did not go as planned. Changing all "222" instances made geany and nano freeze my entire system, probably because the .bak file is one line long with thousands of characters. However, both geany and nano changed all "222" instances (and the null values to "" for the duration and quality fields) with no issues at all at the human readable json, but that one can not be imported to the addon.

---edit Sd (sed for noobs) did change all "222" instances with no issue, but the addon still fails to import it.

2024-06-27 18:46:47.693 T:140385047643904   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.TypeError'>
                                            Error Contents: 'NoneType' object is not callable
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 177, in <module>
                                                sys.exit(main())
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 170, in main
                                                process_queries(argv)
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 161, in process_queries
                                                url_dispatcher.dispatch(mode, queries)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/url_dispatcher.py", line 130, in dispatch
                                                cls.func_registry[mode](*args, **kwargs)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/favorites.py", line 476, in restore_fav
                                                backup_content = json.load(fav_file)
                                              File "/usr/lib/python2.7/json/__init__.py", line 291, in load
                                              File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
                                              File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
                                              File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
                                              File "/usr/lib/python2.7/json/decoder.py", line 36, in errmsg
                                            TypeError: 'NoneType' object is not callable

[12asdfg12]

It doesn't matter if the JSON file is formatted or not. If you have errors when loading, first of all, check if the JSON file is valid.

[pitsi]

How do I check if the file is valid? Please don't tell me to run it through a linter! It has thousands of lines and my pc will probably freeze while processing it! Truth be told, after all that mess, I'd rather delete the file and start over.

[12asdfg12]

I use Notepad++ with the JSON Viewer plugin. Formatting works instantly, even with large files, and highlights any errors present in the file.

[pitsi]

As it seems, blindly replacing any instance of "222" made a huge mess. First of all, 222 was part of many timestamps, so adding a text string... with quotes in it... you get the idea. Second, on some occasions, the mode variable was changed to ""chaturbate.Playvid"" (yes, these are 2 double quotes). Third, I deleted the formatted json and tried to create a new one with jq, but it fails to parse the text on character 320! Note that this happens on the original backed up file. I will make a new backup as it seems.

$ cat cumination-favorites_2024-06-21_18-00-53.bak | jq .
jq: parse error: Invalid numeric literal at line 1, column 320

Is there a way, like an sqlite3 or jq command, to delete ALL entries which are there got from uwc? For instance, delete all entries which have 222 in the mode field. If that does not work, I have no other choice but to delete everything and start over.

[12asdfg12]

You can do that with sqlite3. Open favorites in sqlite3: sqlite3 favorites.db Run the command: delete from favorites where mode = 222;

You can list the values ​​from the mode column with the command: select distinct mode from favorites;

[pitsi]

Almost there! I can now enter the favorites submenu from the main menu, but when selecting the relevant option within chaturbate, I now get this

2024-06-29 12:28:47.042 T:139721068164864   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.UnicodeEncodeError'>
                                            Error Contents: 'ascii' codec can't encode character u'\U0001f3e0' in position 8: ordinal not in range(128)
                                            Traceback (most recent call last):
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 177, in <module>
                                                sys.exit(main())
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 170, in main
                                                process_queries(argv)
                                              File "/storage/.kodi/addons/plugin.video.cumination/default.py", line 161, in process_queries
                                                url_dispatcher.dispatch(mode, queries)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/url_dispatcher.py", line 130, in dispatch
                                                cls.func_registry[mode](*args, **kwargs)
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/sites/chaturbate.py", line 337, in onlineFav
                                                + "[COLOR deeppink]Watching: [/COLOR]" + str(model["num_users"]) + " viewers"
                                              File "/storage/.kodi/addons/plugin.video.cumination/resources/lib/utils.py", line 854, in cleantext
                                                text = h.unescape(text.decode('utf8')).encode('utf8')
                                              File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
                                            UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f3e0' in position 8: ordinal not in range(128)
                                            -->End of Python script error report<--

The character in question is described as "house building" and the exported backup file is ascii only.

# file -k cumination-favorites_2024-06-29_12-32-06.bak
cumination-favorites_2024-06-29_12-32-06.bak: JSON data\012- , ASCII text, with very long lines, with no line terminators\012-

[12asdfg12]

Edit the chaturbate.py file from Cumination and comment out these lines: 335, 336, 337. It should look like this:

            # subject = utils.cleantext(subject.split(' #')[0]) + "[CR][CR][COLOR deeppink]Location: [/COLOR]" + utils.cleantext(model["location"]) + "[CR]" \
            #     + "[COLOR deeppink]Duration: [/COLOR]" + str(round(model["seconds_online"] / 3600, 1)) + " hrs[CR]" \
            #     + "[COLOR deeppink]Watching: [/COLOR]" + str(model["num_users"]) + " viewers"

BTW, why are you still using Kodi 18 and not upgrading to the latest version?

[pitsi]

It worked! Thank you very much! My python knowledge is as elementary as my french, but I can understand some of those lines there, so I guess some parameter/value pulled by chaturbate's api broke it.

As for kodi. I do not use "just" kodi, I use libreelec which is a "just enough os" with kodi as its (desktop) enviroment, so upgrading kodi means upgrading libreelec too, and that is the major issue. In short:

Libreelec 10 (kodi 19) was picking up the wrong driver for my vga, so xorg failed to start, and in turn kodi. I reported it and it was fixed after a few days. However, kodi 19 gives me visible framedrops even on videos as low as 360p, while the cpu is not stressed at all. Libreelec 11 (kodi 20) had a major issue with wireless, on all platforms, because they decided to change the backend from wpa_supplicant to iwd. Let's say that iwd and connman did not like each other, which means that a lot of reboots were required until they worked properly, scanned for networks and connected to the needed one. Since network connectivity is fundamental for playing streams, I did not test it any further for framedrops. Libreelec 12 (kodi 21) is said to have fixed the forementioned wireless issue, but it is still present on my end, so I can't test any streams again.

Long story short, I am stuck on libreelc 9.2, kodi 18 and python2 with its shortcomings, because it has worked with zero issues since mid 2019.

I test every major libreelec version the moment its beta is released and I tested libreelec 11 with ALL its nightlies during the summer of 2022 (late May to early September) on my rpi3b. I almost damaged its poor sd card with the daily updates (~100 in total), none of which fixed the wireless issue, so I gave up on it completely for the rpi3b.

Libreelec is also the only distro that has an x64 version, because all others like coreelec and osmc are for arm devices only, so I have nothing else to try. I tried creating my own setup that mimics libreelec's functionality with alpine a few years ago, but it was way too hard to maintain.

All libreelec (and alpine) installations are done on a quality sandisk usb stick and use persistency (= use the usb stick as storage or run mode for libreelec). Thank you for reading!

[pitsi]

I just got the update to v1.1.111 and I checked if it still works. Then I saw the commit which fixes the issue. As it seems, the location variable, returned from the api itself, was the one that could contain non-ascii characters and break the addon. Thank you for fixing it!