multiple subs in one zip crashes

[pyro12]

Hello, Please allow me to post in English as I want to do my best to explain the issue I am having.

It seems that subdivx.com allows multiple subtitles to be contained in a single zip file. When my system encounters this, kodi crashes to desktop. I find in .../userdata/addon_data/service.subtitles.subdivx/tmp9wh1a0z8 'subdivx.zip' and two subtitles (espana and latinoamerica) for the same episode.

I am unsure if the issue is caused by libarchive or subdivx addon.

I am not finding the log to be very helpful:

cat kodi.log | grep libarchive
2022-09-27 10:10:44.154 T:7       DEBUG <general>: CAddonInfoBuilder::ParseXMLTypes: Binary addon found: vfs.libarchive
2022-09-27 10:10:44.245 T:7        INFO <general>: CAddonMgr::FindAddons: vfs.libarchive v19.1.0 installed
2022-09-27 10:10:44.248 T:7       DEBUG <general>: SECTION:LoadDLL(/app/lib/kodi/addons/vfs.libarchive/vfs.libarchive.so.19.1.0)
2022-09-27 10:10:44.248 T:7       DEBUG <general>: Loading: /app/lib/kodi/addons/vfs.libarchive/vfs.libarchive.so.19.1.0
2022-09-27 10:10:45.182 T:7       DEBUG <general>: LocalizeStrings: loaded 18 strings from file /app/share/kodi/addons/vfs.libarchive/resources/language/resource.language.en_gb/strings.po
2022-09-27 10:10:48.046 T:49      DEBUG <general>: ADDONS: update check: addonID = vfs.libarchive / Origin = b6a50484-93a0-4afb-a01c-8d17e059feda / Version = 19.1.0

tail kodi.log | grep subdivx
2022-09-27 10:12:27.991 T:885     DEBUG <general>: SUBDIVX - workdir (<class 'str'>): /home/pyro/.var/app/tv.kodi.Kodi/data/userdata/addon_data/service.subtitles.subdivx/tmp9wh1a0z8/
2022-09-27 10:12:27.992 T:885     DEBUG <general>: SUBDIVX - get_url: Fetching https://www.subdivx.com/X666XNjQ5NTk2X-cobra-kai-s05e09
2022-09-27 10:12:28.564 T:885     DEBUG <general>: SUBDIVX - get_url: Fetching https://www.subdivx.com/sub9/649596.rar
2022-09-27 10:12:29.165 T:885   WARNING <general>: SUBDIVX - get_url: Failed to fetch https://www.subdivx.com/sub9/649596.rar (HTTP status: 404)
2022-09-27 10:12:29.166 T:885     DEBUG <general>: SUBDIVX - get_url: Fetching https://www.subdivx.com/sub9/649596.zip
2022-09-27 10:12:29.738 T:885     DEBUG <general>: SUBDIVX - _save_subtitles: Saving subtitles to '/home/pyro/.var/app/tv.kodi.Kodi/data/userdata/addon_data/service.subtitles.subdivx/tmp9wh1a0z8/subdivx.zip'

The final line you see in the subdivx paste is the final line in the log. Kodi just vanishes after that. As best I can tell, subdivx.zip does get unzipped (the files exist), but I don't see it happening in the log. I have loglevel set to '2' in advancedsettings.xml

I have pedrochiuaua's subdivx version 0.3.7.2 and, as seen above, libarchive 19.1.0. I also have rararchive 19.1.0 installed, but I don't think that matters (the log only shows that it is found and does some update checks, and unloads it).

I do see that there are updated versions of libarchive and I'm willing to install them manually if that is the fix, but I suspect the problem lies in the subdivx addon. I'm not sure it is equipped to handle this situation.

Thank you for any guidance.

[ramiro]

Hi @pyro12 Thanks for your detailed report.

What version of Kodi are you using? On which platform?

[ramiro]

Also, @pyro12 my memory is probably failing me here, or perhaps something has changed. But weren't all the subdivx.com subtitles stored in RAR format instead of Zip? Can you confirm if that subdivx.zip file is actually a Zip one with the file(1) command or similar?

[pyro12]

I am running Debian 11 Bullseye. I have kodi 19.4.0 installed via flatpak (Debian Bullseye is currently using python 3.9.2 which was causing some issues with some addons. Hence, the flatpak which ships with its own version of python3)

I manually downloaded 649596.zip using subdivx.com. Unzipped, it includes the two subtitles I am speaking of.

$ file subdivx.zip 
subdivx.zip: Zip archive data, at least v2.0 to extract

As best I can tell, it is indeed a zip archive. It seems to me the code assumes RAR and tries to download .rar. If that fails then tries downloading the file as .zip. The ZIP is successfully downloaded and unzipped, but I think that is the last thing that happens before kodi crashes. In the same directory as subdivx.zip, I can see and open as text the two srt files.

[pyro12]

I remembered that I also have kodi installed on my Android phone. I tried the same subtitle download and it worked! It automatically selected the first file (Espana). Same kodi version (19.4.0) and same subdivx version (0.3.7.2). However, it is using libarchive version 19.2.0.

I will upgrade libarchive on my PC and post results.

[pyro12]

Well, I can't figure out how to upgrade the libarchive binary for the flatpak. I will close this for now as it seems it's an issue with libarchive and the flatpak.

I am still open to suggestions, though!

[pyro12]

As a test, I tried using a typical kodi installation instead of the flatpak. This is Debian Bullseye, kodi 19.4.0, subdivx 0.3.7.2, and libarchive 2.0.1. It again crashed at the same moment. I tried libarchive 19.2.0 as well and got the same results. The log looks the same.

I did run kodi from a terminal window and saw:

df: /run/user/1000/doc: Operation not permitted
Segmentation fault (core dumped)

[ramiro]

Looks like a call/fork very deep in the call chain. I'd say it will be very hard to research, isolate this from our position, let alone solve it outside of trying different versions of the dependencies.

if you have experience using the strace(1) it might provide some further insight about the call stack if you save its output fo a [set of] file(s), but the challenge will be finding the right combination of command line switches to pass to it to get useful data considering Kodi probably uses multi-threading

[pyro12]

Thank you for your input. I think I'll just chalk this up to one of the few but weird downsides to using Debian Stable. Maybe if I have a spare afternoon, I'll try a complete reinstall of kodi.

[pyro12]

Hello again,

I've been trying to figure this out in my spare time and I'd like to know if what I have found sparks any ideas in your mind:

Firstly, I have reproduced this issue on two Debian systems and one Arch system. The only thing that might be unusual is that the OS is running in English.

Due to my extremely limited debug skills, I started putting log('line [line number]') anywhere that I thought there might me trouble. This way, I could see the last line that was successfully executed in the log before the seg fault. I think I learned that the line that is causing the trouble is in main and is:

listitem = xbmcgui.ListItem(label=sub["path"])

(approximately line 723)

I tried to print the value of sub["path"] into the log so I could see it, but I get an error and subdivx exits:

log("725 %s" % sub["path"] , level=LOGWARNING)

However, I can print the entire list/array/item (sorry - the nomenclature still confuses me):

log("725 %s" % sub , level=LOGWARNING)

Using 'Charmed S01E01 as an example, I get:

WARNING <general>: SUBDIVX - main: 725 {'forced': False, 'path': '/home/pyro/.var/app/tv.kodi.Kodi/data/userdata/addon_data/service.subtitles.subdivx/tmp4v2mhywc/Charmed - 1x01 - Algo Extra\udcc3\udcb1o se aproxima Ingles.srt'} for the file named:

Charmed - 1x01 - Algo Extraño se aproxima Ingles.srt

Is it possible that Linux takes issue with the escape strings in the path? I've never seen escape strings before and had to use google just to learn what they are, but I've never seen a path like that.

I just tried pyro=sub["path"] and log("725 %s" % pyro, level=LOGWARNING and the log now says:

UnicodeEncodeError: 'utf-8' codec can't encode characters in position 142-143: surrogates not allowed

I feel as though I'm getting close to the issue. Any suggestions?

[ramiro]

@pyro12 From previous comments we know you are using Kodi 19. Are you using version 0.4.0 of the add-on?

[ramiro]

@pyro12 Taking advantage of the fact you have an environment to reproduce the issue and that you are already editing the add-on source code:

Please change line 443:

from

files = os.listdir(workdir)

to

_, files = xbmcvfs.listdir(workdir)

And then retry the downloads(s) that are failing on you.

Let's see if the change helps to get Zip file contents file listing with paths that aren't as broken as they are now for you.

[pyro12]

yes, I am using version 0.4.0

That worked! I chose an unfortunate example and got English subs, but that's far better than a segment fault. I also tried a couple other subs and it is working!

Thank you so much for taking the time to dig into this! I'm getting older and my ears don't work as well as they used to so subtitles are quite useful. I like to unwind and watch TV at night, but I always felt a little guilty - like I was wasting time. My solution was to watch with Spanish subtitles so I would at least be exposed to a second language. I can actually read conversational Spanish pretty well now - thanks in large part to this addon.

I am pleased that I have contributed to it in even a small way.

Thank you for your help - I was a long way from finding the solution.

[ramiro]

Thanks for the report and willingness to test a workaround/fix. The issue was that somehow I had decided to use Python standard library's os.listdir() to get a listing of a file system folder and it seems it can return bad things (an Unicode string with surrogates) under certain conditions like the one(s) you ended with.

Also, it's strange is that Kodi segfaults completely when handling something so seemingly innocuous like such a string.

I should have used Kodi xbmcvfs.listidr() abstraction (like in fact I'm already doing elsewhere in the add-on code) which has has a higher chance of isolating us of problems like this.

I will try to pack that fix as [part of] 0.4.1 ASAP.

Thanks again!