search

jurialmunkey / plugin.video.themoviedb.helper

GNU General Public License v3.0
199 stars 95 forks source link

[BUG] Random Crashes on Nexus Nightlies #948

Closed xen0m0rph2020 closed 1 year ago

xen0m0rph2020 commented 1 year ago

Skin section

Other

Current Behavior

Randomly when moving around home screen or widgets or playing video Kodi crashes and restarts This does not happen on nightlies prior to 15th October and on skins without TVMBHelper,

Expected Behavior

Does not crash

Steps To Reproduce

Move down or left or right selection or do nothing

Screenshots and Additional Info

kodi_crashlog_20221130182515.log

Checklist

xen0m0rph2020 commented 1 year ago

another crash log when Kodi is idle kodi_crashlog_20221130182850.log

xen0m0rph2020 commented 1 year ago

here is debug turned on kodi_crashlog_20221130200446.log

xen0m0rph2020 commented 1 year ago

please move to tvmb helper as I think its related to that

jurialmunkey commented 1 year ago

Log doesn't give me many clues really. Let's try disabling the TMDbHelper service to confirm whether it is actually TMDbHelper.

First up, disable "Blur" and "Clearlogo" from the appearance settings. You can use "Stretch" as a native Kodi replacement for Blur.

Then we're going to add a button to your skinshortcuts to disable the TMDbHelper service. Add a button either in a submenu or the powermenu with the custom action Skin.Reset(TMDBHelper.Service)

Clicking this button will disable the TMDbHelper service monitor. You won't get ratings any more (they'll be frozen on last item used) and a few other things won't work correctly, so it isn't suitable for long term use. But it will work well enough for testing.

If you still get crashes with the service disabled then the problem isn't TMDbHelper as that's the only thing running whilst idle.

PS: The skin will renable the Service setting on restart, so don't restart Kodi after changing it to test as it will just turn back on.

roidy commented 1 year ago

I can also confirm this issue, however it's not related to this plugin as it also happens with script.embuary.helper and script.embuary.info. I suspect it's a reoccurrence or related to this issue:-

https://forum.kodi.tv/showthread.php?tid=368069

xen0m0rph2020 commented 1 year ago

Log doesn't give me many clues really. Let's try disabling the TMDbHelper service to confirm whether it is actually TMDbHelper.

First up, disable "Blur" and "Clearlogo" from the appearance settings. You can use "Stretch" as a native Kodi replacement for Blur.

Then we're going to add a button to your skinshortcuts to disable the TMDbHelper service. Add a button either in a submenu or the powermenu with the custom action Skin.Reset(TMDBHelper.Service)

Clicking this button will disable the TMDbHelper service monitor. You won't get ratings any more (they'll be frozen on last item used) and a few other things won't work correctly, so it isn't suitable for long term use. But it will work well enough for testing.

If you still get crashes with the service disabled then the problem isn't TMDbHelper as that's the only thing running whilst idle.

PS: The skin will renable the Service setting on restart, so don't restart Kodi after changing it to test as it will just turn back on.

Ok I have just done this and created button and selected it. I will see if it crashes now. Although this test is dificult as the crash does not always happen right away sometimes I can boot kodi and watch a episode and then be ok and then next day I start kodi and it crashes. I not had issues with default skin but I did have the same issue on Cosmic skin by heppenowski which also uses the plugin but not sure what else it has in common with AZ2

xen0m0rph2020 commented 1 year ago

I can also confirm this issue, however it's not related to this plugin as it also happens with script.embuary.helper and script.embuary.info. I suspect it's a reoccurrence or related to this issue:-

https://forum.kodi.tv/showthread.php?tid=368069

i do not agree those bugs are in pure Linux not LE. I have no crashes with Embury or its related plugins. I get this crash on AZ2 and Cosmic only.

Nuklear92 commented 1 year ago

Glad someone posted about this, yes! I'm actually getting lots of crashes with AH2 using it with Kodi 20.0 Beta 1. I get a crash mostly when I load lists using any addon even TMDb Helper. I have no clue what's wrong nor I get any log error issues. Kodi just crash flat out randomly when you load any lists, being about new movie/tv show releases, popular, trending, etc. It can happen with any lists and random. So, it's hard to ping point where the issue is coming from. 🤷‍♂️

Edit: BTW I'm on a Nvidia Shield TV Pro just in case it matters.

xen0m0rph2020 commented 1 year ago

@jurialmunkey so after turning TVMBHelper off via the shortcut it has not crashed. I turned off FLixArt and ClearLogo and it crashed before I got to use shortcut and then after it booted again I used the shortcut and since then no reboots. I will let you know if I get a reboot over weekend after disabling it.

jurialmunkey commented 1 year ago

@Nuklear92 - I think that's a different issue on Android with that Beta version specifically as other users were able to roll back to an earlier Kodi nightly and no longer had the issue.

Fairly sure this issue on Libre/CoreELEC is something different related to their implementation of python not liking something TMDbHelper is doing and instead of it properly handling an exception it is crashing.

On Windows with latest nightlies I have absolutely no issues at all. I switched over my main loungeroom NUC to Nexus about a week ago and haven't had a single crash. And that machine is getting a particularly good workout at the moment being Test Cricket season -- I've been able to run Live TV in Kodi uninterrupted for 8 hours a day without issue.

jurialmunkey commented 1 year ago

@xen0m0rph2020 - Thanks. Okay that's useful info. If it crashed with clearlogo/blur off then that rules out PIL being the culprit for once! Must be something the service monitor is doing specifically that LibreELEC or its python implementation doesn't like.

~I'm going to try to install LibreELEC in a virtual machine and see if I can recreate the issue~. Bit difficult to find out what the cause is if it doesn't happen for me personally on Windows.

EDIT: Turns out can't run LibreELEC in a VM.

I've tried to reproduce this issue on Linux Mint but can't. Skin and plugin work fine for me on both Linux and Windows nightlies.

xen0m0rph2020 commented 1 year ago

@xen0m0rph2020 - Thanks. Okay that's useful info. If it crashed with clearlogo/blur off then that rules out PIL being the culprit for once! Must be something the service monitor is doing specifically that LibreELEC or its python implementation doesn't like.

~I'm going to try to install LibreELEC in a virtual machine and see if I can recreate the issue~. Bit difficult to find out what the cause is if it doesn't happen for me personally on Windows.

EDIT: Turns out can't run LibreELEC in a VM.

I've tried to reproduce this issue on Linux Mint but can't. Skin and plugin work fine for me on both Linux and Windows nightlies.

That must mean its isolated to LE builds. If I could I would buy you a Pi so you can test lol Is there any debug output you can add to the plugin which will enable more logging to occur?

xyzfre commented 1 year ago

@Nuklear92 - I think that's a different issue on Android with that Beta version specifically as other users were able to roll back to an earlier Kodi nightly and no longer had the issue.

Fairly sure this issue on Libre/CoreELEC is something different related to their implementation of python not liking something TMDbHelper is doing and instead of it properly handling an exception it is crashing.

On Windows with latest nightlies I have absolutely no issues at all. I switched over my main loungeroom NUC to Nexus about a week ago and haven't had a single crash. And that machine is getting a particularly good workout at the moment being Test Cricket season -- I've been able to run Live TV in Kodi uninterrupted for 8 hours a day without issue.

This is great news to hear I've been holding back on Nexus but after reading this post I'm also running Windows with AH2 definitely going to install Nexus today to give it a test run I'm hearing it's super fast using AH2

Nuklear92 commented 1 year ago

@Nuklear92 - I think that's a different issue on Android with that Beta version specifically as other users were able to roll back to an earlier Kodi nightly and no longer had the issue.

Fairly sure this issue on Libre/CoreELEC is something different related to their implementation of python not liking something TMDbHelper is doing and instead of it properly handling an exception it is crashing.

On Windows with latest nightlies I have absolutely no issues at all. I switched over my main loungeroom NUC to Nexus about a week ago and haven't had a single crash. And that machine is getting a particularly good workout at the moment being Test Cricket season -- I've been able to run Live TV in Kodi uninterrupted for 8 hours a day without issue.

Sorry for the late reply, so is this something us, Android TV users will have to deal with using Kodi Nexus for the foreseeable future. Or this is something that could eventually get sort out as nexus moves over to Public Releases and then you put all your focus on Nexus instead of Matrix currently? 🤔

jurialmunkey commented 1 year ago

so is this something us, Android TV users will have to deal with using Kodi Nexus for the foreseeable future. Or this is something that could eventually get sort out as nexus moves over to Public Releases and then you put all your focus on Nexus instead of Matrix currently?

@Nuklear92

I can't speak to Android or ELECs specifically as I don't use those platforms but my focus is already on Nexus.

Nexus is now my daily driver on all my devices and the main TMDbHelper branch is Nexus. Any recent new feature was specifically developed+tested in Nexus, and the Matrix versions were backports.

More to the point: It's difficult for me to "fix" something that I can't recreate and the only info I'm given is "it crashes". If there's no discernable pattern and the logs don't have errors then there's not much for me to go on. Obviously TMDbHelper is agitating some underlying issue but that doesn't make it the cause -- otherwise you could tell me which version introduced the problem.

To explain by analogy: if you break your leg (platform bug) then riding a bicycle (TMDbHelper) will be difficult. It would be strange to suggest that the bicycle is broken because you can still row a boat (another plugin). I would also be strange to say the bicycle manufacturing process is broken and they need "fix" their bicycles by figuring out some workaround like adding a hand crank instead of pedals. You definitely wouldn't expect the bicycle maker to xray your leg and diagnose it.

If you didn't think your leg was broken and it was a problem with the new bike, then you could test it by riding the old bike (rolling back to the last "stable" version you used). If the problem was that your leg was broken, you'd quickly find out that you were also unable to ride the old bike.

Nuklear92 commented 1 year ago

so is this something us, Android TV users will have to deal with using Kodi Nexus for the foreseeable future. Or this is something that could eventually get sort out as nexus moves over to Public Releases and then you put all your focus on Nexus instead of Matrix currently?

@Nuklear92

I can't speak to Android or ELECs specifically as I don't use those platforms but my focus is already on Nexus.

Nexus is now my daily driver on all my devices and the main TMDbHelper branch is Nexus. Any recent new feature was specifically developed+tested in Nexus, and the Matrix versions were backports.

More to the point: It's difficult for me to "fix" something that I can't recreate and the only info I'm given is "it crashes". If there's no discernable pattern and the logs don't have errors then there's not much for me to go on. Obviously TMDbHelper is agitating some underlying issue but that doesn't make it the cause -- otherwise you could tell me which version introduced the problem.

To explain by analogy: if you break your leg (platform bug) then riding a bicycle (TMDbHelper) will be difficult. It would be strange to suggest that the bicycle is broken because you can still row a boat (another plugin). I would also be strange to say the bicycle manufacturing process is broken and they need "fix" their bicycles by figuring out some workaround like adding a hand crank instead of pedals. You definitely wouldn't expect the bicycle maker to xray your leg and diagnose it.

If you didn't think your leg was broken and it was a problem with the new bike, then you could test it by riding the old bike (rolling back to the last "stable" version you used). If the problem was that your leg was broken, you'd quickly find out that you were also unable to ride the old bike.

I'll try to turn on Debugg and start loading lists until Kodi crashes, hopefully there's something unusual you can catch from it. By the way, I'm using Kodi Nexus 20.0 Beta 1.

jurialmunkey commented 1 year ago

@xen0m0rph2020 @Nuklear92 - Could you test using the log_testing_nexus branch version and provide me the logs: https://github.com/jurialmunkey/plugin.video.themoviedb.helper/archive/refs/heads/log_testing_nexus.zip

This version will generate a LOT of log spam so it is not suitable for normal usage. However, it will help identify exactly where in the code Kodi is crashing. If we're able to repeat it a few times in the same spot then it will make it much easier to report the underlying bug and I possibly might even be able to build a test case plugin for Estuary as a "proof" of the problem.

Nuklear92 commented 1 year ago

@xen0m0rph2020 @Nuklear92 - Could you test using the log_testing_nexus branch version and provide me the logs: https://github.com/jurialmunkey/plugin.video.themoviedb.helper/archive/refs/heads/log_testing_nexus.zip

This version will generate a LOT of log spam so it is not suitable for normal usage. However, it will help identify exactly where in the code Kodi is crashing. If we're able to repeat it a few times in the same spot then it will make it much easier to report the underlying bug and I possibly might even be able to build a test case plugin for Estuary as a "proof" of the problem.

I definitely can try, just tell me exactly what do I need to do with the download link apart from download it and where does it install.

I'm currently using Nexus RC 2 and it's still crashing whenever I use any skins, AuraMod, Arctic Horizon 2 which is my main one. I also tried even Arctic Zephyr Reloaded which I believe it's based on your Arctic Zephyr old skin too. They all crash randomly when loading menus, lists, etc within any addons including TMDB Helper.

Nuklear92 commented 1 year ago

@jurialmunkey Check if this Kodi log works for you, I enabled debug option for both Kodi and the TMDb's helper addon with the version you gave me. I was just browsing a list and opening and closing some context menu options and out of nowhere it just crashed. The skin is just unusable at least on Android tv devices like the shield using Kodi Nexus.

If you want me to do anything additional or create a different log with an option I might not know, let me know.

kodi.txt

jurialmunkey commented 1 year ago

@Nuklear92 - thanks. You can actually turn debug logging setting off as that version will still log the relevant messages I need.

I'm basically looking for a pattern so if you can run it a couple more times and get me the log each time it crashes that'd be excellent. If I can identify some sort of pattern then we can try turning a few things off to pinpoint what Kodi is having a problem with. That will make it much easier to report the issue for Android devs to have a look at.

Nuklear92 commented 1 year ago

@Nuklear92 - thanks. You can actually turn debug logging setting off as that version will still log the relevant messages I need.

I'm basically looking for a pattern so if you can run it a couple more times and get me the log each time it crashes that'd be excellent. If I can identify some sort of pattern then we can try turning a few things off to pinpoint what Kodi is having a problem with. That will make it much easier to report the issue for Android devs to have a look at.

Absolutely, I'll try to do it a couple of more times then. Just to make sure when you say to turn off the debug setting, are you referring to the Kodi's debug setting or the one that TMDB Helper's debug setting?

jurialmunkey commented 1 year ago

Also another thing worth mentioning is to check the reuselanguageinvoker setting of any plugins you use as widgets (it will be listed in the addon.xml of the plugin if it uses it).

Any plugin which has reuselanguageinvoker set to true will crash kodi randomly if you try to use it for any widgets.

jurialmunkey commented 1 year ago

@Nuklear92 - thanks. You can actually turn debug logging setting off as that version will still log the relevant messages I need. I'm basically looking for a pattern so if you can run it a couple more times and get me the log each time it crashes that'd be excellent. If I can identify some sort of pattern then we can try turning a few things off to pinpoint what Kodi is having a problem with. That will make it much easier to report the issue for Android devs to have a look at.

Absolutely, I'll try to do it a couple of more times then. Just to make sure when you say to turn off the debug setting, are you referring to the Kodi's debug setting or the one that TMDB Helper's debug setting?

Both can be disabled. The relevant messages will still get logged because I've set their notice level at info not debug.

Nuklear92 commented 1 year ago

@jurialmunkey Okay, here are 4 consecutive Kodi logs again using AH2 skin, each time they crashed in different menu lists. Hopefully you can catch something unusual. Let me know if you need more or this is good enough. 👍

kodi.txt

kodi (2).txt

kodi #3.txt

kodi #4.txt

tekker commented 1 year ago

Hi, I have been having the same apparent issue as well with the Nexus nightlies in combination with TMDBHelper and AH2. Disabling all Ratings in AH2 reduces the frequency of crashes and disabling the TMDBHelper service via (Skin.Reset(TMDBHelper.Service)) reduces the frequency of the crashes further. Often there are FFMPEG errors (maybe relating to image corruption?) in the logs of the form:

ffmpeg[0x671f580]: [image2] Custom AVIOContext makes no sense and will be ignored with AVFMT_NOFILE format.

I have been wrangling with these crashes for the last month and picking through the logs to try and figure it out. (I have a similar install of AH2 + TMDBHelper / Autowidget / Fen with Nexus installs on Pi4/Pi400 (LibreELEC), MacOS M1 (Arm64 Kodi Nightly), and Android armv7 (FireTV/Kodi Nightly).

The performance of AH2 on Nexus is amazing - if this issue could be overcome I can roll it out to home users. I am happy to help in any way if I can.

tekker commented 1 year ago

Fen has the reuselanguageinvoker set to true so I will disable it and see if this bug still has an impact on latest Nexus nightlies.

Nuklear92 commented 1 year ago

Hi, I have been having the same apparent issue as well with the Nexus nightlies in combination with TMDBHelper and AH2. Disabling all Ratings in AH2 reduces the frequency of crashes and disabling the TMDBHelper service via (Skin.Reset(TMDBHelper.Service)) reduces the frequency of the crashes further. Often there are FFMPEG errors (maybe relating to image corruption?) in the logs of the form:

ffmpeg[0x671f580]: [image2] Custom AVIOContext makes no sense and will be ignored with AVFMT_NOFILE format.

I have been wrangling with these crashes for the last month and picking through the logs to try and figure it out. (I have a similar install of AH2 + TMDBHelper / Autowidget / Fen with Nexus installs on Pi4/Pi400 (LibreELEC), MacOS M1 (Arm64 Kodi Nightly), and Android armv7 (FireTV/Kodi Nightly).

The performance of AH2 on Nexus is amazing - if this issue could be overcome I can roll it out to home users. I am happy to help in any way if I can.

Glad I'm not the only one getting these weird crashes which are making Kodi nexus unstable and unusable with these skins. Hopefully, we can all somehow help @jurialmunkey get to the bottom of this! If you read some of the previous comments. He posted a download link for a special version of TMDb helper. Read that comment altogether, download, install it and provide him with logs Everytime Kodi crashes that way he can hopefully find a pattern for this and create a potential fix so we can all enjoy this skin in Nexus as we did back in Matrix. 👍

Nuklear92 commented 1 year ago

@jurialmunkey Heres another log aside from the 4 previous ones I attached on my previous comment, yesterday. This one is interesting, because prior this the other ones were with one addon I have that had as you said Language Invoker was set to True/ON. I set it to OFF for this log and still I got a crash. Give it a look whenever you have a free time as well as the other ones and see if you can notice a pattern or something. Again, the previous ones are with the an addon that had Language Invoker set to ON. This one is with it OFF.

kodi.txt

tekker commented 1 year ago

@Nuklear92 Thanks for the assist, yes I did see the comment for the branch with logging of the TMDBHelper service monitor and installed the addon from that branch of the repository.

Not sure if you are able to get SSH access / are using LibreELEC, if so this might be useful to see if the reuselanguageinvoker problem is impacting what's going on in your build . The following one-line bash script will safely return a list of all addons using reuselanguageinvoker (login to LibreELEC via SSH using Terminal / puTTY and run this line:

 grep -n "<reuselanguageinvoker>true<" /storage/.kodi/addons/**/*.xml -
R|awk -F\: '{ print $1 }'|sort

ie:

❯ ssh root@bunny.local
##############################################
#                 LibreELEC                  #
#            https://libreelec.tv            #
##############################################

LibreELEC (community): nightly-20221220-d479071 (RPi4.arm)

bunny:~ # grep -n "<reuselanguageinvoker>true<" /storage/.kodi/addons/**/*.xml -
R|awk -F\: '{ print $1 }'|sort

/storage/.kodi/addons/metadata.themoviedb.org.python/addon.xml
/storage/.kodi/addons/plugin.program.autowidget/addon.xml
/storage/.kodi/addons/plugin.program.iptv.merge/addon.xml
/storage/.kodi/addons/plugin.video.au.freeview/addon.xml
/storage/.kodi/addons/plugin.video.fen/addon.xml
/storage/.kodi/addons/plugin.video.foxtel.go/addon.xml
/storage/.kodi/addons/plugin.video.youtube/addon.xml
/storage/.kodi/addons/script.module.slyguy/addon.xml
/storage/.kodi/addons/slyguy.10play/addon.xml
/storage/.kodi/addons/slyguy.7plus/addon.xml
/storage/.kodi/addons/slyguy.9now/addon.xml
/storage/.kodi/addons/slyguy.dependencies/addon.xml

I was able to exclude addons with this issue and modify the latest AutoWidget to change reuselanguageinvoker setting and modifying the version number / provider attributes in AutoWidgets addon.xml file, re-zipping the /storage/.kodi/addons/plugin.program.autowidget directory and re-installing it.

FEN controls this setting (by rewriting the addon.xml file for itself and then reloading / enabling itself) using an item in Tools->Set Language Invoker.

I temporarily removed the slyguy.uk addons but I think at least now the reuselanguageinvoker issue is not confounding the problem with these crashes in Nexus...

(Attached a build of AutoWidget with reuselanguageinvoker disabled) plugin.program.autowidget-tek-1579.zip

tekker commented 1 year ago

@jurialmonkey From what I have read if any addons have reuselanguageinvoker set to True, Kodi will enable the reusing python thread setting within Kodi system-wide, does that sound right?

I can confirm I am seeing what appears to be the same pattern of crashes in Nexus / AH2 w/ TMDB across MacOS arm64 builds, Android armv7, and LibreELEC builds with Pi4. I have been trying to narrow it down before sending logs but its very difficult to pin down. Crashes occur while scrolling EPGs, navigating back to the home screens, etc. The only thing it has in common is it looks like it involves sending HTTP requests, manipulating images, and setting ListItem properties, but I can tell you are close to finding the root cause of this :)

Nuklear92 commented 1 year ago

I already disabled language Invoker in FEN and it still crashes. So, this issue is definitely not related to that. I even shared a log from a crash I had after turning off the language Invoker thing. 🤷‍♂️ still waiting for @jurialmunkey to check all the logs I've provided since yesterday night, hopefully when he gets a free time. And see if he can ping point some sort of pattern of where these crashes are coming when using AH2 in Nexus.

tekker commented 1 year ago

@jurialmunkey @Nuklear92

With the following changes, and after testing with Pi4 / LibreELEC on very latest nightly from last night, I have not been able to reproduce the error at all.

Remove use of reuselanguageinvoker within all addons FEN and AutoWidget. Confirmed no addons using reuselanguageinvoker on the system (@Nuklear92 - I was also seeing crashes while browsing views inside the addons, not only on the home screen, and it also appears that the youtube addon includes the reuselanguageinvoker=true setting, so it is possible that it is not enough to disable reuselanguageinvoker in FEN only).

Disable use of all Ratings and placeholder options by changing skin settings menu.

Execute Skin.Reset(TMDBHelper.Service) within Powermenu as soon as Kodi reboots.

I will now monitor and upload the logs as soon as any crash occurs and try and hit error condition across Android / MacOS builds.

Nuklear92 commented 1 year ago

@jurialmunkey @Nuklear92

With the following changes, and after testing with Pi4 / LibreELEC on very latest nightly from last night, I have not been able to reproduce the error at all.

Remove use of reuselanguageinvoker within all addons FEN and AutoWidget. Confirmed no addons using reuselanguageinvoker on the system (@Nuklear92 - I was also seeing crashes while browsing views inside the addons, not only on the home screen, and it also appears that the youtube addon includes the reuselanguageinvoker=true setting, so it is possible that it is not enough to disable reuselanguageinvoker in FEN only).

Disable use of all Ratings and placeholder options by changing skin settings menu.

Execute Skin.Reset(TMDBHelper.Service) within Powermenu as soon as Kodi reboots.

I will now monitor and upload the logs as soon as any crash occurs and try and hit error condition across Android / MacOS builds.

I can't seem to find any setting for language Invoker within the YT addon. And also I don't use AutoWidget. 🤷‍♂️

tekker commented 1 year ago

No, actually I don't think there is a setting inside the Youtube addon - I actually modified the addon.xml file used by the addon itself. I think you could download the addon from github, increment the version number and change provider name (to stop the prev version being reinstalled), and change the line: < reuselanguageinvoker>true< to set that option to false in the addon.xml file. And then re-zip the addon (ie. zip -r plugin.video.youtube-new.zip plugin.video.youtube/ ) and install the new addon from this zip instead of from the repository.

this is the file that needs changing: https://github.com/anxdpanic/plugin.video.youtube/blob/master/addon.xml (line 27)

Alternatively you could probably modify the addon.xml in your Kodi Home directory (ie. addons/plugin.video.youtube/addon.xml) directly to change the reuselanguageinvoker line and disable / re-enable the addon, then restart your Kodi. Its a fairly low-level change and might be tricky. It didn't work so well for the other addons so I disabled them but at least I now have a test environment where I have removed this as a cause.

Additionally in AH2->Appearance I have set Background artwork to Stretch, Foreground to use FlixArt setting, with Clearlogo enabled.

jurialmunkey commented 1 year ago

@tekker - Disabling ratings in the skin only hides the label in the GUI, not the lookup itself. The only way to stop ratings being retrieved is disabling the service. I've already identified the service as what is triggering kodi to crash, so disabling it defeats the purpose of gathering the log.

The purpose of the special logging version of TMDbhelper linked above is to output continuously what the service is doing so that I can identify where in the process the crash occurs.

Disabling all the plugins with reuselanguageinvoker is important as it is a well known bug with multiple widgets and as a result is simply incompatible with skins allowing custom widget setups. The dev who made the reuselanguageinvoker setting has left kodi development so that bug will never be fixed and I can't do anything about plugin authors who continue to insist on using it despite the problems it causes.

Basically I need logs of the crash with the service on, using the special version linked above, and with all reuselanguageinvoker plugins disabled to avoid false positives. That will allow me to identify a pattern and then make a further test case to confirm.

tekker commented 1 year ago

Thanks @jurialmunkey that was my assumption regarding ratings etc and the TMDBHelper logging by looking through the code.

I totally understand where you are coming from regarding reuselanguageinvoker, however it seems difficult for users to disable or even know which addons are using the "reuselanguageinvoker" settings but so far it looks possible to disable it system wide using a shell script in Linux (or python script / service within kodi). Is it only a setting used internally to improve performance, or is this wrong?

I am running three instances of kodi in parallel at present - all with the TMDbhelper-logging version. After a full days use of instance 1 (No reuselanguageinvoker plugins (verified) with Skin.Reset(TMDBHelper.Service) I have had NO crashes. In instance 2 (just finished resetting reuselanguageinvoker plugins / no Skin Reset) I have had 1 crash. It took me awhile to write something to deal with the reuselanguageinvoker. I can now monitor this condition between the two instances. Will send you a crash log with TMDB service debug of the next time it crashes.

I very much appreciate all the work you have done on AH2 / TMDBHelper it is in a different league to the other skins.

For reference, this Bash script can be used in Linux to switch off reuselanguageinvoker on all addons:

grep -n "<reuselanguageinvoker>true<" /storage/.kodi/addons/**/*.xml -R|awk -F\: '{ print $1 }'|sort|uniq|while read file; do \
    sed -i 's/<reuselanguageinvoker>true</<reuselanguageinvoker>false</g' $file
    touch $file.reuse-off
    parentname=$(basename "$(dirname $file)")
        kodi-send -a "UpdateLocalAddons()"
        sleep 4
done

If needed I could probably hack together an addon to reset all the addons reuselanguageinvoker across the other platforms.

tekker commented 1 year ago

Okay, Instance 2 with confirmation of no reuselanguage crashed fairly quickly after turning ratings on (Kodi stars / IMDB as you say probabaly not related just letting you know the sequence) and navigating through list of TV shows.

Here is the logs (crash log 1 may not be an actual crash) crash_log_1_no_reuse_with_ratings.log

crash_log_2.log

crash_log_3.log

Following crashes within TMDBHelper->TV Show->Trending / Popular etc. screens

kodi_log_5.log

I have switched the loglevel to 1 in advancedsettings.xml so might have more info in the next set of logs.

kodi_log_6.log

Following a fairly long period of navigation testing without errors in TMDB->TV Shows->Top Rated, a crash occured soon in TMDB->TV Shows->Popular after clicking to view a Series.

kodi_log_7.log

Followed soon by another crash navigating TMDB->TV Shows->Top Rated.

kodi_log_8.log

jurialmunkey commented 1 year ago

@tekker Excellent thanks for this. Really useful. Interesting it is failing quite a bit on the readahead so lets take that out of the equation by disabling it and then see if/where it continues to crash.

Could you try this version (basically same but disables readahead function) and get me a few logs (assuming it continues to crash) https://github.com/jurialmunkey/plugin.video.themoviedb.helper/archive/refs/heads/log_testing_nexus_noreadahead.zip

Is it only a setting used internally to improve performance, or is this wrong?

Yeah reuselanguageinvoker improves performance quite a bit by keeping the python instance active between directories. It means that plugins don't need to reimport all their modules when the user navigates to a new section or page.

Problem is it was only tested in Estuary where you can't set plugins for widgets so it wasn't identified as a problem until after release. Also originally the crashes that were popping up were blamed on poor coding practices in the plugins using it, especially since it would only happen intermittently and no one was able to make a test case -- that is until I did recently (see https://github.com/xbmc/xbmc/issues/21653).

Unfortunately the dev who made the original reuselanguageinvoker code (peak3d) is no longer around so it is highly unlikely someone else will pickup further development on the feature. Personally I think it should be removed since it is broken and no one is working on it but there's always a lot of friction suggesting the removal of things that work fine in Estuary and only break other skins allowing customisation.

You can see it has been a long standing issue as reported here: https://github.com/xbmc/xbmc/issues/16844

tekker commented 1 year ago

@jurialmunkey Thanks mate I am glad I have helped and you were able to narrow it down. I see that reuselanguageinvoker has been a thorn in your side as a developer :)

I noticed Peak3d's comment: "Suggestion to all of the ppl. who mislike this PR: Buy a rbpi, install LibreELEC, and play around with your addon. Testing addons on a Intel i7 does not cover kodi userbase." when adding the function. That's an interesting approach to introduce a major system change ;) (from https://github.com/xbmc/xbmc/pull/13814)

I'm still reading through those issues and my knowledge of python is very basic, but I did notice a series of commits that that Castagnait made for the Netflix addon way back in 2019 bundled in this commit - looks like there were a fairly scary set of changes needed to his caching logic:

https://github.com/CastagnaIT/plugin.video.netflix/commit/2a6b1beb63e9d8940bff9426b906907139d38101

A couple of notes he made in his commit history seemed a little helpful at least for me to understand what is happening with python threading:

https://github.com/CastagnaIT/plugin.video.netflix/commit/17ad8d67dad2852d33c13bfa0c6598837ad9434f https://github.com/CastagnaIT/plugin.video.netflix/commit/17ad8d67dad2852d33c13bfa0c6598837ad9434f

I started work on an addon to try and learn a bit more about Kodi etc, its a Tailscale addon that allows configuration / setup of exit nodes on Mac / Linux / Android, but "development has stalled". It seems like the Kodi devs have set a trap for the unprepared developer so hopefully this hasn't tripped me up...

Let me know if there is any other testing I can do to help.

tekker commented 1 year ago

After updating with the readahead change, I have a new crash log (same area as before):

kodi_log_9.log

This crash was not long after updating the addon, now the system has restarted and all is clear at the moment. Will add a new crash log if I get another crash...

After an hour or so testing, I have not been able to cause another crash condition - so it looks promising mate :)

Switching to a secondary build post readahead update and restart, change Ratings to show IMDB TMDB Rotten Tomatoes, all seemed fine except after changing the option Settings > Media > General > Show Parent folder items (disable) to remove the list of other shows from the TV Show Season view, I had another crash (in same area as previous crash - navigating through Landscape Combined view of TMDB Helper TV Shows)

kodi_log_10.log

Another crash a bit later:

kodi_log_11.log

Nuklear92 commented 1 year ago

I'm kind of lost here as y'all speaking in advanced mode compared to my level of knowledge with Kodi as a whole which is pretty limited. Is @jurialmunkey abled to ping point what's causing the crash exactly or still more time to fully be confident of what's exactly causing it?

And if a fix will require every user to turn off a certain setting in order to prevent the crashes, etc. Because, as far I have know Matrix didn't cause any crashes when using skins with widgets ragardless of certain addons using reuse language Invoker. What I'm trying to find out is if we will need to turn off certain settings after a patch is provided for the crashes or it could potentially fix the issue regardless of some users not knowing they have language Invoker set to TRUE. 🤔

Nuklear92 commented 1 year ago

No, actually I don't think there is a setting inside the Youtube addon - I actually modified the addon.xml file used by the addon itself. I think you could download the addon from github, increment the version number and change provider name (to stop the prev version being reinstalled), and change the line: < reuselanguageinvoker>true< to set that option to false in the addon.xml file. And then re-zip the addon (ie. zip -r plugin.video.youtube-new.zip plugin.video.youtube/ ) and install the new addon from this zip instead of from the repository.

this is the file that needs changing: https://github.com/anxdpanic/plugin.video.youtube/blob/master/addon.xml (line 27)

Alternatively you could probably modify the addon.xml in your Kodi Home directory (ie. addons/plugin.video.youtube/addon.xml) directly to change the reuselanguageinvoker line and disable / re-enable the addon, then restart your Kodi. Its a fairly low-level change and might be tricky. It didn't work so well for the other addons so I disabled them but at least I now have a test environment where I have removed this as a cause.

Additionally in AH2->Appearance I have set Background artwork to Stretch, Foreground to use FlixArt setting, with Clearlogo enabled.

Thanks for the input, I think I can do that via the Kodi folder. I've changed stuff in the past so I don't think it would be that hard as it's only changing TRUE to FALSE. However, will updating YT addon when they release a new update overwrite this back to TRUE? 🤔

meesterexx commented 1 year ago

this is the file that needs changing: https://github.com/anxdpanic/plugin.video.youtube/blob/master/addon.xml (line 27)

This is an incredibly useful tip. I have been turning off reuse language invoker in addons for years and never knew about this in the yt addon. Thanks!

tekker commented 1 year ago

@Nuklear92 Will updating YT addon when they release a new update overwrite this back to TRUE? It will definitely be overwritten on update, you can try to modify the version attribute (to a higher number than any release from the original repository) or alternatively I think you can change the provider-name attribute... Probably making these changes and then asking FEN to reset its own "Language Invoker" setting will get Kodi to reload the updated addon.xml file(s)

As an example, the first line in the FEN addon xml (~/.kodi/addons/plugin.video.fen/addon.xml) is:

< addon id="plugin.video.fen" name="Fen" provider-name="Tikipeter" version="3.0.32" >

lebigboss59 commented 1 year ago

Hi, for me it was Skin Helper service the problem. I had uninstall it and skin helper service backup, now it's ok for me

jurialmunkey commented 1 year ago

@Nuklear92 - Disabling plugins with reuselanguageinvoker is simply for testing purposes. It is a well known cause of unexpected crashes when used in multi widget setups so we need to make sure it isn't affecting the test results.

That being said, in general usage you should disable reuselanguageinvoker for any plugins you intend to use as widgets or else you will experience random unexplained crashes. That goes for any skin with a customisable multi widget setup, not just mine.

In regards to the tmdbhelper crashes, any "fix" is going to have to be made by kodi android/elec etc devs. I won't be able to fix it myself via python as the issue is platform specfic. Changes to tmdbhelper to workaround new platform specific issues are like swerving to avoid a new pothole on a road. At some point you need to fix the road because all the swerving creates more hazards.

The point of these tests is to identify exactly what part is causing the crash so a more accurate bug report can be made (to expand the analogy we are trying to identify the exact part of the road we are driving on so we can report it to the council). Since I don't have access to these platforms and no one actually experiencing the issue has made an official bug report, I now need to do this extremely roundabout way of trial and error testing to be able to report it myself.

I noticed that the readahead routine (precaches next item in list when idle) was triggering crashes in the above logs. The more recent testing version I linked above disables that readahead routine to avoid that clouding the results.

Nuklear92 commented 1 year ago

@Nuklear92 - Disabling plugins with reuselanguageinvoker is simply for testing purposes. It is a well known cause of unexpected crashes when used in multi widget setups so we need to make sure it isn't affecting the test results.

That being said, in general usage you should disable reuselanguageinvoker for any plugins you intend to use as widgets or else you will experience random unexplained crashes. That goes for any skin with a customisable multi widget setup, not just mine.

In regards to the tmdbhelper crashes, any "fix" is going to have to be made by kodi android/elec etc devs. I won't be able to fix it myself via python as the issue is platform specfic. Changes to tmdbhelper to workaround new platform specific issues are like swerving to avoid a new pothole on a road. At some point you need to fix the road because all the swerving creates more hazards.

The point of these tests is to identify exactly what part is causing the crash so a more accurate bug report can be made (to expand the analogy we are trying to identify the exact part of the road we are driving on so we can report it to the council). Since I don't have access to these platforms and no one actually experiencing the issue has made an official bug report, I now need to do this extremely roundabout way of trial and error testing to be able to report it myself.

I noticed that the readahead routine (precaches next item in list when idle) was triggering crashes in the above logs. The more recent testing version I linked above disables that readahead routine to avoid that clouding the results.

I appreciate the explanation, I really hope you end up getting the exact cause and hopefully devs at Kodi can patch this. I'm dying to go back and using AH2 as it's a phenomenal skin, at the moment I'm sitting in Estuary for the time being as it doesn't make Kodi Nexus crash.

MoojMidge commented 1 year ago

@jurialmunkey - I am just speculating, but the problem may be because you are adding to/updating the Kodi GUI in separate threads. The check for the _locked member variable, from the originating thread, is not actually thread safe. The reason why this may be more apparent on different platforms is because of subtle differences in the underlying thread scheduling that is handled by the OS.

I have seen similar problems in my own plugins that result in Kodi crashing when there are multiple threads calling back to the Kodi service broker, checking list lengths/bounds, and adding items to lists/groups/containers that are already being modified. I believe the root cause of that problem was that I was creating offscreen listitems, which does not create a gui lock, but which, as a result of not locking the gui, results in crashes when actually adding the listitems to the gui due to generating multiple, simultaneous gui mutations.

One suggestion, that may help if this is the actual cause of the problem, could be to do something like this: https://github.com/jurialmunkey/plugin.video.themoviedb.helper/blob/c38a8b3daff1039920f1523a97f4b5f0125804b3/resources/lib/monitor/readahead.py#L13

...

import threading

...

self._lock = threading.Lock()

https://github.com/jurialmunkey/plugin.video.themoviedb.helper/blob/c38a8b3daff1039920f1523a97f4b5f0125804b3/resources/lib/monitor/readahead.py#L50-L54

def next_readahead(self):
    with self._lock:
        status = self._next_readahead()
    return status

Or alternatively with the lock check in next_readahead:

def next_readahead(self):
    if self._lock.locked():
        return None
    with self._lock:
        status = self._next_readahead()
    return status

https://github.com/jurialmunkey/plugin.video.themoviedb.helper/blob/c38a8b3daff1039920f1523a97f4b5f0125804b3/resources/lib/monitor/listitem.py#L275-L280

def _next_readahead():
    if self._readahead._lock.locked():
        return
    if self._readahead.next_readahead() != READAHEAD_CHANGED:
        return
    self._readahead = None

Or alternatively if the lock check is in next_readahead:

def _next_readahead():
    if self._readahead.next_readahead() != READAHEAD_CHANGED:
        return
    self._readahead = None

A similar issue may also result from the threaded _process_artwork method you use.

tekker commented 1 year ago

@jurialmunkey Forgive me if I am confusing the issue / adding extra complexity and you are probably are aware of all this and a lot more than I will ever, so please feel free to ignore this comment.

Reuselanguageinvoker reuses the python module name space for subsequent plugin directory listings -- it is an attempt to improve performance by skipping module imports. I do not know what the exact cause of the crash is, but I suspect that reuselanguageinvoker was only ever tested in the library where plugin directories load one at a time and it wasn't properly tested in an environment where multiple thread instances are all attempting to access the same namespace at the same time.

I noticed looking through the commit history that there were some main changes that developers had performed regarding thread reuse was the way globals are handled and around imports but mainly around the use of classes instead of module functions, as just enabling reuselanguageinvoker in the addon settings is not sufficient to use it without the chance of intermittent crashes. Its annoying that the Kodi devs introduced this change without also providing enough documentation or ways of testing compliance (such as with settings changes or the python2-python3), but it looks like reuselanguageinvoker is being integrated more into the XMBC core over time. They really introduced a breaking change but rather than expend resources on mitigating the problems it causes they justified it by saying devs don't have to enable it and users don't have to use addons with it enabled without taking into account how inter-dependent many addons are.

From what I could discern there are fixed rules I could follow to code safely with reuselanguageinvoker turned on but I couldn't find clear documentation for them.

class GlobalVariables(object):
    """Encapsulation for global variables to work around quirks with
    Kodi's reuseLanguageInvoker behavior"""
    # pylint: disable=attribute-defined-outside-init
    # pylint: disable=invalid-name, too-many-instance-attributes

def __init__(self):
        """Do nothing on constructing the object"""
        # Define here any variables necessary for the correct loading of the modules
        self.IS_ADDON_FIRSTRUN = None
        self.ADDON = None
        self.ADDON_DATA_PATH = None
        self.DATA_PATH = None

def init_globals(self, argv, skip_database_initialize=False):
        Needs to be called at start of each plugin instance!
        This is an ugly hack because Kodi doesn't execute statements defined on
        module level if reusing a language invoker."""
        # IS_ADDON_FIRSTRUN specifies when the addon is at its first run (reuselanguageinvoker is not yet used)
        self.IS_ADDON_FIRSTRUN = self.IS_ADDON_FIRSTRUN is None
        self.PY_IS_VER2 = sys.version_info.major == 2
        self.COOKIES = {}
        self.ADDON = xbmcaddon.Addon()

...

# pylint: disable=invalid-name
# This will have no effect most of the time, as it doesn't seem to be executed
# on subsequent addon invocations when reuseLanguageInvoker is being used.
# We initialize an empty instance so the instance is importable from run_addon.py
# and run_service.py, where g.init_globals(sys.argv) MUST be called before doing
# anything else (even BEFORE OTHER IMPORTS from this addon)
g = GlobalVariables()
import xbmc

# Import and intiliaze globals right away to avoid stale values from the last
# addon invocation. Otherwise Kodi's reuseLanguageInvoker option will cause
# some really quirky behavior!
from resources.lib.globals import g
g.init_globals(sys.argv)

# -*- coding: utf-8 -*-
from resources.lib.routing import ROUTER

import sys

ROUTER.initialize(sys.argv)
ROUTER.route()
jurialmunkey commented 1 year ago

@MoojMidge - the global interpreter lock built into python prevents asynchronous multithreading so thread safety shouldn't be a concern. It isn't possible for two threads to have access to the interpreter at the same time ever.t

More importantly, the _locked property is only there to prevent the readahead from spawning multiple threads. We only want one readahead so I lock it whilst it is working and then destroy it once it finishes or respawn it as a new thread once the user is no longer idle.

The point about the offscreen property is good though as I think I might've accidentally removed the False kwarg for it at some stage. Technically it should be set as false whenever the listitem might change whilst on screen. Threading is irrelevant - all that matters is if it is going to be changed after it is added which in this case it does. I can knock together a simple test for this - what will be interesting is if it is the cause then there's still a regression somewhere in the kodi code because it previously wasn't causing a crash. Either way, it should be false in this context so ill fix that up anyway.