Since ISA v21.1.1, some high bitrate content has abnormal memory usage that can cause Kodi to crash

[Liqianyu]

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

Stream some new episodes(?) such as THE DAY, Black Mirror Season 6 using the Netflix add-on. Playback suffers severe frame drops for 1-2 minutes and eventually freezes. It's worth noting that the problem seems to only occur at the highest bitrate. Maybe it is necessary to reach a certain bit rate. So we need to manually select the highest bit rate video stream when playing. ISA v21.1.2 didn't change anything, even worse I think.

Expected Behavior

Here is a clear and concise description of what was expected to happen:

Smooth video playback

Actual Behavior

Severely dropped frames that don't recover over time and eventually freeze.

Possible Fix

Apparently this is a problem introduced since v21.1.1, because if I use v21.1.0, this problem doesn't happen.

To Reproduce

Steps to reproduce the behavior:

1.Open Netflix add-on 2.Play specific content, such as Black Mirror Season 6 3.Choose 4K, best bitrate to play.

Debuglog

The debuglog can be found here: https://paste.kodi.tv/nufepuyara.kodi

MPD/M3U8s/ISM

An example or copy of a manifest (or manifests for HLS - master and variants) can be found here: https://paste.kodi.tv/uruwirupiy

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Your Environment

Used Operating system:Shield TV Pro 2019

• [X] Android

• [ ] iOS

• [ ] tvOS

• [ ] Linux

• [ ] OSX

• [ ] Windows

• [ ] Windows UWP

• Operating system version/name:9.1.1(Android TV 11)

• Kodi version:Kodi 21 20230613-2307e4d6

• inputstream.adaptive version:v21.1.1

note: Once the issue is made we require you to update it with new information should that be required. Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

[Kampfader]

I can confirm the error. I wanted to open an issue about it a few days ago, but couldn't get a useful debug log with the stupid firetv stick 4k. Hardware to slow for debuging.

I tested Disney (StarWars movies) play 5min And netflix movie (the mother) play 10min

[CastagnaIT]

please test with this test build (i deleted my previous message, different link): https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1292/7/artifacts

[Liqianyu]

@CastagnaIT I tested this build, but it didn't work. About 1 minute and 30 seconds into playback, frames start dropping. Eventually after several severe frame drops Kodi freezes.

[Ronny-nerd]

Hardware: Nvidia Shield TV 2015 and Nvidia Shield TV Pro 2019

Nightly Kodi 21 Omega

inputstream.adaptive Version: 21.1.2 = Not OK inputstream.adaptive Version: 21.1.0 = OK

Netflix Addon = 4K Movies For example

A Beautiful Life Blood & Gold The Mother

Disney+ Addon = 4K Movies For example

Star Wars: Das Erwachen der Macht ( Episode VII ) Star Wars: Die letzten Jedi ( Episode VIII ) Star Wars: Der Aufstieg Skywalkers ( Episode IX )

Have the same problem and can confirm it.

[Ronny-nerd]

Hardware: Nvidia Shield TV 2015

Nightly Kodi 21 Omega Alpha inputstream.adaptive version: 21.1.2

Netflix Addon = 4K movies = Not OK Disney+ Addon = 4K movies = Not OK Amazon VOD Addon = 4K movies/series = OK

4K movie after a few minutes the picture and sound freezes and then you are out of Kodi.

Here is a Kodi Debug LOG with Netflix Addon: https://paste.kodi.tv/eyuvohebul.kodi

Here is a Kodi Debug LOG with Disney+ Addon: https://paste.kodi.tv/okubagelaz.kodi

Translated with www.DeepL.com/Translator (free version)

[CastagnaIT]

i dont have any 4k account and no 4k tv on my hand so i am going blind on some log show some kind of problem with ae sync for audio but its weird that the problem happens only for some streams

my first suspect for now is for PR 1238 i tried change some code by using a vector buffer no idea if could improve it you can test here: inputstream.adaptive-android-armv7-omega-078b8e1.zip inputstream.adaptive-android-aarch64-omega-078b8e1.zip

note test will not works with D+ because the code change is incomplete, you can test only with N€tflix

[Ronny-nerd]

Thanks for your help, just tested it but unfortunately it did not solve the problem.

[CastagnaIT]

there is no problems with Kodi 20?

[Ronny-nerd]

With this InputStream Adaptive version, there are no more problems under Kodi 20 Nexus on the Nvidia Shield TV 2015 and the Nvidia Shield TV Pro 2019.

Netflix Addon = 4K films/series = OK Disney+ Addon = 4K movies/series = OK Amazon VOD Addon = 4K movies/series = OK

https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/Nexus/275/artifacts

[Kampfader]

there is no problems with Kodi 20?

with the crashes, there are no problems.

I've now watched two films of 20-30 minutes each. only N€tflix

N€tflix The Mother and A Beautiful Life, Audio Passthrough-OFF == OK Audio Passthrough-ON (AudioTrack(RAW), Android IEC packer) / Dolby Atmos was recognized == Not OK (after jumping forward 10min) logcat= https://paste.kodi.tv/ovepojirum.kodi

Amazon The Ground Tour, Audio Passthrough-OFF == OK Audio Passthrough-ON (AudioTrack(RAW), Android IEC packer) == Not OK (after jumping forward 10min) logcat = https://paste.kodi.tv/alicuqewek.kodi

Disney Star Wars IX, Audio Passthrough-OFF == Not OK (play max 5min) file to big https://www.file-upload.net/download-15159229/logcat.txt.html

[Ronny-nerd]

But maybe @glennguy has another idea to solve the problem in this direction:

Kodi 21 Omega: inputstream.adaptive Version: 21.1.2

[glennguy]

Ok so might need some more info.

@Ronny-nerd could you please also supply the manifest file? In inputstream.adaptive settings you can turn on the option to save the manifest file, then please attach here.

Also can you try switching the audio track between 6 channel and 2 channel and see what the behaviour is?

I'm looking at the manifest in the OP and seeing that the NF video played there only has EC3 tracks, yet... CDVDAudioCodecFFmpeg::Open() Successful opened audio decoder aac Maybe parsing or setting the wrong codec info @CastagnaIT ? Have to go to bed but can look more later.

[CastagnaIT]

Maybe parsing or setting the wrong codec info @CastagnaIT ? Have to go to bed but can look more later.

mmh i will try check tonight or tomorrow

[Ronny-nerd]

Hope that worked out with the manifest file.

https://www.dropbox.com/s/32qsyibxk7cod1z/manifests.zip?dl=0

[Ronny-nerd]

Können Sie auch versuchen, die Audiospur zwischen 6-Kanal und 2-Kanal umzuschalten und zu sehen, wie sich das verhält?

Had changed it in the meantime, but the same problem is still there.

Nvidia Shield TV 2015 is connected directly to the HDR monitor.... no AVR or soundbar is used.

manifests.zip

[Kampfader]

A question for the developers, I often see the manifest data being distributed via .zip.

When size allows, I always use paste.kodi.tv The question is, will the manifest text be edited again in paste.kodi? And therefore not presented in a way that is easy to understand for development?

[CastagnaIT]

so by reading n€tflix log at comment https://github.com/xbmc/inputstream.adaptive/issues/1295#issuecomment-1595710628 UpdateSampleDescription: codec 1836069985 == mp4a more likely has been selected the stereo audio track in that log

but in the log attached in the Issue description: UpdateSampleDescription: codec 1700998451 == ec-3 so same to my shield despite i cannot get atmos but only dolby 5.1

i think its not a problem of wrong format type

i noticed there are tons of prints: CAEStreamParser::TrySyncAC3 - Unexpected stream type: 0 (wantEAC3dependent: true) that is not printed if you use an old kodi 21 nightly e.g. (Git:20230423-6e84065c25)

however maybe it could means some data problem? maybe slightly corrupted? consider this affect both HLS (D+) and DASH (n€tflix) there should be some common code used that has been changed that cause this problem

@Kampfader depends from use cases, for me pastebin its not safe to use with different text encoding than utf8 or text that contains particular unicode chars like some subtitles, e.g. SmoothStreaming manifest its not safe use pastebin due to different encoding but for HLS/DASH ok, pastebin avoid us to download many times a file that often need only to be read temporary, but also have an expire date so over time it will no longer be possible to make comparisons, at the end its also your choice

[CastagnaIT]

a question you are using "android IEC packer"? if so go to kodi audio settings and set "Kodi IEC packer" and try again

i have no way to investigate on this the way to try find the problem is that someone test each code change starting from last commit of v21.1.0 then build each PR until the problem appears

[Liqianyu]

No, I always use Kodi IEC Packers. at this time I am unable to test further and hope someone else can. I agree that the best way to troubleshoot the issue is to try all commit changes for v21.1.0-v21.1.1 until we find the problem. It shouldn't take long to use the dichotomy.

[Kampfader]

Does it really run without crashing with 21.1.0? I haven't tested the version myself, I'll do it today after work. I use the last kodi nightly for

[Liqianyu]

Someone on the Kodi forums reported the failure as a memory leak issue.

[CastagnaIT]

i made some incremental builds if someone can test and report what works or not

i suggest to start test from the "test 5" and when fail try olders no need to test all use cases only one e.g. only n€tflix

inputstream.adaptive-android-aarch64-omega-d660799_test5.zip inputstream.adaptive-android-aarch64-omega-f5adcc9_test4.zip inputstream.adaptive-android-aarch64-omega-65dbb6f_test3.zip inputstream.adaptive-android-aarch64-omega-4b6ca27_test2.zip inputstream.adaptive-android-aarch64-omega-454eaf0_test1.zip inputstream.adaptive-android-aarch64-omega-77a58c0_test0.zip (this is same of 21.1.0)

[Ronny-nerd]

Hardware: Nvidia Shield TV 2015 My test:

_test0.zip (this is same of 21.1.0) = OK _test1.zip = OK _test2.zip = Not OK _test3.zip = Not OK _test4.zip = Not OK _test5.zip = Not OK

LOG _test2.zip = Not OK

Problem InputStream Adaptive.zip

[Ronny-nerd]

_test1.zip = OK

Netflix Addon = OK Disney+ Addon = OK Amazon VOD = Not OK (HDR)

[CastagnaIT]

please test also this same as before with n€tflix inputstream.adaptive-android-aarch64-omega-b4b069f_test6.zip

[Ronny-nerd]

_test6.zip = Not OK

After one minute System System memory utilisation: over 90%

After two or three minutes = Kodi Crash

[Ronny-nerd]

LOG _test6.zip = Not OK

Problem 2 InputStream Adaptive.zip

[CastagnaIT]

luckily is not mutex problem, apparently is due to parser rework of PR 1067 but that pr dont touch segment data management difficult to understand

with test6 if you set 1080p what happens? do you still see a progressive increase on occupied memory (during playback)?

[Ronny-nerd]

What happens to test6 if you set it to 1080p? Still seeing a progressive increase in used memory (during playback)?

After 8 - 10 minute System System memory utilisation: = 67%

[Ronny-nerd]

With this InputStream Adaptive version there are no more problems with Kodi 20 Nexus on the Nvidia Shield TV 2015 and the Nvidia Shield TV Pro 2019.

Netflix addon = 4K movies/series = OK Disney+ addon = 4K movies/series = OK Amazon VOD Addon = 4K Movies/Series = OK

https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/Nexus/275/artifacts

2160p After 8 - 10 minute System System memory utilisation: = 75-76%

[CastagnaIT]

I cannot understand it this way values above can be regular due to internal buffering

you should use keyboard press "O" to show the memory occupied when playback start then keep GUI panel info on screen during playback and check during playback if increase with regular pattern

if there is a mem leak the memory occupied will increase until the end of video, or until a crash when is almost full

[Ronny-nerd]

you should use keyboard press "O" to show the memory occupied when playback start then keep GUI panel info on screen during playback and check during playback if increase with regular pattern

I did, but those were the values at 1080p, just as those were the values under Nexus at 2160p.

[Ronny-nerd]

Will there be 32-bit test versions of 21.1.0 available for testing in addition to the 64-bit test versions?

[Ronny-nerd]

luckily is not mutex problem, apparently is due to parser rework of PR 1067 but that pr dont touch segment data management difficult to understand

Hello @CastagnalIT and @glennguy, are there already new findings about version 21.1.0 _test2.zip = Not OK up to version 21.1.2 = Not OK that include this problem and then crash Kodi?

Sorry my english is not good.

[CastagnaIT]

as said above apparently is due to parser rework of PR 1067 but that PR is a very huge rework and its not possible to try-test and analyse individual parts

since its limited to android and with 4k payed accounts its also hard to debug... i have no idea on what and where the problem could be because this problem to me its not clear i ran out of ideas, then from my part i will not do anything else at least for the moment

[Kampfader]

I don't know if it will help you further. I have a ftp app, running on the cube. A second before the kodi crashes, crash my ftp connection. I have me with that @Ronny-nerd converse. He doesn't have an ftp app on his device.

[Liqianyu]

Apparently there is a memory leak in ISA and Kodi crashes when it runs out of memory, and Android terminates background programs to reclaim memory. What's strange is that the problem only occurs with some content, not all 4K content encounters this problem. Memory leaks are not easy to locate, so we need to keep this issue open.

I think we can start by listing the affected platforms and the content for now.

For example Android Netflix 'Black Mirror' Season 6 4K Bitrate(Optional) Time(Crash time Optional)

[CastagnaIT]

Android Netflix 'Black Mirror' Season 6 4K Bitrate(Optional) Time(Crash time Optional)

can you explain better what means "Optional"?

[Liqianyu]

Android Netflix 'Black Mirror' Season 6 4K Bitrate(Optional) Time(Crash time Optional)

can you explain better what means "Optional"?

Bitrate: Does it happen when a certain bitrate is required? Time: Approximately how long it will take for the crash to occur Optional means that it can not be filled in, because the highest bitrate is considered by default and a crash will occur soon.

[Ronny-nerd]

[CastagnaIT]

when you have time please try to see if there is some improvement with this test build: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1310/1/artifacts

[Kampfader]

Im testing the 239525e

N€tflix - 4k The Mother 30min viewed = no crash System memory usage: max 90% There is only 3% left to crash, it could be possible that it will crash in an hour run time.

Amazon Vod - 4k StarWars IX with and without HDR10 = crash after 2min System memory usage: max 93%

Disney - 4k Avatar = crash after 2min System memory usage: max 93%

I tested the v21.1.0 to see how far going the system memory usage. Max was 65%

[Ronny-nerd]

Wenn Sie Zeit haben, versuchen Sie bitte, mit diesem Test-Build zu sehen, ob sich eine Verbesserung ergibt: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1310/1/artifacts

Tested it and still have the same problem with this version.

[Ronny-nerd]

But the hope remains that you will solve the problem.

[MauriceW67]

I did some testing of my own with the Disney+ addon and Kodi Omega Alpha 2.

I don't think the problem has something to do with the bitrate of the video, but rather the length of the video.

With the movie Avatar, the problem also occurs if I choose the 2560x1440 stream with a bitrate of 15 or 18 Mbit/s. Same thing happens with the movie Black Panther: Wakanda Forever. With these movies the memory usage starts at over 80% and quickly rises to 93-95% after which Kodi crashes to the Shield's homescreen.

But if I play a TV show episode of about 40 mins in 4K with a bitrate of 23-24 Mbit/s, the problem does not occur and the memory usage stays at about 73-74%.

Also did some tests with Kodi Nexus and IS:A 20.3.9 and no issues there.

EDIT:

Did some more tests with the Netflix addon and that seems to behave a bit different. I played two episodes of Altered Carbon Season 1 (episode 6 with a length of about 46 mins and episode 7 with a length of about 1h07.).

In both cases the 18 Mbit stream made the memory usage go up to 94-95% and Kodi crashed. The 10 Mbit stream kept playing, although the memory usage did grow to about 83-85%.

[Ronny-nerd]

In this series "Welcome to Eden" Kodi does not crash.

[Liqianyu]

It doesn't look like this should be a memory leak failure because certain content also takes up higher memory but the memory footprint doesn't keep growing. If the memory is large enough (looks like it needs to be 4GB?) then memory exhaustion will not occur and it will not cause Kodi to crash.

However mainstream Android TV set-top boxes only have 2-3G of RAM, which is why the issue only seems to affect devices like Android at the moment, as they have less RAM.

[Ronny-nerd]

For comparison, the Amazon VOD addon with the movie "Dune (2021)", Kodi does not crash.

Installed InputStream Adaptive version now

https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1311/4/artifacts

[CastagnaIT]

@glennguy i think i have found a way to investigate also under Windows os

by using following code snippet you can get HLS manifest from youtube (manifest url expire)

from yt_dlp import YoutubeDL # yt-dlp - https://github.com/yt-dlp/yt-dlp
url = "https://www.youtube.com/watch?v=Hf5enZVznC4"
ydl = YoutubeDL({'format':'bestvideo*+bestaudio/best'})
result = ydl.extract_info(url, download=False)
for format in result['formats']:
    if "manifest_url" in format.keys():
        print(f"found a manifest_url {format['manifest_url']}\n")

specified yt link on the code above, is a video of ~1h with 8K resolution, (i have taken last url of the manifest list generated) now playing the 8k resolution with ISA (ask quality) on Window 11 i have these results:

• Omega master branch: 3 minuts played, Kodi filled 1700Mb RAM (1,7gb)
• Omega at version v21.1.0 + recent VP9 commit patch: 3 minuts played, Kodi filled 860Mb RAM

i can see the gradual memory increase over the time i hope this makes an easier investigation, since it dont require DRM we can also debugging

[glennguy]

@CastagnaIT awesome work! And thanks to everyone else for persisting in looking for a solution. I'll dive in to this tomorrow night