Overlay container become cleared and prevents webvtt subtitles from being displayed

CastagnaIT commented 11 months ago

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

An user reported to InputStream Adaptive a bug that prevent webvtt subtitles to beign displayed, so no subtitles at all... ref. https://github.com/xbmc/inputstream.adaptive/issues/1419 there could be more than one bug, but this case need to be fixed as first before all

So i tried to investigate but i need someone help me or someone that make the fix

Before i explain the steps to reproduce it, because it is not so immediate to do:

Use video stream from this STRM

#KODIPROP:inputstream=inputstream.adaptive
#KODIPROP:inputstream.adaptive.manifest_type=hls
https://playertest.longtailvideo.com/adaptive/elephants_dream_v4/index.m3u8

Install ISAdaptive add-on
Ensure that Kodi subtitle language is set to "English", its not so important the language set, but important is start the video with subtitles disabled, so a mismatch of language will do this
Start the STRM video, enable subtitle immediately by using GUI, and without change subtitle track, when the old man speaks in the video, subtitles should be displayed, but ofc is not so
Before stop the video, remember to disable subtitles, otherwise next time video start with subtitles enabled
Do not use video resume, that may cause different behaviours

These are the circustances: When you start playback, Kodi ask to ISAdaptive the stream tracks (a/v + sub), this use case the sub track at video start its disabled, but anyway ISAdaptive feed kodi player demuxer with subtitles packets and you can see that webvtt data is parsed correctly in the COverlayCodecWebVTT despite "not used" by the OverlayRender because the m_buffers has been deleted, and the buffer loop: https://github.com/xbmc/xbmc/blob/21.0b1-Omega/xbmc/cores/VideoPlayer/VideoRenderers/OverlayRenderer.cpp#L146-L155 is so stopped (being stopped libass will not be called to generate subtitles).

Now when you turn subtitle ON by using Kodi GUI while in playback, kodi already have the parsed subtitles data in memory, but Kodi core call CloseStream it clear also the OverlayContainer here: https://github.com/xbmc/xbmc/blob/21.0b1-Omega/xbmc/cores/VideoPlayer/DVDOverlayContainer.cpp#L117 the overlay previously added is deleted and never readded, and the OverlayRender buffer loop dont run, with the result that no subtitles are displayed on screen. The only case when the overlay could be readded is when ISAdaptive read a new subtitle data packet, but this cannot happens (not immediately) because a single packet aready sent may contains minuts/hours of subtitles (in theory you could force read packet with a video seek)

Solutions i dont know exactly the best way the overlay need to be re-added or not deleted,

Debuglog

The debuglog can be found here: https://paste.kodi.tv/vuhihukuwo.kodi

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Your Environment

Used Operating system:

[ ] Android
[ ] iOS
[ ] tvOS
[ ] Linux
[ ] macOS
[x] Windows
[ ] Windows UWP
Operating system version/name:
Kodi version: Kodi 21 (not tested yet on Kodi 20 but should the same)

note: Once the issue is made we require you to update it with new information or Kodi versions should that be required. Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

CastagnaIT commented 11 months ago

i was reading again the log and i was thinking of another solution, but i havent go in deep, you can understand that when kodi have to open a stream make the OpenStream callback with also GetStream callback

when you turn ON / OFF subtitles you can look at similar bahaviour callbacks

2023-11-30 13:40:54.820 T:19496   debug <general>: AddOnLog: inputstream.adaptive: EnableStream(1005: false)
2023-11-30 13:40:54.826 T:19496   debug <general>: AddOnLog: inputstream.adaptive: OpenStream(1005)
2023-11-30 13:40:54.826 T:19496   debug <general>: AddOnLog: inputstream.adaptive: GetStream(1005)

with the difference that call also EnableStream callbacks (to disable previously used track/stream) but EnableStream has been forgotten when you start the playback or else so there could be another 2 solutions

solution 1 - perhaps the best

avoid make callbacks OpenStream/ GetStreamwhen you start playback with a disabled stream

2023-11-30 13:39:36.362 T:19496   debug <general>: AddOnLog: inputstream.adaptive: OpenStream(1005)
2023-11-30 13:39:36.364 T:19496   debug <general>: CurlFile::XFILE::CCurlFile::Open - <https://playertest.longtailvideo.com/adaptive/elephants_dream_v4/chinese/ed.m3u8>
2023-11-30 13:39:36.630 T:19496   debug <general>: AddOnLog: inputstream.adaptive: Download finished: https://playertest.longtailvideo.com/adaptive/elephants_dream_v4/chinese/ed.m3u8 (downloaded 429 byte, speed 1618.00 byte/s)
2023-11-30 13:39:36.632 T:19496   debug <general>: AddOnLog: inputstream.adaptive: Manifest saved to: D:\KodiPortable_Builded_20\portable_data\userdata\addon_data\inputstream.adaptive\manifests\manifest_1701347976_child-subtitle.txt
2023-11-30 13:39:36.634 T:19496   debug <general>: AddOnLog: inputstream.adaptive: GetStream(1005)

these two callbacks are done in any case when you re-enable the stream

so the current behaviour always enable the streams, regardless of whether they are enabled or not and it is also not efficient, because make start downloads and process data to demuxers for no reasons

solution 2

let know to binary addons when the video starts with subtitle disabled, then just after sent the first callback OpenStream (or maybe the first GetStream callback after the GetStreamIds callback) add an additional callback to EnableStream with "false" value, when the stream (id) must start as disabled

IMO these solutions could be the right way, i tried force the solution 2 with a rough fast test, and subtitles now are displayed correctly when you enable subs, but i havent test in deep yet so i have not yet investigated whether these solutions are applicable

CastagnaIT commented 11 months ago

so i discovered that there are two problems with subtitles for use cases like VOD HLS that have webvtt subs on separate packets on this video sample each packet/segment have (contains subtitles) for the timeline duration of 150 secs each one

problem 1) When you start playback OpenStream method is called to allow start/download the stream from ISA, but if playback is started with subtitles disabled, kodi dont let know to ISA that sub stream is disabled, this is the 1° Kodi bug.

problem 2) When you disable/re-enable subtitles, Kodi delete the overlay's in the overlay container, and wait for the next subtitle packet to add again the overlay, in order to display subtitles again. This generally its ok, for example in live, where the stream have very short buffer, then new subtitle packets are more most of time sent immediately, but for VOD HLS case this is different: streams like VOD HLS have packets, but the timeline of each packed may not be the same of the audio/video, so we download each packet in sequence, the problem is that when you disable subs, kodi kill the overlay with libass containing the parsed data, same thing when you switch sub track, the sub data is lost

Solutions: problem 1) i have opened a fix PR #24186 to add the missing callback

problem 2) this is more complex to found a solution, and currently i dont know yet what could be a good solution

CastagnaIT commented 10 months ago

For the unsolved problem n°2: i made a kind of workaround in to ISAdaptive, by storing locally the last pts fed to kodi DemuxRead callback and use this pts as reference where start get subtitle packages when are re-enabled https://github.com/xbmc/inputstream.adaptive/pull/1454/commits/6bc1df7903870c9ac3a9344ad4d6331c907aa460 It is not perfect but at the moment nothing better can be done at today, since kodi binary interface dont provide any kind of pts of reference, nor a way to know the current player time

so i close it, in the future we will see if/how it can be improved

xbmc / xbmc