Some HLS WebVTT subtitles doesn't work

[Dis90]

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

WebVTT subtitles in some HLS streams doesn't work.

(Case 1) On Kodi 19 and Kodi 20 these doesn't work. Kodi 19 doesn't show subtitles and on Kodi 20 video playback doesn't start. prog_index.txt caption_0.txt caption_1.txt master.txt Log Kodi 19: kodi19.log Log Kodi 20: kodi20.log

(Case 2) These does work on Kodi 19 but doesn't work on Kodi 20 (subtitles is listed in subtitles menu) subs.txt caption_0.txt master_new.txt Log Kodi 19: kodi19_new.log Log Kodi 20: kodi20_new.log

Expected Behavior

Here is a clear and concise description of what was expected to happen:

Subtitles to work.

Actual Behavior

Possible Fix

To Reproduce

Steps to reproduce the behavior:

Debuglog

The debuglog can be found here:

MPD/M3U8s/ISM

An example or copy of a manifest (or manifests for HLS - master and variants) can be found here:

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Your Environment

Used Operating system:

• [ ] Android

• [ ] iOS

• [ ] tvOS

• [ ] Linux

• [x] OSX

• [ ] Windows

• [ ] Windows UWP

• Operating system version/name:

• Kodi version: 19 and 20

• inputstream.adaptive version:

note: Once the issue is made we require you to update it with new information should that be required. Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

[CastagnaIT]

i can check the 2nd case These does work on Kodi 19 but doesn't work on Kodi 20 (subtitles is listed in subtitles menu) for Kodi 20

the other one for Kodi 19 from my part i will not make any changes the parser used is too unstable and incomplete

[CastagnaIT]

if i provide you a Kodi 20 build for Windows, are you able to test it?

[Dis90]

Yes, I have old Windows laptop which I can use for testing.

[CastagnaIT]

Good, this is the test build of Kodi 20 (64bit) extract it in a new folder then start Kodi by running _RunAsPortable.bat file, to start kodi as portable mode

build: https://www.dropbox.com/s/5ig2t3foez3zwo4/KodiPortable_Builded_v20_HLSwebvtt.zip?dl=1

This should fix the case 2 (These does work on Kodi 19 but doesn't work on Kodi 20 (subtitles is listed in subtitles menu)) only please check if the subs are sync with the video

I have also saw in the log some other warnings like this: WARNING <general>: DecodeLine - Malformed cue, is missing the subtitle text

this may result in the loss of some subtitles because some webvtt segment file seem to be malformed if you can attach here also the other files caption_1.vtt and ...2 and ...3 i would like to check these files

[Dis90]

Tested and subtitles are now working and in sync with video for case 2.

Here is caption.vtt files: caption_0.txt caption_1.txt caption_2.txt caption_3.txt

[CastagnaIT]

perfect thanks i will push the changes on Kodi, maybe @glennguy can suggest how to proceed to check the problem in the Case 1, where the video not starts

[glennguy]

@Dis90 unfortunately the logs don't say much about why the video won't start. But seeing as it's going ahead and downloading hundreds of segments for subtitles - way beyond the 120 seconds buffer

@CastagnaIT - I had this commit to fix the ISA parser not looking at the LOCAL parameter: https://github.com/glennguy/inputstream.adaptive/commit/661ded45d84bb39dfffc7dc726ca4d7621cbd337 which never made it in as I still had issues (which I think are now solved from that patch I did a month or so ago)... does the Kodi implementation look at these values now?

[Dis90]

Is there something I can do to make logs show more info? After clicking tabulator I can see video controls (play, pause etc.) but video doesn’t play.

[CastagnaIT]

@glennguy in the webvtt parser that i have implemented on kodi i had no way of testing this HLS case, so i had not completed this code, after i completely forgot it... However yes looks at them, i have add/fix the same things on kodi parser

[Dis90]

Here is screenshot how things look. After video starts busy spinner spins forever but if I click backspace and after that tabulator I can get video controls to show.

[CastagnaIT]

in my tests when was deleting the parser on ISA I had noticed that downloading segments can cause a delay in the execution of the video the specifications say that you should only download the required part of segments and not all segments at once

now that i see from prog_index.txt this video have 1327 segments (files) a little long to handle all these files in one time

[CastagnaIT]

maybe I'm saying something dumb... but if we theorize that all segments are downloaded at same time is possible that the #EXTINF: value is badly handled?

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:2
#EXTINF:2.000,
caption_0.vtt?hdntl=exp=1641511966~acl=/73f4c9e7-eeb8-4f1f-bde1-4fa743e25cd7/0ec14928-6f4c-4393-9c88-41cd557ebac8/hls_clear_ts/*~data=hdntl~hmac=9b20dad78a080dc2cd028619be5e7491e28d559d92774692a424b3a694eaa1ff
#EXTINF:2.000,
caption_1.vtt?hdntl=exp=1641511966~acl=/73f4c9e7-eeb8-4f1f-bde1-4fa743e25cd7/0ec14928-6f4c-4393-9c88-41cd557ebac8/hls_clear_ts/*~data=hdntl~hmac=9b20dad78a080dc2cd028619be5e7491e28d559d92774692a424b3a694eaa1ff
#EXTINF:2.000,
caption_2.vtt?hdntl=exp=1641511966~acl=/73f4c9e7-eeb8-4f1f-bde1-4fa743e25cd7/0ec14928-6f4c-4393-9c88-41cd557ebac8/hls_clear_ts/*~data=hdntl~hmac=9b20dad78a080dc2cd028619be5e7491e28d559d92774692a424b3a694eaa1ff
#EXTINF:2.000,
caption_3.vtt?hdntl=exp=1641511966~acl=/73f4c9e7-eeb8-4f1f-bde1-4fa743e25cd7/0ec14928-6f4c-4393-9c88-41cd557ebac8/hls_clear_ts/*~data=hdntl~hmac=9b20dad78a080dc2cd028619be5e7491e28d559d92774692a424b3a694eaa1ff

all are 2secs maybe all are downloaded at same 2sec pts? instead of 2,4,6,etc...

[glennguy]

Pts accumulates so that's not the issue, the tag is announcing the length of the below segment

[glennguy]

First guess for me would be that IA is searching for the pts and can never seem to seek through enough segments to ever find it.

@Dis90 it's all good, I was just thinking that it's hard to debug when we don't have access to the service, and nothing obvious in the logs.

[CastagnaIT]

also with dysn€y+ happens same thing all segments are downloaded in full at starts, but i cannot test on d+ anymore i have deleted the account

[glennguy]

ok good to know, I think I still have an account with them to test with.

[CastagnaIT]

i have just found a free hls webvtt segmented: https://playertest.longtailvideo.com/adaptive/elephants_dream_v4/index.m3u8 result to be:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:150
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:150.0,
subtitlechunk_lfra_w1588523518_b160000_slen_t64RW5nbGlzaA==_0.webvtt
#EXTINF:150.0,
subtitlechunk_lfra_w1588523518_b160000_slen_t64RW5nbGlzaA==_25.webvtt
#EXTINF:150.0,
subtitlechunk_lfra_w1588523518_b160000_slen_t64RW5nbGlzaA==_50.webvtt
#EXTINF:150.0,
subtitlechunk_lfra_w1588523518_b160000_slen_t64RW5nbGlzaA==_75.webvtt
#EXT-X-ENDLIST

i think you can use this

[glennguy]

Without looking right now (but remembering what I know) there's a few possibilities.

I will double check tomorrow evening but in my mind here are some possible scenarios

• ISA is not observing the settings for max buffer length when it comes to subtitles
• VideoPlayer is not satisfied asking for subtitle demux packets until the whole file is loaded (and in the OP case perhaps after downloaded all 1000+ segments the file will start playing)
• Some other issue in ISA that keeps it looping until EOS is found

[glennguy]

Ok can't help myself so had a quick look at code before bed.

My guess is here https://github.com/xbmc/inputstream.adaptive/blob/ed714ad56a9e36d46f153bfa750d82b10de264b5/src/main.cpp#L3903 is always (after giving a video and audio) returning the subtitle sample reader

Because here https://github.com/xbmc/inputstream.adaptive/blob/ed714ad56a9e36d46f153bfa750d82b10de264b5/src/main.cpp#L3071 the PTS/DTS of the sub sample reader isn't being updated

[Dis90]

VideoPlayer is not satisfied asking for subtitle demux packets until the whole file is loaded (and in the OP case perhaps after downloaded all 1000+ segments the file will start playing)

I tested this and after waiting few minutes video starts. Subtitles doesn't work on my Mac but I tried Windows and @CastagnaIT test build and with it subtitles works but only for language what Kodi chooses to use (media default or what language I've chosen to use). Maybe other languages also works if I wait long enough after change?

[CastagnaIT]

@Dis90 due to the higher delay in theory yes, because when you change subtitles you clear all data

@glennguy i have made a test, i have found kind of fix, but only for the packets set to kodi, as you said the pts was not set, if i add the pts like this subtitle segments packets are sent to kodi while in playback instead all in one at video starts: https://github.com/CastagnaIT/inputstream.adaptive/commit/eab8488b2327a0c6fed08bd54578b77eff25d66d I don't know the best way to do this, but the pts here was forced to "0", of course this fix not fix downloads that still happens all in one at video starts

if you try this fix, you can see also a side effect, when you seek foreward/backward the video multiple times, the packet of a subtitle segment already sent to kodi, seem to be sent again, causing to display subtitle text twice or threefold etc... this because of missing "flush" operation from kodi parser, I don't think it's a big problem to solve, but I think it is better to talk about this only after the download problem has been fixed

[glennguy]

@CastagnaIT I didn't have access to your proposed fix on Kodi's side but using Matrix with 20.0.2 I think the change needed is just to change here https://github.com/xbmc/inputstream.adaptive/blob/ed714ad56a9e36d46f153bfa750d82b10de264b5/src/main.cpp#L1609 to m_codecHandler->Transform(m_pts, m_sample.GetDuration(), result, 1000); - the other modifications in your branch were unnecessary.

With that I can see now that DemuxRead is only getting the subtitle packets when they're needed, but it's still grabbing all available (only 4) segments so that could be still an issue in AdaptiveStream.

[glennguy]

I think there's no further issue, just that with the current design the minimum amount of buffers for a stream is 4.

Aside from that, to add further comment to the above one - you can see that the webvtt segment downloading now 'takes turns' with the audio and video, rather than downloading all sub segments one after the other without other things happening.

[CastagnaIT]

to m_codecHandler->Transform(m_pts, m_sample.GetDuration(), result, 1000); - the other modifications in your branch were unnecessary.

@glennguy your proposed change not works, because in this use case we have segmented files, and are handled by AP4_ByteStream* input contructor then not url constructor. You can see also by yourself by appling this kodi patch to keep track of the received segmented packets: https://github.com/CastagnaIT/xbmc/commit/91180642f9be8c38fa02cfde9705f56fc19baa7a

with your change in the log output you can see all 4 segmented packet received all in one (+ skipped):

2022-01-10 13:43:29.163 T:4284    DEBUG <general>: CApplication::OnAVChange: "CApplication::OnAVChange"
2022-01-10 13:43:29.164 T:23304   DEBUG <general>: COverlayCodecWebVTT::Decode - PACKET RECEIVED
2022-01-10 13:43:29.170 T:23304   DEBUG <general>: Skipped 3 duplicate messages..
2022-01-10 13:43:29.170 T:23304   DEBUG <general>: VideoPlayer::Sync - Video - pts: 42000.000000, cache: 50000.000000, totalcache: 100000.000000
2022-01-10 13:43:29.171 T:23304   DEBUG <general>: CVideoPlayer::SetCaching - caching state 0

this not happens with my commit change (where the packets are send to kodi only when needed)

PS: the download of files segments are executed from AdaptiveStream::download_segment()

I think there's no further issue, just that with the current design the minimum amount of buffers for a stream is 4.

ah now i have understand the 4 download at once is due to min buffer, in future these magic numbers in the code have to take place with a proper constant on an unnamed namespace

[CastagnaIT]

@Dis90 with the build for windows that i provided previously please update ISAdaptive with this build: inputstream.adaptive_patched.zip (you can install it via GUI) then provide a feedback with a log for the case 1

[Dis90]

Didn't work or I'm doing something wrong.

2022-01-10 15:45:28.453 T:8068 ERROR : Win32DllLoader::Load: Failed to load "C:\Users\tin_d\Downloads\KodiPortable_Builded_v20_HLSwebvtt\portable_data\addons\inputstream.adaptive\inputstream.adaptive.dll" with error 126: "Määritettyä osaa ei löydy.

kodi.log

[CastagnaIT]

This should works: inputstream.adaptive_patchedRel.zip

[Dis90]

Now video starts immediately and subtitles works but only for first two sentences (I think). kodi_new.log

Edit. If I change subtitle language while playing I can get other language to display but again only for two sentences and when changing back to preferred language Kodi displays two sentences and then stops to display subtitles.

[CastagnaIT]

Try also with this: inputstream.adaptive_patched_2.zip

[Dis90]

This time Kodi crashes after two sentences of subtitles kodi_new2.log

Do you need kodi_stacktrace or kodi_crashlog...dmp?

[CastagnaIT]

no not needed, the problem seem always around the pts value to be set but compared to the test stream, your video has different behaviours ATM i have no others ideas let's see if glennguy find other

[glennguy]

@CastagnaIT my apologies - I actually quoted the wrong line of code in my post above (used the correct one however in my local build). I meant to say modify this line https://github.com/xbmc/inputstream.adaptive/blob/8df723673134b3c1ddccf4c3ab2a776980f0319a/src/main.cpp#L1665 to m_codecHandler->Transform(m_pts, m_sample.GetDuration(), result, 1000);

Can't be 100% sure with the discovery one with hundreds of segments but again it looks to do the job on the test stream. Tested with your merged PR built from source.

[CastagnaIT]

this change:

m_codecHandler->Transform(m_pts, m_sample.GetDuration(), result, 1000);

is impossible that works, m_pts is populated by m_sample.GetCts() * 1000 that happens after, and GetCts return the value provided from Transform(0, ...) (before forced to 0) then if you have change to Transform(m_pts, ...) you will create a variable loop of "0" value...

in fact the log result the same wrong behaviour:

2022-01-12 13:24:57.321 T:9152    DEBUG <general>: CVideoPlayer::HandleMessages - player started 2
2022-01-12 13:24:57.321 T:9152    DEBUG <general>: COverlayCodecWebVTT::Decode - PACKET RECEIVED
2022-01-12 13:24:57.330 T:9152    DEBUG <general>: Skipped 3 duplicate messages..
2022-01-12 13:24:57.330 T:9152    DEBUG <general>: VideoPlayer::Sync - Video - pts: 42000.000000, cache: 50000.000000, totalcache: 100000.000000

also add m_sample.GetDuration() to Transform(0, ...) is a loop of "0" value, for the same reason

i think that an idea could be popupate the pts/duration by taken the values from the m3u8 segments list data i have made a try in test mpd has worked but in the @Dis90 test has crashed... i suspect because some variable pointers was inacessible

[Dis90]

For my point of view it's not critical to get case 1 working. By default discovery+ add-on uses DRM/MPD streams (website also) and subtitles are now working in Kodi 19 and 20 for those.

HLS is only used if users sets DRM to disabled. Previously user had to set DRM disabled in order to get subtitles to work on newer videos (Kodi 19). Older videos are in case 1 format.

I'm not even sure if discovery+ uses non DRM HLS streams even if they provide them.

[CastagnaIT]

yes may not be prioritary to fix when i implemented the new webvtt parser on Kodi i had already noticed this problem with Disn€y+ but i didn't go into it because I'm not very knowledgeable about various aspects of ISA

as is, it can work fine only if an HSL stream has few segments because the problem stay hidden, but for streams like your with 1000+ segments the problem become heavy

however other video services may have many segments, then sooner or later this need to be fixed since you can provide us support for testing it would be useful to find a solution

[Dis90]

Yeah of course I can help with the testing.

[glennguy]

@CastagnaIT you're completely right, I thought I observed a change in behaviour but it's just sending 0 again. Learning all the time! Will have another look tonight and see what I can come up with.

Also

i think that an idea could be popupate the pts/duration by taken the values from the m3u8 segments list data

Yes this is what using m_ptsOffset is effectively doing

[glennguy]

@CastagnaIT I think your proposed changes are the best we can have, there is no other way to get timing information other than the playlist->segment times. Please PR when you're ready :)

[CastagnaIT]

my initial solution: m_codecHandler->Transform(m_ptsOffset, 0, result, 1000); could works, but we cannot full test it now, because we havent the possibility to disable the adaptive stream switching on ISA and/or having a solution to DEMUX_SPECIALID_STREAMCHANGE problem on kodi side, the DEMUX_SPECIALID_STREAMCHANGE still fuckup the kodi parser and the subtitle data at every call...

as @Dis90 said only first 2 sentenced are shown on screen, from the log you can see DEMUX_SPECIALID_STREAMCHANGE call who almost certainly messed up

for test purpose i have also found another HLS stream of 30 minuts lenght with webvtt segmented: https://s3.amazonaws.com/_bc_dml/example-content/bipbop-advanced/bipbop_16x9_variant.m3u8 this have 60 webvtt segments maybe this is a better test for the segments, but in every test that i done with this stream, the DEMUX_SPECIALID_STREAMCHANGE call still happen causing problems, plus for some reason the first subtitle segment is not displayed (not check possible cause yet)

[CastagnaIT]

however before the DEMUX_SPECIALID_STREAMCHANGE call seems that the segments are downloaded and sent correctly, without a way to disable adaptive stream the only way to be sure is apply my PR https://github.com/xbmc/xbmc/pull/20579 on ISA/Kodi to see if works I will see soon...

[glennguy]

I know this needs to be made a setting but it's fairly easy to make a small hack to turn it off.

New representations are chosen here https://github.com/xbmc/inputstream.adaptive/blob/8df723673134b3c1ddccf4c3ab2a776980f0319a/src/common/AdaptiveStream.cpp#L558-L575

Just comment out and replace with newRep = current_rep_;

I was testing too last night with a multi period HLS with webvtt using LOCAL parameter, eventually after seeking around enough the subs would stop. I'll try to investigate some more.

[CastagnaIT]

thanks! now it's a bit easier do tests

I have made an almost complete solution this seem to works, I had to revert the #841 was causing problems to segments but now there is a problem to solve with the video seek if you rewind forward, there is no problems but if you rewind backward (at a point where you did not play the video) the subs packets are no longer sent to kodi for 1 or 2 minuts, can you have a look at seek with my commit? https://github.com/CastagnaIT/inputstream.adaptive/commit/a7d7c3f11c8a4616876e139777ec847cc562fc7a

@Dis90 can you install this update and test it for case 1? in theory this should works, please do not use video seek for this test inputstream.adaptive_patched_3.zip

[Dis90]

Tested and subtitles works now without crash. Do you want log?

[CastagnaIT]

@Dis90 not needed thanks before you will ask please note that when this fix will be done you will still have broken subtitles due to stream adaptive change (not fixed yet, in this test has been forced "disabled")

[CastagnaIT]

@glennguy sorry for the noise i think i have just fixed also the seek problem, I will double-check the changes i need to do more tests also with single file webvtt to be sure after i will prepare the PR

[CastagnaIT]

Issue closed all bugs has been fixed on Kodi 20