search

xbmc / inputstream.adaptive

kodi inputstream addon for several manifest types
Other
446 stars 240 forks source link

[WebVTT][Dev-Question] Moving WebVTT parser to Kodi sources #773

Closed CastagnaIT closed 2 years ago

CastagnaIT commented 2 years ago

When the rendering transition to LibASS will be merged in kodi (currently opened Kodi WIP PR) I would like to move the WebVTT parser to Kodi source parsers

this would allow some improvements like

there are two problems that i require info, maybe someone hame more info than me:

1) How to allow Kodi FFMPEG to identify the ISA subtitles as WebVTT

currently ISA always send SRT string to kodi FFMPEG https://github.com/xbmc/inputstream.adaptive/blob/2.6.23-Matrix/src/TSReader.cpp#L81-L83 https://github.com/xbmc/inputstream.adaptive/blob/2.6.23-Matrix/src/TSReader.cpp#L135 Here kodi ffmpeg translate the SRT codec name string to the ffmpeg codec ID: https://github.com/xbmc/xbmc/blob/19.1-Matrix/xbmc/cores/VideoPlayer/DVDInputStreams/InputStreamAddon.cpp#L387-L397 then kodi the subtitles always as "subrip" subtitles

for this reason we should be add webvtt to the enum (first link^^). I have tried track the code to try understand where the enum value for the SRT enum is picked, and i found that come from /lib/mpegts/tsDemuxer.cpp i think from this method: https://github.com/xbmc/inputstream.adaptive/blob/Matrix/lib/mpegts/tsDemuxer.cpp#L917-L987 but i dont understand desc_tag byte code, where does it come from? could a new hex code be added to identify a new format?

2) After resolved step 1, pass the stream file to kodi, maybe for initial test can be reused current implementation with some change to send the header metadata before start sending the subtitles lines

time permitting, I am planning this change for the future but for now, I am only gathering information on how this could be done

glennguy commented 2 years ago

currently ISA always send SRT string to kodi FFMPEG

The 2 links below only relate to ts files which are only in HLS. IA can support INCLUDEDSTREAMs for the case where HLS has ts files that include video and audio, but subtitles are not implemented.

For fmp4, it will get set here: https://github.com/xbmc/inputstream.adaptive/blob/851d1c49c038704d106ffd8889ffb2c95e39524f/src/main.cpp#L2815-L2817 We set as srt for any case it seems.

CastagnaIT commented 2 years ago

thanks for this explanation i will try to change that line

yes every subtitle format is currently "converted" into "srt" by ISA, but after kodi re-convert the ISA "srt" data again before display on screen, it is a double step that can be avoided, we should let kodi natively manage subtitles formats types in this way we can provide a better support for subtitles types

after this i will have to figure out how to send the subtitle data stream without parsing it

CastagnaIT commented 2 years ago

i have finish to develop the real WebVTT parser in Kodi core, and currently is fully functional with external files (*.vtt), (although I have already started writing part of the implementation for now this not include WebVTT ISOBMFF format type, need to be developed after make full working the regular text based format type)

I am one step closer to eliminate the fake WebVTT implementation of ISA (●'◡'●)

I was able to route the raw data directly in to kodi core overlay codec parser, in this way: https://github.com/CastagnaIT/inputstream.adaptive/commit/bf0c6e1c26bdfa366753ee7a17782209bbc6f54c

this has worked correctly until now today i have tried rebase my branch to ISA "Nexus" branch but now this change make crash Kodi/ISA, my suspects are due to the new rework of adaptive streams

Log: https://paste.kodi.tv/agicebired.kodi

i see some CAddonVideoCodec: GetPicture: EOF after that crash

glennguy commented 2 years ago

Is the current Nexus release (i.e without your changes) crashing as well?

CastagnaIT commented 2 years ago

without my commit seem there are no crashes

glennguy commented 2 years ago

Do you have a crashlog dmp file you could attach here?

CastagnaIT commented 2 years ago

here, LOG: https://paste.kodi.tv/ebihutikez.kodi related DMP: kodi_crashlog-20.0-ALPHA1 Git_20211004-708c424318-20211005-082419.dmp.zip

glennguy commented 2 years ago

Hmm I haven't got your xbmc branch loaded so can't see anything, I would suggest building locally and running with the debugger. Try this unprotected test stream: https://dash.akamaized.net/akamai/test/caption_test/ElephantsDream/elephants_dream_480p_heaac5_1_https.mpd to avoid widevinecdm crashing the debugger

glennguy commented 2 years ago

Whoops lol, too slow - you found the issue

CastagnaIT commented 2 years ago

that was an error, but not works good again...

i had not yet tried to play a video for a few minutes only tried if starts, and i have found that the video stop without reason after about more or less a minute

I thought it was a problem with the code I'm working on, but it's other that has problems https://paste.kodi.tv/ajesoyeyoc.kodi because now i have tested on a fresh kodi 20 nightly + nfx

i noticed that after this line: DEBUG <general>: AddOnLog: inputstream.adaptive: DEMUX_SPECIALID_STREAMCHANGE (i do not know what this is this DEMUX_SPECIALID_STREAMCHANGE) seem that all streams loaded are resetted/reinitialized also the subtitle stream/rendering, this last one could be a problem with subtitles data, involves full parsing of subtitle data every time this happens also maybe lead to other side effects like delay to show subtitles and possible problems if is needed to keep some subtitle metadata in memory during the entire execution of the video because in this case they will be lost

if for DEMUX_SPECIALID_STREAMCHANGE means stream bitrate change, IMO at least the subtitle stream should remain intact without being reset

glennguy commented 2 years ago

yes DEMUX_SPECIALID_STREAMCHANGE is a special packet sent to Kodi to let it know to reset/reinitialize the player. Only video streams are bitrate switched, audio and subs are left alone.

From looking at the log when the switch occurs this happens in the log AddOnLog: inputstream.adaptive: WVDecoder initialization returned status 3

Status 3 is kInitializationError

Perhaps this may be related to the new pre init workflow you implemented? Maybe the init data is only making its way through for the initial stream? Just thinking out aloud here, haven't looked into it.

With the auto stream switching Kodi is only told about 1 video stream, in your case stream 1001. When the stream switches it closes 1001 and reopens 1001, this time it has segments from the new stream. I'm not sure if you can tell from the segment urls which dash representation they belong to, let me know if you want help getting some more info into the log.

CastagnaIT commented 2 years ago

Do you know a test MPD with subtitle stream of webvtt ISOBFF format type?

if i want to test a MPD there is a easy way to test it, or is needed some kind of test addon?

CastagnaIT commented 2 years ago

i have found this MPD http://media.axprod.net/ExoPlayer/Captions/Manifest.mpd should have webvtt ISOBFF format how i can test it?

glennguy commented 2 years ago

Just make a .strm file is the easiest way. Put this in a file called eg. test.strm :

#KODIPROP:inputstream=inputstream.adaptive
#KODIPROP:inputstream.adaptive.manifest_type=mpd
http://media.axprod.net/ExoPlayer/Captions/Manifest.mpd

Then open through file manager in Kodi. I usually dump them in the profile folder, easy to access.

CastagnaIT commented 2 years ago

thanks works! ~subtitles are listed but not displayed, something is not working i do not know~ I will try to investigate

update: the link above have mp4 with wrong mime type this one works: http://media.axprod.net/ExoPlayer/Captions2/Manifest.mpd

CastagnaIT commented 2 years ago

@glennguy for the DEMUX_SPECIALID_STREAMCHANGE problem that disable the subtitles in kodi GUI settings, i have opened an issue here https://github.com/xbmc/xbmc/issues/20404

i noticed that the "manual video stream selection" of ISA works correctly with the netfx non-DRM videos (trailers) then you can try reproduce the problem also in this way without apply limits to the badwidth:

CastagnaIT commented 2 years ago

i close this issue is not more needed the PR is on the way