Subtitles m4s segments don't always work as if livesim2 fails if ttml xml fragments don't fill the entire timeline without gaps

[Murmur]

(Livesim2 from the latest master branch) Livesim2 manifest manipulation is able to modify stpp-ttml.xml fragments inside the segment files, but 500InternalError happens every few seconds. Does it fail if input ttmlx.xml file had gaps in a subtitles timing? Mp4box do writes a continuous sequence of sub_1...N.m4s segment files but not all segments carry <p> elements or multiple segments may carry the same <p> elements (render in a future or rendered in past). This is the way how mp4box does it to provide a continuous +1 numbered sequence of sub m4s files.

segmentDur=3.84s https://refapp.hbbtv.org/livesim2/tsbd_240/mup_4/spd_8/utc_keep/00_llama_h264_v9/manifest_subib_evtib_1080p.mpd segmentDur=8s https://refapp.hbbtv.org/livesim2/tsbd_240/mup_4/spd_8/utc_keep/00_llama_h264_v9_8s/manifest_subib.mpd input xml files muxed to a m4s segments, see gaps in begin-end timings https://refapp.hbbtv.org/videos/00_llama_h264_v9_8s/sub_eng.xml https://refapp.hbbtv.org/videos/00_llama_h264_v9_8s/sub_fin.xml https://refapp.hbbtv.org/videos/00_llama_h264_v9_8s/sub_swe.xml

This is what happens in LiveSim2 500 Internal Error every few segments.

Manifest
https://refapp.hbbtv.org/livesim2/tsbd_240/mup_4/spd_8/utc_keep/00_llama_h264_v9_8s/manifest_subib.mpd

https://refapp.hbbtv.org/livesim2/tsbd_99999/mup_4/spd_8/utc_keep/00_llama_h264_v9_8s/sub_fin/sub_214481481.m4s
   <p begin="476625:30:41.000" end="476625:31:03.000" ...

500:Internal Error on sub_214481482.m4s - sub_214481487.m4s files

https://refapp.hbbtv.org/livesim2/tsbd_99999/mup_4/spd_8/utc_keep/00_llama_h264_v9_8s/sub_fin/sub_214481488.m4s
   empty <tt> without subtitle divs

500:Internal Error on sub_214481489.m4s - sub_214481490.m4s files

https://refapp.hbbtv.org/livesim2/tsbd_99999/mup_4/spd_8/utc_keep/00_llama_h264_v9_8s/sub_fin/sub_214481491.m4s
  <p begin="476625:32:16.000" end="476625:32:31.000" ....

[tobbee]

The time shift of stpp TTML payload timestamps is just a simple text lookup with regular expressions of the pattern to match hours:minutes:seconds:milliseconds and shifting of these, so the actual values of the TTML timestamps should not matter at all. It does not understand at the TTML structure itself.

I therefore find it more likely that there is some other issue with the segments. There are restrictions like that the content should be a subsample etc.

This should be fairly easy to see in the debugger, but I recognise that the logging of errors for segment generation is not so good. I therefore made a small patch in #186 to improve it. Please try to rerun your samples with the latest from main branch and see if the logs explain more.

[Murmur]

@tobbee Thanks, this is an error message on subtitles segments failing. Logging does not write the actual vod 1..N.m4s filename being processed? It would help to pinpoint files on disk.

OK (sub_214556224):
time=2024-05-23T10:36:44.876+03:00 level=DEBUG msg="requested content" request_id=refapp/UD0dFv2pVi-000021 url=00_llama_h264_v9_8s/sub_eng/sub_214556224.m4s

ERROR (sub_214556225):
time=2024-05-23T10:36:48.461+03:00 level=DEBUG msg="requested content" request_id=refapp/UD0dFv2pVi-000027 url=00_llama_h264_v9_8s/sub_eng/sub_214556225.m4s
time=2024-05-23T10:36:48.461+03:00 level=ERROR msg=writeSegment request_id=refapp/UD0dFv2pVi-000027 code=0 err="convertToLive: shiftStppTimes: not 1 but 3 samples in stpp file" 

Some subs have MOOF/TRAF/TRUN.samplescount=3 | dur=1000, dur=3000, dur=4000 (segdur 8sec). I think this is what livesim2 does not like. One sub.m4s file may have multiple <p begin end..> text lines each go to a separate sample. sample may have an "empty ttml fragment" with just <tt .../> root element without anything else.

[tobbee]

Yes, this seems to be root issue. Good that the enhanced logs helped to pinpoint the issue.

livesim2 assumes that there is exactly one sample (one TTML XML blob) in each stpp segment and that the TTML is in a subsample of that sample. That's the most common arrangement.

The reading of the file is not done in the same place as the processing and it varies between video, audio, and text where audio samples may be fetched from more than one segment, so it is not trivial to output the input file name.

[Murmur]

@tobbee My ticket in gpac was completed and able to force ttml segments to use MDAT TRUN.samplecount=1. New content is now working in Livesim2. I could not find any specification to state ttml xml text lines go to a single sample but that may be a common expectation.

https://github.com/gpac/gpac/issues/2842

[tobbee]

Hi Ari, Thanks for the update. Good that you got it working with gpac. I agree that there is nothing in the spec prohibiting multiple samples, so it is limitation of livesim2 as of now. At least, one get a rather descriptive error message now.

Are you OK with closing this ticket?