EBU Teletext in MPEG-TS (#1344) is not working correctly now

[acris5]

System info

Operating System: docker Shaka Packager Version: b5c2cb8 (main)

Issue and steps to reproduce the problem

I use live mpegts source to create HLS and Dash videos with teletext. Packager Command:

packager  \
'in=/shaka-packager/pipes/pipe1,stream=video,init_segment=/shaka-results/bitrate_1/video_init.mp4,segment_template=/shaka-results/bitrate_1/video_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/video.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=audio,lang=ru,init_segment=/shaka-results/bitrate_1/audio_init.mp4,segment_template=/shaka-results/bitrate_1/audio_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/audio.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=text,format=ttml+mp4,init_segment=/shaka-results/bitrate_1/text_init.mp4,segment_template=/shaka-results/bitrate_1/text_$Number$.m4s,lang=ru,playlist_name=/shaka-results/bitrate_1/text.m3u8' \
 --max_hd_pixels 8294400 --hls_master_playlist_output /shaka-results/demo_master.m3u8 --mpd_output /shaka-results/manifest.mpd --hls_playlist_type LIVE \
 --segment_duration 2 --min_buffer_time 4 --suggested_presentation_delay 10 --time_shift_buffer_depth 12 --allow_approximate_segment_timeline --preserved_segments_outside_live_window 20 

What is the expected result? Live playing in HLS.js and dash.js players. This feature worked correctly with this version Shaka Packager Version: e21519bb28d848720a4d8a4491e2ac6d1a8c16bb

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 1ad0a4c150-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT4S" type="dynamic" publishTime="2024-08-29T08:27:37Z" availabilityStartTime="2024-08-29T08:25:44Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT12S" suggestedPresentationDelay="PT10S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="10669404" codecs="avc1.640028" mimeType="video/mp4" sar="1:1">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/video_init.mp4" media="bitrate_1/video_$Number$.m4s" startNumber="50">
          <SegmentTimeline>
            <S t="8967600" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="ru" segmentAlignment="true">
      <Representation id="1" bandwidth="141930" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="bitrate_1/audio_init.mp4" media="bitrate_1/audio_$Number$.m4s" startNumber="50">
          <SegmentTimeline>
            <S t="8821179" d="180480" r="4"/>
            <S t="9723579" d="176640"/>
            <S t="9900219" d="180480" r="1"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="text" lang="ru" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Representation id="2" bandwidth="3216" codecs="stpp" mimeType="application/mp4">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/text_init.mp4" media="bitrate_1/text_$Number$.m4s" startNumber="50">
          <SegmentTimeline>
            <S t="8820000" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

What happens instead? Shaka produces wrong teletext segments time. Starts from 0, then increases but when it reaches current time of video and audio then packager stops working and writes to log that packaging is finished.

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 183f1b2e05-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT4S" type="dynamic" publishTime="2024-08-29T08:29:31Z" availabilityStartTime="2024-08-29T08:29:26Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT12S" suggestedPresentationDelay="PT10S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="4965184" codecs="avc1.640028" mimeType="video/mp4" sar="1:1">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/video_init.mp4" media="bitrate_1/video_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="187200" d="180000" r="1"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="ru" startWithSAP="1" segmentAlignment="true">
      <Representation id="1" bandwidth="138940" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="bitrate_1/audio_init.mp4" media="bitrate_1/audio_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="137108" d="44160"/>
            <S t="181268" d="180480" r="1"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="text" lang="ru" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Representation id="2" bandwidth="1708" codecs="stpp" mimeType="application/mp4">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/text_init.mp4" media="bitrate_1/text_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="0" d="180000"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 183f1b2e05-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT4S" type="dynamic" publishTime="2024-08-29T08:32:43Z" availabilityStartTime="2024-08-29T08:29:26Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT12S" suggestedPresentationDelay="PT10S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="10493236" codecs="avc1.640028" mimeType="video/mp4" sar="1:1">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/video_init.mp4" media="bitrate_1/video_$Number$.m4s" startNumber="92">
          <SegmentTimeline>
            <S t="16567200" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="ru" startWithSAP="1" segmentAlignment="true">
      <Representation id="1" bandwidth="144160" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="bitrate_1/audio_init.mp4" media="bitrate_1/audio_$Number$.m4s" startNumber="92">
          <SegmentTimeline>
            <S t="16382228" d="180480" r="2"/>
            <S t="16923668" d="176640"/>
            <S t="17100308" d="180480" r="3"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="text" lang="ru" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Representation id="2" bandwidth="6652" codecs="stpp" mimeType="application/mp4">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/text_init.mp4" media="bitrate_1/text_$Number$.m4s" startNumber="33">
          <SegmentTimeline>
            <S t="5760000" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Players can't play manifest with different time inside so they freeze video at start.

[acris5]

https://github.com/shaka-project/shaka-packager/commit/2ba67bc24cf6116349ad16d3bfa1a121c95a3173 It seems we need to specify text zero bias to more than default zero (for example 10 minutes) to get teletext working

packager  \
'in=/shaka-packager/pipes/pipe1,stream=video,init_segment=/shaka-results/bitrate_1/video_init.mp4,segment_template=/shaka-results/bitrate_1/video_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/video.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=audio,lang=ru,init_segment=/shaka-results/bitrate_1/audio_init.mp4,segment_template=/shaka-results/bitrate_1/audio_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/audio.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=text,format=ttml+mp4,init_segment=/shaka-results/bitrate_1/text_init.mp4,segment_template=/shaka-results/bitrate_1/text_$Number$.m4s,lang=ru,playlist_name=/shaka-results/bitrate_1/text.m3u8' \
 --max_hd_pixels 8294400 --hls_master_playlist_output /shaka-results/demo_master.m3u8 --mpd_output /shaka-results/manifest.mpd --hls_playlist_type LIVE \
 --segment_duration 2 --min_buffer_time 4 --suggested_presentation_delay 10 --time_shift_buffer_depth 12 --allow_approximate_segment_timeline --preserved_segments_outside_live_window 20 --default_text_zero_bias_ms 600000

[tobbee]

@acris5 I'm working on improving the segment generation for live teletext in MPEG-2 TS.

One of the issues I found is that the TextPadder does not work properly for live TS, so I commented it out in packager.cc and then things worked properly for me.

In any case, it is not correct in the sense that it does not check the timescale that is 90000 for the teletext subtitles, but is considered to be 1000. Since the incoming timestamps can be arbitrary, I think that all type of start time determination algorithm should be turned off for live TS input. In any case, it would be good to have a solution that makes things work for live TS subtitles.

For live text segmentation from MPEG-2 TS, there are two other major problems that I'm working with.

1. Segments generation is only triggered by incoming data on the teletext pid
2. Segments are not generated until a cue is finished, even though that may be seconds later than video and audio

To fix the first problem, I first try to extract heart beats (stuffing data with PTS) from the teletext PID, and then I've added a configuration to send heartbeats from other elementary stream in the same demuxer. In the latter case, they have a configurable timestamp shift in order to wait for subtitle cues to be included.

To fix the second issue, I'm looking into sending the cue content as the cue start, but with duration 0, and then later send the end time. Then the text chunker can produce a segment without waiting for the end of the cue, and just let it go on to the end of the segment. This is a bit trickier, so it will take some time before I have it working.

If you have any ideas about other ways to go, please let me know.

[rantoniello]

Hi

Many thanks for posting your messages; I am integrating shaka-packager as a library and was facing similar issues with live TS stream.

Just a few comments on @tobbee 's post:

In any case, it is not correct in the sense that it does not check the timescale that is 90000 for the teletext subtitles, but is considered to be 1000.

Definitely true. On method "TextPadder::OnTextSample" we have the code:

if (max_end_time_ms_ < 0) { max_end_time_ms_ = zero_start_bias_ms_ && sample.start_time() > zero_start_bias_ms_ ? sample.start_time() : 0; }

'max_end_timems' is supposed to be milliseconds and 'sample.start_time()' is @90KHz clock. This is a minor bug, just needs a timescale normalization. Nevertheless, setting 'zero_start_biasms' to '1' is enough for shaka packager not to assume '0' start time, and to wait until the first time-stamp comes in to initialize.

@acris5 For the cases I am working with -that is, live streaming MPEG2-TS starting with any random initial time-stamp- setting 'default_text_zero_bias_ms'=1 seems to work (be careful of using a high value, see the code snippet above).

Segments generation is only triggered by incoming data on the teletext pid

Shaka expects each live input stream contribution to be "conformant" (more or less :-) ). Having an insight on the code, shaka completely ignores MPEG2-TS PCRs and only uses PTSs for sampling and chunking timing. But, as far as I'm concerned, just sending PAT and PMT tables, let's say once each 100 ms -which should be mandatory in MPEG recommendation- is enough. It seems it does not need periodic time-stamps for triggering, as PSI tables do not carry time references (the solution might be simpler, just send correspondent PAT/PMT). In the case I am working with, streaming periodic PAT and PMT is enough.

Segments are not generated until a cue is finished, even though that may be seconds later than video and audio

mmm... I am not facing this issue... perhaps is due to conformance issues with the input teletext you are using (if you have a sample, I can try it). In the teletext standard (ETS 300 706), it states that "If a magazine has only one displayable page (and in some other circumstances), it may be necessary to indicate the completion of the transmission of that page by closing it with another page header packet.". For the case of subtitling, it is necessary to add "filling headers" to explicitly mark the end of a subtitle page.