search

shaka-project / shaka-packager

A media packaging and development framework for VOD and Live DASH and HLS applications, supporting Common Encryption for Widevine and other DRM Systems.
https://shaka-project.github.io/shaka-packager/
Other
1.95k stars 504 forks source link

EBU Teletext in MPEG-TS (#1344) is not working correctly now #1428

Open acris5 opened 2 weeks ago

acris5 commented 2 weeks ago

System info

Operating System: docker Shaka Packager Version: b5c2cb8 (main)

Issue and steps to reproduce the problem

I use live mpegts source to create HLS and Dash videos with teletext. Packager Command:

packager  \
'in=/shaka-packager/pipes/pipe1,stream=video,init_segment=/shaka-results/bitrate_1/video_init.mp4,segment_template=/shaka-results/bitrate_1/video_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/video.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=audio,lang=ru,init_segment=/shaka-results/bitrate_1/audio_init.mp4,segment_template=/shaka-results/bitrate_1/audio_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/audio.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=text,format=ttml+mp4,init_segment=/shaka-results/bitrate_1/text_init.mp4,segment_template=/shaka-results/bitrate_1/text_$Number$.m4s,lang=ru,playlist_name=/shaka-results/bitrate_1/text.m3u8' \
 --max_hd_pixels 8294400 --hls_master_playlist_output /shaka-results/demo_master.m3u8 --mpd_output /shaka-results/manifest.mpd --hls_playlist_type LIVE \
 --segment_duration 2 --min_buffer_time 4 --suggested_presentation_delay 10 --time_shift_buffer_depth 12 --allow_approximate_segment_timeline --preserved_segments_outside_live_window 20

What is the expected result? Live playing in HLS.js and dash.js players. This feature worked correctly with this version Shaka Packager Version: e21519bb28d848720a4d8a4491e2ac6d1a8c16bb

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 1ad0a4c150-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT4S" type="dynamic" publishTime="2024-08-29T08:27:37Z" availabilityStartTime="2024-08-29T08:25:44Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT12S" suggestedPresentationDelay="PT10S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="10669404" codecs="avc1.640028" mimeType="video/mp4" sar="1:1">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/video_init.mp4" media="bitrate_1/video_$Number$.m4s" startNumber="50">
          <SegmentTimeline>
            <S t="8967600" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="ru" segmentAlignment="true">
      <Representation id="1" bandwidth="141930" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="bitrate_1/audio_init.mp4" media="bitrate_1/audio_$Number$.m4s" startNumber="50">
          <SegmentTimeline>
            <S t="8821179" d="180480" r="4"/>
            <S t="9723579" d="176640"/>
            <S t="9900219" d="180480" r="1"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="text" lang="ru" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Representation id="2" bandwidth="3216" codecs="stpp" mimeType="application/mp4">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/text_init.mp4" media="bitrate_1/text_$Number$.m4s" startNumber="50">
          <SegmentTimeline>
            <S t="8820000" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

What happens instead? Shaka produces wrong teletext segments time. Starts from 0, then increases but when it reaches current time of video and audio then packager stops working and writes to log that packaging is finished.

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 183f1b2e05-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT4S" type="dynamic" publishTime="2024-08-29T08:29:31Z" availabilityStartTime="2024-08-29T08:29:26Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT12S" suggestedPresentationDelay="PT10S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="4965184" codecs="avc1.640028" mimeType="video/mp4" sar="1:1">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/video_init.mp4" media="bitrate_1/video_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="187200" d="180000" r="1"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="ru" startWithSAP="1" segmentAlignment="true">
      <Representation id="1" bandwidth="138940" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="bitrate_1/audio_init.mp4" media="bitrate_1/audio_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="137108" d="44160"/>
            <S t="181268" d="180480" r="1"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="text" lang="ru" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Representation id="2" bandwidth="1708" codecs="stpp" mimeType="application/mp4">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/text_init.mp4" media="bitrate_1/text_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="0" d="180000"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 183f1b2e05-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT4S" type="dynamic" publishTime="2024-08-29T08:32:43Z" availabilityStartTime="2024-08-29T08:29:26Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT12S" suggestedPresentationDelay="PT10S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="video" width="1920" height="1080" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <Representation id="0" bandwidth="10493236" codecs="avc1.640028" mimeType="video/mp4" sar="1:1">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/video_init.mp4" media="bitrate_1/video_$Number$.m4s" startNumber="92">
          <SegmentTimeline>
            <S t="16567200" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="ru" startWithSAP="1" segmentAlignment="true">
      <Representation id="1" bandwidth="144160" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="bitrate_1/audio_init.mp4" media="bitrate_1/audio_$Number$.m4s" startNumber="92">
          <SegmentTimeline>
            <S t="16382228" d="180480" r="2"/>
            <S t="16923668" d="176640"/>
            <S t="17100308" d="180480" r="3"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="text" lang="ru" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Representation id="2" bandwidth="6652" codecs="stpp" mimeType="application/mp4">
        <SegmentTemplate timescale="90000" initialization="bitrate_1/text_init.mp4" media="bitrate_1/text_$Number$.m4s" startNumber="33">
          <SegmentTimeline>
            <S t="5760000" d="180000" r="6"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Players can't play manifest with different time inside so they freeze video at start.

acris5 commented 1 week ago

https://github.com/shaka-project/shaka-packager/commit/2ba67bc24cf6116349ad16d3bfa1a121c95a3173 It seems we need to specify text zero bias to more than default zero (for example 10 minutes) to get teletext working

packager  \
'in=/shaka-packager/pipes/pipe1,stream=video,init_segment=/shaka-results/bitrate_1/video_init.mp4,segment_template=/shaka-results/bitrate_1/video_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/video.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=audio,lang=ru,init_segment=/shaka-results/bitrate_1/audio_init.mp4,segment_template=/shaka-results/bitrate_1/audio_$Number$.m4s,playlist_name=/shaka-results/bitrate_1/audio.m3u8' \
'in=/shaka-packager/pipes/pipe1,stream=text,format=ttml+mp4,init_segment=/shaka-results/bitrate_1/text_init.mp4,segment_template=/shaka-results/bitrate_1/text_$Number$.m4s,lang=ru,playlist_name=/shaka-results/bitrate_1/text.m3u8' \
 --max_hd_pixels 8294400 --hls_master_playlist_output /shaka-results/demo_master.m3u8 --mpd_output /shaka-results/manifest.mpd --hls_playlist_type LIVE \
 --segment_duration 2 --min_buffer_time 4 --suggested_presentation_delay 10 --time_shift_buffer_depth 12 --allow_approximate_segment_timeline --preserved_segments_outside_live_window 20 --default_text_zero_bias_ms 600000
tobbee commented 1 week ago

@acris5 I'm working on improving the segment generation for live teletext in MPEG-2 TS.

One of the issues I found is that the TextPadder does not work properly for live TS, so I commented it out in packager.cc and then things worked properly for me.

In any case, it is not correct in the sense that it does not check the timescale that is 90000 for the teletext subtitles, but is considered to be 1000. Since the incoming timestamps can be arbitrary, I think that all type of start time determination algorithm should be turned off for live TS input. In any case, it would be good to have a solution that makes things work for live TS subtitles.

For live text segmentation from MPEG-2 TS, there are two other major problems that I'm working with.

  1. Segments generation is only triggered by incoming data on the teletext pid
  2. Segments are not generated until a cue is finished, even though that may be seconds later than video and audio

To fix the first problem, I first try to extract heart beats (stuffing data with PTS) from the teletext PID, and then I've added a configuration to send heartbeats from other elementary stream in the same demuxer. In the latter case, they have a configurable timestamp shift in order to wait for subtitle cues to be included.

To fix the second issue, I'm looking into sending the cue content as the cue start, but with duration 0, and then later send the end time. Then the text chunker can produce a segment without waiting for the end of the cue, and just let it go on to the end of the segment. This is a bit trickier, so it will take some time before I have it working.

If you have any ideas about other ways to go, please let me know.