Packaging webvtt subtitles into live stream

[jakubvojacek]

System info

Operating System: latest debian Shaka Packager Version: packager version daac686-release

Issue and steps to reproduce the problem

Trying to supply our live stream with subtitles that we continuously generate into vtt file

Packager Command:

packager  'in=udp://239.100.200.2:1234?interface=172.16.128.42&reuse=1,stream=video,bandwidth=400000,init_segment=ct24-9_video_640_360_400.mp4,segment_template=video_640_360_400/ct24-9-$Number$.mp4' 'in=udp://239.100.200.2:1235?interface=172.16.128.42&reuse=1,stream=video,bandwidth=700000,init_segment=ct24-9_video_640_360_700.mp4,segment_template=video_640_360_700/ct24-9-$Number$.mp4' 'in=udp://239.100.200.2:1236?interface=172.16.128.42&reuse=1,stream=video,bandwidth=1000000,init_segment=ct24-9_video_720_405_1000.mp4,segment_template=video_720_405_1000/ct24-9-$Number$.mp4' 'in=udp://239.100.200.2:1255?interface=172.16.128.42&reuse=1,stream=audio,language=cs, init_segment=ct24-9_audio_cs.mp4,segment_template=audio_cs/ct24-9-$Number$.mp4' 'in=test.vtt,stream=text,init_segment=ttv_init.mp4,segment_template=ttv_$Number$.m4s,language=cs' --time_shift_buffer_depth 15000 --hls_playlist_type LIVE --hls_master_playlist_output ct24-9.m3u8 --mpd_output ct24-9.mpd --fragment_duration 3.2 --segment_duration 3.2  --suggested_presentation_delay 5 --min_buffer_time 3.2 --minimum_update_period 5 --default_language=cs

Extra steps to reproduce the problem? (1) this is the test.vtt test.vtt.zip. This file is growing along with the stream (once we detect subtitles from dvb stream, we add them into the test.vtt file)

What is the expected result? Having live mpd manifest with subtitles

What happens instead? It ends with an error [0412/205943:ERROR:packager_main.cc(473)] Packaging Error: 14 (INTERNAL_ERROR): Failed to parse WebVTT source. See log for details..

Is there a way to make this work? I am sorry for multiple questions regarding subtitles but the documentation is quite brief on this matter. Once I grasp this concept, I can try to make a PR to the doc.

Thank you

[vaage]

@jakubvojacek Don't worry about the multiple issues. We appreciate your participation in helping improve the packager.

Just looking at the text.vtt I believe the problem is with the STYLE tag. Right now we don't support style tags. I believe it is trying to parse it as a cue and failing. If you look earlier in the logs you should need a message about not being able to classify the cue.

Next week I will can add some code that will classify the STYLE tag and skip it so that it can read the file, however no style information will be outputted. Full support for STYLE tags will have to wait as they are a bigger problem that will require some time to design.

[jakubvojacek]

The problem is that the vtt file is being generated by 3rd party tool and we have no control over it, therefore, I cannot remove STYLE tag. For VODs, we can just pre-process the vtt file but for live it's trickier.

Thank you

[jakubvojacek]

We have managed to update the subtitle generator to remove the STYLE tag. However, it is still not working.

The MPD can be found on https://edge1.motv.eu/channels/ct24-9/live/ct24-9.mpd, when I attempt to play it in shaka player, it says it cannot find text segment. Without subtitles, it is playing without issues, so I am guessing it has something to do with the subtitles. The packager command is:

packager  'in=udp://239.100.200.2:1234?interface=172.16.128.42&reuse=1,stream=0,bandwidth=400000,init_segment=ct24-9_video_640_360_400.mp4,segment_template=video_640_360_400/ct24-9-$Number$.mp4' 'in=udp://239.100.200.2:1234?interface=172.16.128.42&reuse=1,stream=1,bandwidth=700000,init_segment=ct24-9_video_640_360_700.mp4,segment_template=video_640_360_700/ct24-9-$Number$.mp4' 'in=udp://239.100.200.2:1234?interface=172.16.128.42&reuse=1,stream=2,bandwidth=1000000,init_segment=ct24-9_video_720_405_1000.mp4,segment_template=video_720_405_1000/ct24-9-$Number$.mp4' 'in=udp://239.100.200.2:1234?interface=172.16.128.42&reuse=1,stream=3,language=cs, init_segment=ct24-9_audio_cs.mp4,segment_template=audio_cs/ct24-9-$Number$.mp4' 'in=test.vtt,stream=text,init_segment=ttv_init.mp4,segment_template=ttv_$Number$.m4s,language=cs' --time_shift_buffer_depth 45000 --hls_playlist_type LIVE --hls_master_playlist_output ct24-9.m3u8 --mpd_output ct24-9.mpd --fragment_duration 3.2 --segment_duration 3.2  --suggested_presentation_delay 5 --min_buffer_time 3.2 --minimum_update_period 5 --default_language=cs

Subtitle file is here https://edge1.motv.eu/channels/ct24-9/live/test.vtt and is growing (around 50% of events has subtitles available).

What can be the issue? Is it correct, that the vtt file is having start times from zero and not from the current timestamp? Or does packager ignores the timestamps in file and just assumes that the subtitle should be shown at the time its appended to the file?

Thank you

[vaage]

@jakubvojacek Sorry I have not gotten to this yet. The packager uses the timestamps in the content. If the timestamps in the text does not align with the timestamps of the other assets, then there will be an issue.

Looking at the content you provided and based on what you say, the text is not aligned with the video and audio. Which would explain why the shaka player can't find the segments.

So going forward, the question is what should be done? It would be possible for us to use an offset to re-align the text with video and audio. However that would require you to know ahead of time (when writing the command) to know what offset you would need. Does a solution like that work for you?

@kqyang Do you have any other thoughts on this?

[jakubvojacek]

@vaage it would be tricky to use with an offset I guess. I would have to calculate the offset in before running packager and somehow manage to turn on the subtitle extractor right on time. But that will not happen, because before it starts to output data, it analyses the stream which takes variable time.

The tool for extracting subtitles (ccextractor) supports timing either from 0 or unix timestamp (something like 423401:22:56.720 --> 423401:22:58.760), but I guess that is not supported by shaka packager as of now. The easiest solution would be if ccextractor team could add support for stream timestamp, I'll ask at their github whether they could consider that

[kqyang]

@jakubvojacek I think ccextractor already supports stream timestamp. This is what I get from their documentation:

https://www.ccextractor.org/doku.php?id=public:general:command_line_usage

        -gt --goptime: Use GOP for timing instead of PTS. This only applies
                       to Program or Transport Streams with MPEG2 data and
                       overrides the default PTS timing.
                       GOP timing is always used for Elementary Streams.
    -nogt --nogoptime: Never use GOP timing (use PTS), even if ccextractor
                       detects GOP timing is the reasonable choice.

[jakubvojacek]

@kqyang Thank you, I have tried these options but they do not seem to be working. No error is thrown but the output is still same

ccextractor -out=webvtt -nobom -s -nofc -gt -udp 239.10.10.4:5004 -o test.vtt produces output starting with 0 (00:00:00.800 --> 00:00:02.840) and ccextractor -out=webvtt -nobom -s -nofc -nogt -udp 239.10.10.4:5004 -o test.vtt produces the same (starting with 00:00:02.840 --> 00:00:07.920). Either I am doing something wrong or these parameters do not work as intended I guess.

[kqyang]

@jakubvojacek Have you talked to ccextractor team about this?

[jakubvojacek]

@kqyang I tried, with no response unfortunately - https://github.com/CCExtractor/ccextractor/issues/975

[jakubvojacek]

Hello,

we have managed to update & fix the ccextractor ourselves and its now generating correct timestamps in vtt file. But still, it does not work with the packager. The command is

packager  'in=pipe9_live,stream=0,bandwidth=400000,init_segment=ct24-9_video_640_360.mp4,segment_template=video_640_360/ct24-9-$Number$.mp4' 'in=pipe9_live,stream=1,bandwidth=1000000,init_segment=ct24-9_video_720_410.mp4,segment_template=video_720_410/ct24-9-$Number$.mp4' 'in=pipe9_live,stream=2,language=cs, init_segment=ct24-9_audio_cs.mp4,segment_template=audio_cs/ct24-9-$Number$.mp4' 'in=channel.vtt,stream=text,init_segment=ct24-9_subtitle_init.mp4,segment_template=subtitle/ttv_$Number$.m4s,language=cs' --time_shift_buffer_depth 15000 --hls_playlist_type LIVE --hls_master_playlist_output ct24-9.m3u8 --mpd_output ct24-9.mpd --fragment_duration 3.2 --segment_duration 3.2  --suggested_presentation_delay 5 --min_buffer_time 3.2 --minimum_update_period 5 --default_language=cs --allow_approximate_segment_timeline --utc_timings urn:mpeg:dash:utc:http-head:2014=https://mw.unifiqueplay.com.br/time.txt --io_block_size 65536

The MPD is located here - https://goo.gl/R1wFti, but cannot guarantee how long it will work before it crashes. Attaching how currently the MPD looks like - ct24-9.mpd.txt

<AdaptationSet id="1" contentType="text" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="1" bandwidth="1018" codecs="wvtt" mimeType="application/mp4">
        <SegmentTemplate timescale="1000" initialization="ct24-9_subtitle_init.mp4" media="subtitle/ttv_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="0" d="3200"/>
            <S t="3200" d="3200"/>
            <S t="6400" d="3200"/>
            <S t="9600" d="3200"/>
            <S t="12800" d="3200"/>
            <S t="16000" d="3200"/>
            <S t="19200" d="3200"/>
            <S t="22400" d="3200"/>
            <S t="25600" d="3200"/>
            <S t="28800" d="3200"/>
            <S t="32000" d="3200"/>
            <S t="35200" d="3200"/>
...

Loos all the segments have same duration but theyre not groupped using the r="xx" parameter, is that intenrional? The MPD grows in size a lot. But more importantly, there are no subtitles visible even if they are enabled in the player. I can see that the player is downloading m4s files but they appear to be empty. This is the log of the packager

[0619/211512:INFO:demuxer.cc(88)] Demuxer::Run() on file 'pipe9_live'.
[0619/211512:INFO:demuxer.cc(160)] Initialize Demuxer for file 'pipe9_live'.
[0619/211513:WARNING:webvtt_parser.cc(147)] Missing support for style blocks. Skipping block:
 --- BLOCK START ---
    STYLE
 --- BLOCK END ---
[0619/211716:INFO:mp4_muxer.cc(240)] MP4 file 'ct24-9_subtitle_init.mp4' finalized.

On the last line, I think, it's showing that it finalized and ended subtitles? But that should not happen for live, right? It should be neverending processing. This is how the files look like

The problem when testing is that the subtitles are only broadcasted like 50% of time, not always. But I have been checking the file (tailf) and during the time I was checking, there were subtitles available and written into the channel.vtt file. Any idea what I might be doing wrong?

Thank you

[kqyang]

@jakubvojacek

Loos all the segments have same duration but theyre not groupped using the r="xx" parameter, is that intenrional?

That is really weird. There must be something wrong. Do you see any logging on the command line?

On the last line, I think, it's showing that it finalized and ended subtitles?

That is a bit misleading, but it just means the init segment is written, so nothing alerting here.

we have managed to update & fix the ccextractor ourselves and its now generating correct timestamps in vtt file.

No. The timestamp is not correct. It starts from zero which does not align with other streams and that is why it does not show up during playback.

[jakubvojacek]

@kqyang

That is really weird. There must be something wrong. Do you see any logging on the command line?

No, in the post, I have attached the whole output of the packager's command line. I have just restarted the packager and it's doing the same, the MPD has already over 4k lines in 11 minutes

That is a bit misleading, but it just means the init segment is written, so nothing alerting here.

Are you sure? As per the packagers output that I posted in the previous post, why did it took 2 minutes for the subtitle int segment to be written and why it did not write this for other streams? I see this message for other streams when the packager is stopping (killing the ffmpeg that pipes data into the packager). I did this just now to the packager that I started last night - see bellow the output please

[0619/211512:INFO:demuxer.cc(88)] Demuxer::Run() on file 'pipe9_live'.
[0619/211512:INFO:demuxer.cc(160)] Initialize Demuxer for file 'pipe9_live'.
[0619/211513:WARNING:webvtt_parser.cc(147)] Missing support for style blocks. Skipping block:
 --- BLOCK START ---
    STYLE
 --- BLOCK END ---
[0619/211716:INFO:mp4_muxer.cc(240)] MP4 file 'ct24-9_subtitle_init.mp4' finalized.
[0620/010101:WARNING:representation.cc(384)] Found a gap of size 8640 > kRoundingErrorGrace (5). The new segment starts at 1219392240 but the previous segment ends at 1219383600.
[0620/053539:INFO:mp4_muxer.cc(240)] MP4 file 'ct24-9_video_640_360.mp4' finalized.
[0620/053539:INFO:mp4_muxer.cc(240)] MP4 file 'ct24-9_video_720_410.mp4' finalized.
[0620/053540:INFO:mp4_muxer.cc(240)] MP4 file 'ct24-9_audio_cs.mp4' finalized.

The MP4 file xxx finalized message was issued for audio / video when it was stopping (5:30 in the morning) but for the subtitles, it was issues at 21:17 previous day, is this correct?

No. The timestamp is not correct. It starts from zero which does not align with other streams and that is why it does not show up during playback.

The link to the used vtt file is here https://goo.gl/Z4PLVs but seems pretty aligned to me, at least the way I understand it should be aligned in.

ffprobe is showing timestamp 36941.209611 which translates to a little bit over 10 hours, which is what the ccextractor is showing too. Is there something else that should be aligned?

Thank you

[jakubvojacek]

The finalized message appeared in the packager log again

[0620/053612:INFO:demuxer.cc(88)] Demuxer::Run() on file 'pipe9_live'.
[0620/053612:INFO:demuxer.cc(160)] Initialize Demuxer for file 'pipe9_live'.
[0620/053612:WARNING:webvtt_parser.cc(147)] Missing support for style blocks. Skipping block:
 --- BLOCK START ---
    STYLE
 --- BLOCK END ---
[0620/055214:INFO:mp4_muxer.cc(240)] MP4 file 'ct24-9_subtitle_init.mp4' finalized.

it happened about 20 minutes into packaging. After this point, no more m4s files were generated while other segmetns (audio & video) are still being created

Current server time

root@jones:/data/mw/channels/ct24-9/live# date
Wed Jun 20 08:03:50 CEST 2018

Subtitle folder

root@jones:/data/mw/channels/ct24-9/live# ls -hatl subtitle/
total 19M
drwxr-xr-x 7 www-data www-data 4.0K Jun 20 08:01 ..
drwxr-xr-x 2 root     root     164K Jun 20 05:52 .
-rw-r--r-- 1 root     root      278 Jun 20 05:52 ttv_11537.m4s
-rw-r--r-- 1 root     root      416 Jun 20 05:52 ttv_11536.m4s
-rw-r--r-- 1 root     root      245 Jun 20 05:52 ttv_11535.m4s
-rw-r--r-- 1 root     root      275 Jun 20 05:52 ttv_11534.m4s
-rw-r--r-- 1 root     root      364 Jun 20 05:52 ttv_11533.m4s
-rw-r--r-- 1 root     root      373 Jun 20 05:52 ttv_11532.m4s
-rw-r--r-- 1 root     root      251 Jun 20 05:52 ttv_11531.m4s
-rw-r--r-- 1 root     root      406 Jun 20 05:52 ttv_11530.m4s
-rw-r--r-- 1 root     root      299 Jun 20 05:52 ttv_11529.m4s
-rw-r--r-- 1 root     root      390 Jun 20 05:52 ttv_11528.m4s
-rw-r--r-- 1 root     root      243 Jun 20 05:52 ttv_11527.m4s
-rw-r--r-- 1 root     root      259 Jun 20 05:52 ttv_11526.m4s
-rw-r--r-- 1 root     root      192 Jun 20 05:52 ttv_11525.m4s
-rw-r--r-- 1 root     root      192 Jun 20 05:52 ttv_11524.m4s
-rw-r--r-- 1 root     root      192 Jun 20 05:52 ttv_11523.m4s
-rw-r--r-- 1 root     root      192 Jun 20 05:52 ttv_11522.m4s
-rw-r--r-- 1 root     root      192 Jun 20 05:52 ttv_11521.m4s
-rw-r--r-- 1 root     root      192 Jun 20 05:52 ttv_11520.m4s
...

Audio folder

root@jones:/data/mw/channels/ct24-9/live# ls -hatl audio_cs/
total 142M
drwxr-xr-x 7 www-data www-data 4.0K Jun 20 08:04 ..
drwxr-xr-x 2 root     root     104K Jun 20 08:04 .
-rw-r--r-- 1 root     root      51K Jun 20 08:04 ct24-9-2780.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:04 ct24-9-2779.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:04 ct24-9-2778.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:04 ct24-9-2777.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:04 ct24-9-2776.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:04 ct24-9-2775.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:03 ct24-9-2774.mp4
-rw-r--r-- 1 root     root      51K Jun 20 08:03 ct24-9-2773.mp4

[kqyang]

@jakubvojacek Ah, you are right. Thanks for your detail report! I looked into the issue again. There are three different problems.

1. You are right that the "finalized" message should not be there as it is only emitted when the input is closed. Looks like what you are doing is: (1) Extract subtitles from a UDP stream to a file "channel.vtt" using ccextractor; (2) Having packager read from the file "channel.vtt". Consumer (2) could run faster than producer (1). I am afraid that there could be a race condition causing packager to incorrectly think it reaches the end of file. Using pipe may help but there is another problem with pipe as it blocks until sufficient number of bytes are available. Anyway, it worth trying. Can you try using pipe, i.e. write the ccextractor output to a pipe?

2. The timestamps in manifest start from 0. Sorry for my incorrect statement earlier - I thought it was the result of timestamp misalignment between audio/video and text timestamps. This is actually a bug in packager, where we always start text timestamp from 0. It should be fixed.

3. Segment timeline entries are not grouped together. We will see if we can re-pro it in our side.

[vaage]

I have opened a separate issue about text starting at zero and added some background about why we do that (#416). @kqyang could you jump over there and add your thoughts on it. I feel that this is going to be harder than we would like to solve. @jakubvojacek if you have any thoughts, please chime in on #416 as well.

[jakubvojacek]

@kqyang 1) not sure I can force ccextractor to output to a pipe, or is there some general unix output pipe output that I am not aware of? I did few more tests and sometimes the finalize message is outputted right at the start time of the packager, sometimes it takes few hours, its quite random 2) & 3) perfect, thank you, glad I am not (probably) doing anything wrong

@vaage would love to help but not much of a C programmer. But happy to test anything that you guys prepare in real live production where subtitles support is actually required by law :)

[kqyang]

@jakubvojacek In Linux, pipe can be read and written like a normal file, so it may just work with ccextractor, i.e. you can create the pipe using mkfifo then supply the pipe name to ccextractor just like a normal file.

[jakubvojacek]

@kqyang just tested using pipe. Was able to make ccextractor into pipe. But the packager only starts to read from the pipe when the ccextractor starts after packager. If I start ccextractor first, packager ignores the subtitles (and produces MPD without text track without throwing an error or waiting for the pipe). Upon restarting ccextractor subtitles started to appear in the MPD (but again without repeating r="x" parameter). It has been running for 15 minutes without the finalized message in packager log

Would prefer to not use pipe as an output from the ccextractor though. Was planning to use the generated vtt file to feed packager that packages the recorded EPG event. In case we would have to use pipe, we would have to have 2 instances of ccextractor running at all time, one for live writing into a pipe and one for catchup writing into a file, which isn't ideal.

[kqyang]

@jakubvojacek Glad to know pipe works for you. I understand that it is not ideal, but there does not seem to be a perfect solution here.

An alternative solution is that we can add a flag to indicate the "input" will never terminate, forcing packager to keep retrying when reaching end of the file. I am not in favor of it as it is not very clean.

We are planning to have native support for subtitles in TS sometime in Q3 / Q4 - but it will take some time. Do you have a sample input file we can take a look? It will help us speed up the development.

but again without repeating r="x" parameter

Filed #417. Will have a fix soon.

[jakubvojacek]

@kqyang will be happy to provide a sample file but since its live udp (pipe) input not sure what I should share. I have a short recorded stream that has subtitles in it that I could share (https://goo.gl/4hQHBf) that you can play in a loop + subtitles here subtitles.txt.

In case you would like me to share anything else, please let me know

[vaage]

@jakubvojacek It looks like all three issues in @kqyang post have been addressed. Are you okay with us closing this issue now?

[jakubvojacek]

@vaage did some testing and still struggling to make it work. When generating live MPD with subtitles, after a little bit under 3 minutes, I get MPD

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/google/shaka-packager version 1742e03471-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT3.2S" type="dynamic" publishTime="2018-07-03T05:04:53Z" availabilityStartTime="2018-07-02T08:42:19Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT15000S" suggestedPresentationDelay="PT5S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="text" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="0" bandwidth="1183" codecs="wvtt" mimeType="application/mp4">
        <SegmentTemplate timescale="1000" initialization="ct24-9_subtitle_init.mp4" media="subtitle/ttv_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="73203200" d="3200" r="1564"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="video" width="720" height="480" frameRate="90000/3600" segmentAlignment="true" par="20:11">
      <Representation id="1" bandwidth="400000" codecs="avc1.4d401e" mimeType="video/mp4" sar="40:33">
        <SegmentTemplate timescale="90000" initialization="ct24-9_video_640_360.mp4" media="video_640_360/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="496472400" d="288000" r="45"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
      <Representation id="2" bandwidth="1000000" codecs="avc1.4d401e" mimeType="video/mp4" sar="40:33">
        <SegmentTemplate timescale="90000" initialization="ct24-9_video_720_410.mp4" media="video_720_410/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="496472400" d="288000" r="45"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="audio" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="3" bandwidth="138065" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="ct24-9_audio_cs.mp4" media="audio_cs/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="496225401" d="288000" r="46"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
  <UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-head:2014" value="https://mw.motv.eu/time.txt"/>
</MPD>

which has 45/46 segments for video and audio and already 1564 segments for subtitles, should not be the number be approximately the same? I checked for some VODs that we have with subtitles and the amount of segments per subtitles and audio / video are +- same. Also, this content was unplayable by shaka player because it was searching for incorrect video / audio time

it was searching for audio / video at subtitle time (73203200+1565*3200)/1000 instead of (496225401+47*288000)/1000 (there was a small time difference in between saving the MPD content and taking the screenshot so the number are off by a little). I was able to play it with dashjs but not even dashjs was displaying any subtitle.

Also, if counting correctly, that would be around 120 000 m4s files in 4 hours long live buffer which is quite a lot of files for one directory. Can you please point out what I am doing wrong? Hope that it does not make any difference but the subtitles were fed to the packager as text file instead of pipe (see below my comment about pipes, please).

@kqyang

An alternative solution is that we can add a flag to indicate the "input" will never terminate, forcing packager to keep retrying when reaching end of the file. I am not in favor of it as it is not very clean.

I somehow struggle with finding a way to make it work reliably. If I start the ccextractor that writes into the pipe before packager, the packager will not event start generating subtitles. If I restart the ccextractor when the packager is already running, sometimes it does kill the package with an error"[0703/082456:ERROR:packager_main.cc(485)] Packaging Error: 14 (INTERNAL_ERROR): Failed to parse WebVTT source. See log for details.", which results into restarting packager and not processing subtitles (since ccextractor started earlier).

When writing into a regular file, I never have these problems, the only problem is the race condition that the packager after some random time finalizes the subtitles. Cannot the packager make an assumption, that whenever the input of audio / video is UDP, that it's a live content and that it should never finalize the subtitles? That would solve this case and there would be not need to add another flag for this, what do you think?

[vaage]

@jakubvojacek You are right, there should be approximately the same number of audio/video segments as text and that number of text seems way too large. Thank you for checking this for us.

Looking at the times in the segment templates:

• text starts at 73,203.2 (73203200 / 1000) seconds
• video starts at 5,516.3 (496472400 / 90000) seconds
• audio starts at 5,513.6 (9496225401 / 90000) seconds

So there is something weird with the text start time compared to the audio/video start times (@kqyang am I reading that right)?

If I am reading the manifest correctly, it would mean that the text is starting well after the audio/video which would explain why the players would not be able to show any subtitles.

[jakubvojacek]

@vaage now that I think about it again, this might be a mistake on my side. I was extracting subtitles from incorrect multicast. I was extracting them from the input multicast instead of from the one I transcode with ffmpeg that I feed to packager afterwards. But still unable to test really du to the another issue I mentioned in the previous test.

When piping subtitles, it wont start in packager, when using file to provide the subtitles, the packager finalizes them right away, therefore I cannot confirm whether it's working now.

[kqyang]

@jakubvojacek Reply to your previous post, we think having a separate flag is better because

• There are several forms of live input, i.e. UDP is not the only form. It will work for all cases by using a flag.
• I don't think anyone would do it but it is possible to send on-demand input through UDP, so UDP does not mean live. Also, even for live UDP input, it can terminate, i.e. the socket can be closed.

Anyway, we'll give it more thoughts before picking an implementation.

For your use case, would it work if you redirect the output from ccextractor to UDP, possibly using netcat (https://serverfault.com/questions/386405/netcat-udp-file-transfer)?

[jakubvojacek]

@kqyang just tested using netcat and if I understand what you're suggesting, it does not seem to be supported

'in=udp://239.190.100.1:1234,stream=text,init_segment=ct24-9_subtitle_init.mp4,segment_template=subtitle/ttv_$Number$.m4s,language=cs'

results in error

[0704/072236:ERROR:packager_main.cc(480)] Failed to initialize packager: 3 (INVALID_ARGUMENT): Text output format is not support for udp://239.190.100.1:1234

Did you mean something else other then to use UDP as input for packager? Because that does not seem to be supported.

[kqyang]

That is what I meant, but you are right, it is not supported yet.. We'll think about either make text for UDP work or make it possible to not terminate when reaching end of file.

[jakubvojacek]

Thank you, that will be very helpful, both options sounds nice, but I still would prefer the flag.

I was trying to browse the source code to find if I could insert a return somewhere in before the finalize method, to avoid closing the file - so that packager would not stop reading the file and we could test that both ccextractor and packager can work together well and for long without crashing. Would something like this possible? Before you manage to come with a clean solution.

[kqyang]

@jakubvojacek Yes, certainly, here is the change you need:

--- c/packager/file/threaded_io_file.cc
+++ w/packager/file/threaded_io_file.cc
@@ -174,12 +174,14 @@ void ThreadedIoFile::RunInInputMode() {
   while (true) {
     int64_t read_result =
         internal_file_->Read(&io_buffer_[0], io_buffer_.size());
-    if (read_result <= 0) {
+    if (read_result < 0) {
       NoBarrier_Store(&eof_, read_result == 0);
       NoBarrier_Store(&internal_file_error_, read_result);
       cache_.Close();
       return;
     }
+    if (read_result == 0)
+      continue;
     if (cache_.Write(&io_buffer_[0], read_result) == 0) {
       return;
     }

[jakubvojacek]

@kqyang Thank you, hope I managed to apply the patch (here https://github.com/google/shaka-packager/compare/master...jakubvojacek:jakubvojacek-live-subtitles?expand=1) but it appears to not work the way it should. I will try to describe how it behaves now.

Using applied changes, I no longer get the finalized message of subtitle track in the packager output. This is how the output looks like:

[0710/083805:INFO:demuxer.cc(88)] Demuxer::Run() on file 'udp://239.100.200.2:1234?reuse=1&buffer_size=212992000'.
[0710/083805:INFO:demuxer.cc(160)] Initialize Demuxer for file 'udp://239.100.200.2:1234?reuse=1&buffer_size=212992000'.
[0710/083805:WARNING:webvtt_parser.cc(147)] Missing support for style blocks. Skipping block:
 --- BLOCK START ---
    STYLE
 --- BLOCK END ---
[0710/083805:ERROR:mp2t_media_parser.cc(310)] Ignore unsupported MPEG2TS stream type 0x6

The audio and video segments are being created correctly as usual but the subtitle files are not. The number of subtitles segments generated changes from run to run (probably depends on what is the timing of the first subtitle compare to start of the input / package), once it was 6, once 38, once 1000. But the number does not get higher afterwards.

root@jones:/data/mw/channels/ct24-9/live# ls -1alt audio_cs/| head -4
total 3512
drwxr-xr-x 6 www-data www-data  4096 Jul 10 08:41 ..
drwxr-xr-x 2 root     root      4096 Jul 10 08:41 .
-rw-r--r-- 1 root     root     52049 Jul 10 08:41 ct24-9-68.mp4
root@jones:/data/mw/channels/ct24-9/live# ls -1alt subtitle/| head -4
total 148
drwxr-xr-x 6 www-data www-data 4096 Jul 10 08:41 ..
drwxr-xr-x 2 root     root     4096 Jul 10 08:38 .
-rw-r--r-- 1 root     root      409 Jul 10 08:38 ttv_35.m4s
root@jones:/data/mw/channels/ct24-9/live# ls -laht audio_cs/  | wc -l
129
root@jones:/data/mw/channels/ct24-9/live# ls -laht subtitle/  | wc -l
38

It almost seems like if the race condition was still happening? Or am I doing something wrong?

Also, when there were no subtitles at the time when packager starts (the vtt file is empty, just the header in there), the packager will not even start the subtitle segment timeline and the MPD is then unplayable in both dashjs and shaka player. Even when some subtitle becomes available, it does not start. This scenario can happen quite often as the subtitles are available rarely on some channels. Attaching how the generated MPD looks like

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/google/shaka-packager version 608ae0323f-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT3.2S" type="dynamic" publishTime="2018-07-10T06:49:24Z" availabilityStartTime="2018-07-10T06:36:09Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT15000S" suggestedPresentationDelay="PT5S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="text" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="0" bandwidth="0" codecs="wvtt" mimeType="application/mp4">
        <SegmentTemplate timescale="1000" initialization="ct24-9_subtitle_init.mp4" media="subtitle/ttv_$Number$.m4s" startNumber="1"/>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="video" width="720" height="480" frameRate="90000/3600" segmentAlignment="true" par="20:11">
      <Representation id="1" bandwidth="400000" codecs="avc1.4d401e" mimeType="video/mp4" sar="40:33">
        <SegmentTemplate timescale="90000" initialization="ct24-9_video_640_360.mp4" media="video_640_360/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="58676400" d="288000" r="44"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
      <Representation id="2" bandwidth="1000000" codecs="avc1.4d401e" mimeType="video/mp4" sar="40:33">
        <SegmentTemplate timescale="90000" initialization="ct24-9_video_720_410.mp4" media="video_720_410/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="58676400" d="288000" r="44"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="audio" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="3" bandwidth="132453" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="ct24-9_audio_cs.mp4" media="audio_cs/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="58542681" d="211200"/>
            <S t="58753881" d="288000" r="44"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
  <UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-head:2014" value="https://mw.motv.eu/time.txt"/>
</MPD>

[kqyang]

@jakubvojacek Looks like there is something wrong. Will investigate.

As for

Also, when there were no subtitles at the time when packager starts (the vtt file is empty, just the header in there), the packager will not even start the subtitle segment timeline and the MPD is then unplayable in both dashjs and shaka player.

I am curious what is the problem you are seeing. I thought it was working in dashjs but didn't work in shaka player before the fix in https://github.com/google/shaka-player/commit/64d72fe08cde4fdc2e5a2e6b5dace5a8259a1d26. Is that not the case?

[jakubvojacek]

@kqyang that was the case with packaged VOD, not live content. For live it just ends up with

SegmentsUtils.js:238 Uncaught TypeError: Cannot read property 'start' of null
    at decideSegmentListRangeForTemplate (SegmentsUtils.js:238)
    at Object.getSegmentsFromTemplate [as getSegments] (TemplateSegmentsGetter.js:64)
    at Object.getSegments (SegmentsGetter.js:65)
    at updateSegments (DashHandler.js:218)
    at updateSegmentList (DashHandler.js:244)
    at Object.updateRepresentation (DashHandler.js:255)
    at Object.updateData (RepresentationController.js:192)
    at Object.updateData (DashAdapter.js:446)
    at Object.selectMediaInfo (StreamProcessor.js:250)
    at createStreamProcessor (Stream.js:390)

I have prepared a working sample MPD here https://goo.gl/51EYWm.

[jakubvojacek]

@kqyang thought of testing with https://nightly-dot-shaka-player-demo.appspot.com/demo and it does play, seems like the fix has resolved it for both live and vod. I guess I should report this at dashjs repo.

Anyway, the problem is that the channel.vtt file is not empty anymore and contains lots of subtitles, yet the packager did not notice that and did not start processing it & creating segments.

[kqyang]

@jakubvojacek Thanks. Can you try if it works in Shaka Player nightly: https://nightly-dot-shaka-player-demo.appspot.com/demo/?

[kqyang]

Ooops, I posted the above comment before seeing your new comment.

thought of testing with https://nightly-dot-shaka-player-demo.appspot.com/demo and it does play, seems like the fix has resolved it for both live and vod. I guess I should report this at dashjs repo.

Good to know! Yes, please report the issue to dashjs and keep us posted!

As a side note, I wonder if ccextractor supports generating empty cues, which will allow packager to generate segments (albeit empty) for text. This will be another way to workaround the problem.

Anyway, the problem is that the channel.vtt file is not empty anymore and contains lots of subtitles, yet the packager did not notice that and did not start processing it & creating segments.

Look into it now!

[jakubvojacek]

@kqyang

As a side note, I wonder if ccextractor supports generating empty cues, which will allow packager to generate segments (albeit empty) for text. This will be another way to workaround the problem.

Thought of it as well, but could not it lead to another, yet unlikely, race condition? (not sure how does ccextractor work internally and when it writes to file). But when both packager and ccextrator would start right after each other

1. ccextractor starts and writes webvtt header
2. packager starts but since vtt file is empty it keeps on ignoring it
3. ccextractor analyzes input and creates empty cue

I have prepared an example of the other problem here https://edge1.motv.eu/channels/ct24-9/live/ct24-9.mpd. It shows that even if there are subtitles available at the time of packager start, the packager does not process them. In the MPD you can see that it created (at the time that I checked) 93 audio / video segments but only two subtitle segments. For some reason it does not continue to process the channel.vtt file

[jakubvojacek]

@kqyang please ignore my last comment regarding the packager not creating subtitle segments, this was an issue on my side, was checking a wrong commit in the glient sync command.

It does now play and show subtitles on dashjs (https://edge1.motv.eu///channels/ct24-9/live/ct24-9.mpd).

Thank you and I am sorry for the misunderstanding I created. Seems like only one issue to tackle is when there are no subtitles available. I will do some test with ccextractor generating empty cues tomorrow

[jakubvojacek]

@kqyang I have tried adding

00:00:00.100 --> 00:00:00.500
.

after the vtt header, however, that did not help. Is it possible that packager is throwing away cues that are older than current stream time? If so, then it is hard to add an empty cue by the ccextractor, isnt it?Even if the ccextractor would be capable of adding an empty cue to the stream time + small offset (so that it would not interfere with real cues, if any), the packager would have to start in whatever the offset was, right? If would take longer, the cue would be older than current stream time and would be discarded. That is also a race condition, isn't it?

Not sure I made myself clear, this is somewhat hard to explain. Can you please explain further about the empty cues as well as what timing should they have in order for the packager to work? I might have misunderstood something.

Thank you

[kqyang]

@jakubvojacek ccextractor needs to generate continuous empty cues - not just one, with the correct timestamp extracted from the input. I don't know if it is supported in ccextractor right now, but it should be possible to do.

Something like this:

00:00:00.100 --> 00:00:00.500
.

00:00:01.100 --> 00:00:01.500
.

00:00:02.100 --> 00:00:02.500
.

[jakubvojacek]

@kqyang the ccextracot does not support this, we tried adding the feature ourselves but it isnt ideal. To add empty cues (with just a space for example) would be doable for VOD. But for live, you never know when subtitle is going to appear in the stream. And generating empty cues would result in overlap and shifting subtitles in the player - the player tries to show all subtitles, even empty and shifts them down.

Also, I talked to dashsj team and they're saying that the MPD generated is not even valid as per dash specs (https://github.com/Dash-Industry-Forum/dash.js/issues/2708#issuecomment-404433509).

Do you think it would be hard to fix this error this in shaka packager? To somehow "emulate" one empty subtitle cue at the beginning, and that should be enough, right? After the first cue, the packager will keep on creating even empty subtitle files.

Thank you

[kqyang]

A cue with payload '.' is not really empty and it does affect the display. There is a packager bug that it does not support 'real' empty cue, i.e. a cue without payload, right now: https://github.com/google/shaka-packager/issues/433. When the issue is addressed, you'll be able to generate cues without affecting the display during playback.

Do you think it would be hard to fix this error this in shaka packager? To somehow "emulate" one empty subtitle cue at the beginning, and that should be enough, right? After the first cue, the packager will keep on creating even empty subtitle files.

It is actually pretty difficult. The problem is that packager does not know what timestamp should the empty subtitle cue be without a reference timestamp. And for the same reason, packager does not know how much empty subtitle files to create without a reference timestamp.

One possible solution is to synchronize the text stream with A/V streams internally during packaging and get the current timestamps from A/V streams. That requires a lot of work and it is also difficult to implement correctly. I probably prefer supporting CEA-608/708 (I assume that is the subtitle format you have now) parsing directly in packager then we'll be able to get timestamps from the container. That also requires some amount of work, so it cannot be supported in the near term.

Also, I talked to dashsj team and they're saying that the MPD generated is not even valid as per dash specs (Dash-Industry-Forum/dash.js#2708 (comment)).

I agree with DASH IF folks that it is out of spec :). Unfortunately, we do not have much options here as we don't have any segments available. We can address https://github.com/google/shaka-packager/issues/422 by skipping writing the AdaptationSet/Representation with empty SegmentTimeline if it is needed.

Let us know what you think!

[jakubvojacek]

Was hoping it would be possible to take the timestamp from audio / video stream (whichever is first added into the packager), that would resolve it I believe. Another option could be to add a parameter to the segment template, which would reference from which stream (file and or udp + index) to take the timestamp from?

It is not just CEA-608/708 but other as well, right now I am testing mostly on dvb_teletext and later we're planning on supporting dvb_subttiles via OCR as well (not really familiar with what the standards are called).

I guess that when https://github.com/google/shaka-packager/issues/433 will be implemented, we might be able to make it work via doing what you suggested earlier - generating lots of empty cues periodically inside ccextractor and hoping that the packager, for whatever reason will not start before ccextractor. And also hope that players on all platforms will be able to handle the empty cues without moving/replacing existing cues down on the screen

Thank you

[jakubvojacek]

Doing some more tests and probably doing something wrong. I used the patch you shared like a week ago to never stop finalizing subtitle file (https://github.com/jakubvojacek/shaka-packager/commit/3d716b615e85a00dce1fe17225efe54105d3466c).

I updated ccextractor to always include at the very beginning of file this:

WEBVTT

STYLE

00:00:00.100 --> 00:00:00.101
.

00:00:01.100 --> 00:00:01.101
.

00:00:02.100 --> 00:00:02.101
.

00:00:03.100 --> 00:00:03.101
.

00:00:04.100 --> 00:00:04.101
.

00:00:05.100 --> 00:00:05.101
.

00:00:06.100 --> 00:00:06.101
.

00:00:07.100 --> 00:00:07.101

.....

up to 59 seconds. I can start with zero timestamps because I am extracting the subtitles from ffmpeg that starts at the same time as the packager and ccextractor (there is race condition if one would start way faster than the other but lets disregard that for now). The packager picks up the subtitles and starts processing them, the output of packager is

[0717/113421:INFO:demuxer.cc(88)] Demuxer::Run() on file 'udp://239.100.200.2:1234?reuse=1&buffer_size=212992000'.
[0717/113421:INFO:demuxer.cc(160)] Initialize Demuxer for file 'udp://239.100.200.2:1234?reuse=1&buffer_size=212992000'.
[0717/113421:WARNING:webvtt_parser.cc(147)] Missing support for style blocks. Skipping block:
 --- BLOCK START ---
    STYLE
 --- BLOCK END ---
[0717/113421:ERROR:mp2t_media_parser.cc(310)] Ignore unsupported MPEG2TS stream type 0x6

no message about finalizing the subtitle track so it should be processing indefinitely, right. But the problem is that only 17 subtitle segments were created in the subtitle folder and no more. Do you please know what the issue could be? My understanding was that once the packager starts processing the subtitles (picks up at least one valid cue), it will continue generating them indefintely. The MPD looks like.

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/google/shaka-packager version 3d716b615e-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT3.2S" type="dynamic" publishTime="2018-07-17T09:44:54Z" availabilityStartTime="2018-07-17T09:34:21Z" minimumUpdatePeriod="PT5S" timeShiftBufferDepth="PT15000S" suggestedPresentationDelay="PT5S">
  <Period id="0" start="PT0S">
    <AdaptationSet id="0" contentType="text" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="0" bandwidth="890" codecs="wvtt" mimeType="application/mp4">
        <SegmentTemplate timescale="1000" initialization="ct24-9_subtitle_init.mp4" media="subtitle/ttv_$Number$.m4s" startNumber="1">
          <SegmentTimeline>
            <S t="0" d="3200" r="16"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="video" width="720" height="480" frameRate="90000/3600" segmentAlignment="true" par="20:11">
      <Representation id="1" bandwidth="400000" codecs="avc1.4d401e" mimeType="video/mp4" sar="40:33">
        <SegmentTemplate timescale="90000" initialization="ct24-9_video_640_360.mp4" media="video_640_360/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="1371600" d="288000" r="196"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
      <Representation id="2" bandwidth="1000000" codecs="avc1.4d401e" mimeType="video/mp4" sar="40:33">
        <SegmentTemplate timescale="90000" initialization="ct24-9_video_720_410.mp4" media="video_720_410/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="1371600" d="288000" r="196"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet id="2" contentType="audio" lang="cs" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="3" bandwidth="131905" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
        <SegmentTemplate timescale="90000" initialization="ct24-9_audio_cs.mp4" media="audio_cs/ct24-9-$Number$.mp4" startNumber="1">
          <SegmentTimeline>
            <S t="1218681" d="222720"/>
            <S t="1441401" d="288000" r="196"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
  <UTCTiming schemeIdUri="urn:mpeg:dash:utc:http-head:2014" value="https://mw.motv.eu/time.txt"/>
</MPD>

Also, how will packager react to a case where I will have 59 empty cues up to 59 seconds and after that the ccextractor will write some cue at second 10. Will it crash or will it ignore it? If ignore, thats better for use. We can live without having correct subtitles for 59 seconds after start for live content. If it would crash, would it help if we were had 1 very long cue (00:00:01.100 --> 00:00:59.000) instead of 59 short ones? Does the packager use the start time or the end time while deciding whether to start creating subtitles?

[kqyang]

Was hoping it would be possible to take the timestamp from audio / video stream (whichever is first added into the packager), that would resolve it I believe...

[1] Unfortunately it is not a one-time thing. It needs to get the time from an audio / video stream periodically and continuously. There are a lot of challenges. Will think about it more to see if it is possible to make it work.

And also hope that players on all platforms will be able to handle the empty cues without moving/replacing existing cues down on the screen

Don't worry about it. Packager will strip all the empty cues unless there are cues in the segment then the whole segment is an empty cue. This ensures that the actual cues are always correctly displayed no matter how it is handled in the players.

no message about finalizing the subtitle track so it should be processing indefinitely, right. But the problem is that only 17 subtitle segments were created in the subtitle folder and no more.

That is expected. WebVTT allows overlapping cues, so packager will hold the last cue (59th cue) until seeing the next cue, no segments will be created during this time frame. So it is important to generate continous non-stopping empty cues for it to work.

Also, how will packager react to a case where I will have 59 empty cues up to 59 seconds and after that the ccextractor will write some cue at second 10. Will it crash or will it ignore it?

I believe packager will return with an error as time is not allowed to move back. The time for empty cues should reflect the time encoded in the media, which ccextractor has access to. That is one of the challenges of the solution [1] above as packager does not have text time information when there is no text cues; the time hint from audio/video may not align with the time from text exactly.

[jakubvojacek]

So it is important to generate continous non-stopping empty cues for it to work.

Ok, now I guess I finally understand, thought that it just needs subtitles at the beginning. Will see what we can do for that in ccextractor, not an ideal solution but any solution is perfect now :)

Thank you

[jakubvojacek]

I got it working (not in ccextractor but using a small python script). Already got over 100 channels with subtitles working and seems to be quite stable.

@kqyang, @vaage Thank you for all the effort into making this work!

[kqyang]

@jakubvojacek That is great! I am curious how do you get the timestamps though in the python script?