DVBSUB subtitle

[cooliobr]

hi guys, is possible use DVBSUB with packager? today I use ffmpeg to generate HLS but I want to migrate to packager but I don't know how to use/select dvbsub track trom stream.

[kqyang]

@cooliobr We do not support DVBSUB right now. I am not familiar with DVBSUB either. I have a few questions.

• Is it the same as teletext?
• Do you have a sample stream with DVBSUB? It will help us understand the format.
• Do you have a sample command you use in ffmpeg to process DVBSUB?

We have some users using an external tool ccextractor to process the subtitles then pipe it to Shaka Packager. It can be used to handle subtitles that are not supported by Shaka Packager.

[kqyang]

@TheModMaker FYI.

[cooliobr]

@cooliobr We do not support DVBSUB right now. I am not familiar with DVBSUB either. I have a few questions.

• Is it the same as teletext?
• Do you have a sample stream with DVBSUB? It will help us understand the format.
• Do you have a sample command you use in ffmpeg to process DVBSUB?

We have some users using an external tool ccextractor to process the subtitles then pipe it to Shaka Packager. It can be used to handle subtitles that are not supported by Shaka Packager.

Thanks for your reply

1. Yes, it is same as teletext
2. sample file http://200.194.238.229:88/file.ts
3. ffmpeg -loglevel warning -stats -err_detect ignore_err -analyzeduration 9000000 -probesize 9000000 -fflags +genpts -hide_banner -threads 0 -hwaccel_device 0 -hwaccel cuvid -c:v h264_cuvid -canvas_size 1920x1080 -resize 1280x720 -deint 2 -fix_sub_duration -drop_second_field true -ignore_unknown -i udp://@227.0.0.11:1234?overrun_nonfatal=1&fifo_size=758000&buffer_size=758000&timeout=300 -map 0:a:0 -c:a aac -ac 2 -ar 48000 -aspect 16:9 -pixel_format cuda -map 0:s? -scodec copy -map 0:v:0 -c:v h264_nvenc -r 30 -profile:v main -sc_threshold 0 -g 50 -keyint_min 50 -hls_time 4 -fflags +genpts -segment_time 6 -segment_list_size 10 -hls_flags delete_segments -segment_list_flags +live -individual_header_trailer 0 -segment_list_type m3u8 -filter:v scale_npp=w=640:h=360:interp_algo=lanczos -filter:a volume=15dB -b:v 700k -maxrate 945k -bufsize 1225k -b:a 64k -f segment -segment_list /usr/local/nginx/html/live/UFSC/360p.m3u8 /usr/local/nginx/html/live/UFSC/360p_%09d.ts

[cooliobr]

Any update ?

[kqyang]

We are actively working on the feature. It should be ready some time this quarter.

[cooliobr]

Any news about this ? if need new TS file to check dvbsub I can upload it.

[kqyang]

We are still working on it. Please stay tuned.

And yes, if you have samples, you are welcomed to send it to us at shaka-packager-issues@google.com.

[cooliobr]

while dvbsub is not supported, how can I test with ccextractor? I tried in many ways but I couldn't. ccextractor -s -codec dvbsub -in=ts -udp 229.15.15.1:1234 -out=webvtt-full -o my_pipe -ocrlang por -gt -nofc

packager \ 'in=udp://229.15.15.1:1234?reuse=1,stream=0,init_segment=/usr/local/nginx/html/live/TESTE/dash_720p_init.mp4,segment_template=/usr/local/nginx/html/live/TESTE/dash_720p_$Number$.m4s' \ 'in=/root/my_pipe,stream=text,init_segment=/usr/local/nginx/html/live/TESTE/text_init.mp4,segment_template=/usr/local/nginx/html/live/TESTE/text_$Number$.m4s,language=por' \ 'in=udp://229.15.15.1:1234?reuse=1,stream_selector=1,stream=1,init_segment=/usr/local/nginx/html/live/TESTE/por_init.mp4,segment_template=/usr/local/nginx/html/live/TESTE/por_$Number$.m4s,language=por '\ 'in=udp://229.15.15.1:1234?reuse=1,stream_selector=2,stream=2,init_segment=/usr/local/nginx/html/live/TESTE/eng_init.mp4,segment_template=/usr/local/nginx/html/live/TESTE/eng_$Number$.m4s,language=eng '\ --mpd_output /usr/local/nginx/html/live/TESTE/dash.mpd \ --vmodule="*representation*=3"

[TheModMaker]

You need to specify stream=text for most text streams. Even if it is the first stream, we use stream=text to detect it is text. Also, WebVTT doesn't support multiple streams, so you can't have different stream selectors for the same input file. Lastly, stream and stream_selector mean the same thing, so you don't have to use both.

[Nixon197]

Hi

we are also looking forward to implement teletext subtitiles in DASH/HLS. Do you have ETA for this feature?

BR

[kresimirb3]

Hi guys, I am curious if you know when this enhancement will be implemented. Looking forward for to see webvtt subs working in live. @TheModMaker Do you maybe have any estimate?

[TheModMaker]

The feature work is mostly done, mainly just going through code review. So probably a few weeks.

[Canta]

I'm sorry, but I believe there's a misunderstanding here.

@cooliobr We do not support DVBSUB right now. I am not familiar with DVBSUB either. I have a few questions.

• Is it the same as teletext?
• Do you have a sample stream with DVBSUB? It will help us understand the format.
• Do you have a sample command you use in ffmpeg to process DVBSUB?

We have some users using an external tool ccextractor to process the subtitles then pipe it to Shaka Packager. It can be used to handle subtitles that are not supported by Shaka Packager.

Thanks for your reply

1. Yes, it is same as teletext

2. sample file http://200.194.238.229:88/file.ts

3. `ffmpeg -loglevel warning -stats -err_detect ignore_err -analyzeduration 9000000 -probesize 9000000 -fflags +genpts -hide_banner -threads 0 -hwaccel_device 0 -hwaccel cuvid -c:v h264_cuvid -canvas_size 1920x1080 -resize 1280x720 -deint 2 -fix_sub_duration -drop_second_field true -ignore_unknown -i udp://@227.0.0.11:1234?overrun_nonfatal=1&fifo_size=758000&buffer_size=758000&timeout=300 -map 0:a:0 -c:a aac -ac 2 -ar 48000 -aspect 16:9 -pixel_format cuda -map 0:s? -scodec copy -map 0:v:0 -c:v h264_nvenc -r 30 -profile:v main -sc_threshold 0 -g 50 -keyint_min 50 -hls_time 4 -fflags +genpts -segment_time 6 -segment_list_size 10 -hls_flags delete_segments -segment_list_flags +live -individual_header_trailer 0 -segment_list_type m3u8 -filter:v scale_npp=w=640:h=360:interp_algo=lanczos -filter:a volume=15dB -b:v 700k -maxrate 945k -bufsize 1225k -b:a 64k -f segment -segment_list /usr/local/nginx/html/live/UFSC/360p.m3u8 /usr/local/nginx/html/live/UFSC/360p_%09d.ts `

1. _dvbsubtitles, and _dvbteletext, are not the same. The main difference is that "subtitles" is basically images, while teletext is text. At least, that's the case when we're talking about what ffmpeg does with those terms. I haven't read the whole specs yet, but have some experience with both, and they're used in different contexts. _dvbsubtitles mostly end up in open captions (burned subs), while teletext is used for text format conversions.
2. Here are some teletext ts files available for download:
   • https://trac.ffmpeg.org/ticket/5067
   • https://trac.ffmpeg.org/ticket/5934
   • https://www.ccextractor.org/public:general:tvsamples
3. As can be seen in the provided example ffmpeg command, no subtitle handling is made other than copying it. There's no conversion to, let's say, srt or vtt. If you try that with _dvbsubtitles, ffmpeg will fail telling you that you can only convert to text output from text input, and _dvbsubtitles is not such type of input: you need OCR for that, which is not the case if you have _dvbteletext as input.

This is important because, after peeking at the code here, without understanding much, I can see that there's clearly image manipulation (png). So, I guess Shaka won't be able to use DVBSUB to output WEBVTT. But Shaka actually would be able to output WEBVTT from TELETEXT.

With this in mind, I have this questions:

1. Is this ticket working dvb_subtitles, dvb_teletext, or both?
2. Will shaka packager be able to output text based subtitle output from dvbsubtitles? (NOT dvbteletext)
3. I believe Shaka Packager usually targets Shaka Player as reference player. And Shaka Player is mainly web. Will Shaka Player also be able to read _dvbsubtitles as closed captions, or this is targeted to other players? _(I guess non-web ones, as I don't recall right now any web player that implements dvbsubtitles rendering)

Thanks.

[TheModMaker]

So DVB is a huge system that has many specs; it's basically like calling it "MPEG". And unfortunately there are many different ways of carrying subtitles within DVB. The one that I've implemented is DVB-SUB (ETSI EN 300 743). This contains embedded images and does have a part for text, but it doesn't specify the encoding of the text and I've never seen any content using it so can't test it. All of the sample content you listed has a stream with this data in it.

A DVB-SUB stream contains images that are usually of text (pre-rendered). So far, I've implemented this parsing and converting the images to PNG. We can't output these to WebVTT since it doesn't support images well, but you can output to TTML instead. I am testing with Shaka Player and the output is supported by Shaka Player.

Teletext (I think ETS 300 706) is historically stored in VBI (vertical blanking interval) and is similar to CEA, namely a broadcast channel within the waveform. There is another DVB spec for this, DVB-VBI (ETSI EN 301 775). Some of the samples have an additional stream with this data in it. I'm not sure if that stream has duplicate subtitles or if you'd want both. We aren't supporting that yet, but may add support in the future.

[aleek]

Yes, this is a major misunderstanding.

dvb_subtitle is veeeeeeery old format of passing subtitles on analog television. The format is a mix of raster images and vector images, i.e. caption definition can be multiple of take this bitmap, color it with pink and put in X and Y coordinates (taking display size into consider). One dvb_subtitle object can contain multiple bitmaps with different colors in different places of the decoding screen.

To be precise, dvb_subtitles is a spec of passing raw data from analog world in PES packages, of mpegts streams ;)

dvb_teletext is a format of passing raw teletext pages (see Vertical Blanking Interface) in PES packages. The teletext format itself is a digital format of pages, magazines, that you can see and read, clicking on teletext button on your remote. Teletext format defines also charsets (not only ascii chars, but also national charsets, elements of tables and many more crazy signs) other manipulation of chars/signs, timing, etc. Beside regular teletext pages, the standard itself defines two additional types of teletext pages - newsletter and subtitle. The latter is a special type of page, that is treated by decoder in different way - the background is always transparent, you can have at most 3 lines/packets of data (comparing to ~28 in regular Level 1 teletext page), the timing is synced with audio/video stream etc. But in the end, it is text-based subtitle format.

Currently there is no open-source solution to process dvb_teletext/dvb_subtitle to webvtt,smpte-tt. We actually have one, but it is commercial license, so all I can say is that it is definitely doable :)

Given the fact that main aim of packager is to repackage, the only natural solution right now in that particular case is to repackage subtitles from dvb_teletext format into web-ready one. But what if end-user want's smpte-tt text+image?

Why right now?

1. ffmpeg is not yet ready to do the perfect solution (see below). The code assumes that it deals with image frame or audio sample. There is no way to process entity that has text and optionally image.
2. Moreover, if you make the ffmpeg able to process subtitles, how will you pass text+png to packager? I can say from my research, that there is no good streaming subtitle format now.
3. ffmpeg doesn't like variable frame rate of stream, so you must make CBR.

Perfect solution in perfect world:

1. Transcode dvb_subs/dvb_teletext to some clear streaming format that can handle both text, styles and images. The closest format is smpte-tt over mp4, but mp4 isn't streamable per se :) - so again, in that scenario, packager must handle packaged mp4 input
2. Packager handles above input and converts to webvtt clear, webvtt over mp4 (with or without fragmentation), smpte-tt clear/mp4/ fragmented for LIVE and sidecar for VOD.

@Canta, here's my answers to your questions:

Is this ticket working dvb_subtitles, dvb_teletext, or both?

It can be both, but IMO it should be dvb_teletext

Will shaka packager be able to output text based subtitle output from dvb_subtitles? (NOT dvb_teletext)

No and it should not do this. Shaka packager must receive image (png?) and/or clear text and package it into webvtt, smpte-tt etc.

I believe Shaka Packager usually targets Shaka Player as reference player. And Shaka Player is mainly web. Will Shaka Player also be able to read dvb_subtitles as closed captions, or this is targeted to other players? (I guess non-web ones, as I don't recall right now any web player that implements dvb_subtitles rendering)

God no! Web formats of subtitles are WebVTT, SMPTE-TT (and derivatives, such as EBU-TT, TTML

[TheModMaker]

This issue is about DVB-SUB (ETSI EN 300 743), not teletext. I'm going to say that #272 will be for teletext support. Considering that much of the content on the ccextractor site and @Canta listed are using DVB-SUB, it is a fairly popular format.

I'm not sure if we can support any interactive bits of teletext since the we'll be generating TTML from it. We may be able to generate SCTE-35 events for that, but I'm not aware of any specs for interacting with captions. We'll probably focus on static content like text instead of webpages since you already have HTML to do interactive things.

Moreover, if you make the ffmpeg able to process subtitles, how will you pass text+png to packager?

We support TTML pass-through. If you can get FFmpeg or ccextractor to output TTML, we can add it to the manifest as-is. But this only works for VOD.

Packager handles above input and converts to webvtt clear, webvtt over mp4

Once this and #272 are solved, we'll receive the TS input ourselves and convert it to the desired output format. We cannot output WebVTT for DVB-SUB and probably some teletext since WebVTT doesn't really support images. You'd have to do TTML (either text or in MP4).

[Canta]

Oh, I wasn't aware that TTML did support images. Thanks for the explanations. If that's the case, then there's no big problem in this ticket, and my points were just a misunderstanding. With that in mind, some notes.

@TheModMaker

All of the sample content you listed has a stream with this data in it.

I trust your knowledge. But please be aware that ffmpeg semantics may differ. Take a look at this example: https://trac.ffmpeg.org/raw-attachment/ticket/5067/sample.ts This is what ffmpeg says about it:

Input #0, mpegts, from 'sample.ts':
  Duration: 00:00:03.84, start: 72016.973667, bitrate: 4284 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:2[0x102]: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s
    Stream #0:3[0x103](und): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006), 492x250
    Stream #0:4[0x104]: Data: bin_data ([6][0][0][0] / 0x0006)
    Stream #0:5[0x105]: Data: bin_data ([6][0][0][0] / 0x0006)

Take a look at streams 3, 4, and 5. It says 3 is a subtitle, of undefined language, in dvb_teletext format. And the other streams are "bin_data", which is basically "unknown format". ffmpeg do recognize dvb_subtitles as a different thing from dvb_teletext. IDK if under the hood is all the same standard used in different ways (as I understand you're saying), or they're just different standards. I say this because, if you're right and this is what you already know how to read, but in this case with text instead of images, sounds fairly trivial to implement teletext compatibility. That may not be the case if we're talking about two different things.

I'm not sure if that stream has duplicate subtitles or if you'd want both.

Frankly, the only use case I see everybody around me asking for, is simply "subtitles". If you get "subtitles" done, that's fine. Yet, it would obviously better to do it in text format, so computing requirements tends to zero, and interoperability goes to 100.

Considering that much of the content on the ccextractor site and @Canta listed are using DVB-SUB, it is a fairly popular format.

I agree, it is popular. Yet, it is also a pain to handle. TTML may help with that, but if we can have text for subtitles instead of images would be the definitive solution: we can handle text easily in any setup (at least compared with images).

Of course I do not think Shaka Packager should implement OCR of any type. That's where teletext comes in.

I'm not sure if we can support any interactive bits of teletext...

Oh, I don't think anyone wants the interactivity bits. ad-cues are the only real life thing that comes to mind in that regard, and averybody goes for SCTE-35 for that. What the people in the datacenters are struggling with are subtitles. This is video people, not web people. And they have expensive, privative video tools, that they can not mess with. Teletext seems like something everybody could use in their datacenters for subtitles instead of dvbsub. That's the use case.

We support TTML pass-through. If you can get FFmpeg or ccextractor to output TTML, we can add it to the manifest as-is. But this only works for VOD.

And that's a big question: will the implementation in this ticket work with live streams? I work with live streams, generating both HLS and DASH. I got here looking at details of shaka's possible inputs for live subtitles. I did live teletext conversion to webvtt, in order to feed shaka packager with that (haven't it working yet). Would _dvbsubtitles simply be enough in the near future?

@aleek

Currently there is no open-source solution to process dvb_teletext/dvb_subtitle to webvtt,smpte-tt.

I do it with ffmpeg without problem. If we're talking about teletext, that is. Using the same example I've linked before: ffmpeg -txt_format text -i sample.ts -map 0:s:0 -f webvtt sample.vtt

Packager handles above input and converts to webvtt clear, webvtt over mp4 (with or without fragmentation)...

Hmmm... I'm actually feeding packager with mpegts udp streams for live. Could I maybe feed it mp4 instead of mpegts, with vtt packaged inside it? I'm confused about the live part, as mp4 input is tipically VOD. Perhaps... IDK... piping live mp4?

[jakubvojacek]

Hello @TheModMaker

perhaps a bit offtopic but worth trying, are you planning to add support for CEA608/708 with WebVTT output after support for DVB subtitles if finalized?

Thank you Jakub

[Canta]

Regarding some details of the specifications, I found this mention in ffmpeg's mail list: https://ffmpeg.org/pipermail/ffmpeg-devel/2014-January/153114.html

"DVB Subtitles" (EN 300 743) is bitmap-based format, "DVB Teletext" (EN 300 472) is text-based format.

ETS 300 706 do talk about "teletext". But ETS 300 472 actually talks about "dvb teletext". https://www.etsi.org/deliver/etsi_en/300400_300499/300472/01.03.01_60/en_300472v010301p.pdf

So, sadly, I guess the answer to one of my questions, is no: doesn't seem to be the same "dvb" thing. Also, as @TheModMaker says, ETS 300 743 do have specs for text as well as bitmaps, but seems to be quite cumbersome, and frankly I've never seen an example of it. Doesn't look good at all.