No ATSC CEA-608/EIA-708 Closed Caption subtitles with NVDEC

[kmdewaal]

• Platform: Ubuntu 18

• MythTV version: master

• Package version: compiled from source

• Component: mythfrontend

What steps will reproduce the bug?

Use a frontend with an Nvidia card and the Nvidia driver. Then:

• Select a video profile with VDPAU
• Select a recording with ATSC Closed Captions.
• Play the recording
• Select subtitle menu
• Select subtitle NOW the subtitle is displayed. This is correct.

The problem appears when:

• Select a video profile with NVDEC
• Play the same recording NOW: cannot select the subtitle menu, hence cannot select the subtitle and no subtitle is shown.

How often does it reproduce? Is there a required condition?

The issue is 100% reproducible on my ATSC test streams. I do play them back as a Video. The issue has also been reproduced by pgbennett.

What is the expected behaviour?

The expected behavior is that, when a subtitle stream is present, this can be selected and shown on the screen, for NVDEC, for VDPAU and for any other video driver.

What do you see instead?

Additional information

This problem is reported on the mailing list, item https://forum.mythtv.org/viewtopic.php?f=36&t=4458 Reported for MythTV version 31.

Note that this is specific for ATSC CC subtitles; DVB subtitles can be selected and are shown as expected.

[bennettpeter]

The problem is that closed-captioned subtitles are incorporated into the mpeg stream. NVDEC and other hardware decoders do not extract them and we do not have code to examine the stream directly for this. Originally I saw this problem on Raspberry Pi with the hardware decoder.

I have thought about this before, and the solution I thought of was to take the source code of ccextractor and incorporate it into the player logic in a way that could extract subtitles while playing.

[kmdewaal]

I have not yet looked at the code but as I understand it VDPAU and VAAPI are also hardware decoders. Yet ATSC CC subtitling does actually work with these decoders.

[kmdewaal]

Correction: test clip "frida.ts" (re-used from https://github.com/MythTV/mythtv/issues/326) does show the subtitle menu but only the first choice of both the ATSC CC and the VBI CC entries. This means only the Spanish subtitle can be selected; the English subtitle, which is the second entry for each, cannot be selected. Note that although the Spanish subtitle can be selected it will NOT be shown, not for ATSC (708) and not for VBI (608).

[gigem]

That sounds like an ffmpeg bug I've seen before where it only extracts the first subtitles. To work around it on my transcoding tasks, I had to separately extract both subtitles with ccextractor and remux them later as subtitle streams.

[kmdewaal]

This ffmpeg ticket https://trac.ffmpeg.org/ticket/8078 tells that you need to add the command line option "-hwaccel nvdec" in the ffmpeg transcoding command to get the closed captions transferred when transcoding. This then suggests that somewhere in the mythtv playback code where nvdec is selected an option/parameter might need to be added.

[kmdewaal]

After looking at this for a while I do agree with Peter's comment that it would be best to make something separate that collects the closed caption information from the video stream. The way it currently works for VDPAU is that there is a lot of MythTV-specific code in file mpeg12dec.c to capture the closed caption information. To make this work for NVDEC a similar set of changes would be needed in file cuviddec.c where the NVDEC decoding is located. This is then another large modification to the FFmpeg code and, as Peter mentioned, it does not solve the problem for yet other decoders such as the one for the Raspberri Pi. I do however not have a clear view on how the "make something separate" would have to be done. As there is currently also a definite bandwidth limit on my side I unassign myself for now from this ticket.

[ulmus-scott]

Would it be possible to duplicate the stream and pass it to both the hardware decoder and ffmpeg for demuxing the subtitles?

[gigem]

FWIW, the Android frontend has this problem too when using MediaCodec.

[bshanteau]

@ulmus-scott @kmdewaal Rather than adding new closed caption code to cuviddec.c, wouldn't it be easier to pull the closed caption code out of mpeg12dec.c and put it in a separate C program so that any decoder, including cuviddec.c, could use it?

[ulmus-scott]

The FFmpeg change can't really be separated out.

Also, FFmpeg should support A/53 subtitles (if our customization didn't override it), see mpeg_decode_a53_cc() in the same file.

[bshanteau]

@ulmus-scott That's too bad.

What is the import of FFmpeg supporting A/53 subtitles to NVDEC?

Could mpeg_decode_a53_cc() also serve cuviddec.c?

[mspieth]

On 25/11/2022 2:48 pm, bshanteau wrote:

@ulmus-scott https://github.com/ulmus-scott That's too bad.

What is the import of FFmpeg supporting A/53 subtitles to NVDEC?

Could |mpeg_decode_a53_cc()| also serve cuviddec.c?

Check that its not a forces vs unforced issue too.

Ive seen problems with this in the past displaying subtitles when every odd one was forced and erasing the unforced ones.

Mark

[bshanteau]

On Nov 11, 2021 @ulmus-scott wrote:

Would it be possible to duplicate the stream and pass it to both the hardware decoder and ffmpeg for demuxing the subtitles?

Are you saying that while a GPU is decoding the video and audio streams that it is incapable of decoding the subtitles?

According to FFmpeg closed defect #8078, ffmpeg with the flags -hwaccel nvdec and -hwaccel_output_format cuda can indeed handle subtitles.

[ulmus-scott]

What is the import of FFmpeg supporting A/53 subtitles to NVDEC?

Could mpeg_decode_a53_cc() also serve cuviddec.c?

That was mostly a reminder that our FFmpeg customization may be unnecessary and that we should look into removing that FFmpeg customization.

On Nov 11, 2021 @ulmus-scott wrote:

Would it be possible to duplicate the stream and pass it to both the hardware decoder and ffmpeg for demuxing the subtitles?

Are you saying that while a GPU is decoding the video and audio streams that it is incapable of decoding the subtitles?

According to FFmpeg closed defect #8078, ffmpeg with the flags -hwaccel nvdec and -hwaccel_output_format cuda can indeed handle subtitles.

Looking at my suggestion again, it is basically the same as Peter's: add code to detect and parse the various A/53 subtitle flavors to our video player.

The problem is that the subtitles are not a separate stream but encoded as part of each frame. This means the hardware acceleration needs to support separating the subtitles or you need to parse the stream in parallel to extract them.

I am not familiar enough with either the FFmpeg hardware acceleration code or our player code to say exactly what is needed.

[bshanteau]

@ulmus-scott wrote

Looking at my suggestion again, it is basically the same as Peter's: add code to detect and parse the various A/53 subtitle flavors to our video player.

Would it be sufficient to just have MythTV's video player call ffmpeg with the appropriate parameters? For instance, whereas your NVDEC video player deletes subtitles, this call results in output WITH subtitles: ffmpeg -hwaccel nvdec -i 30787_20221123205900.ts -c:v h264_nvenc -f mpegts out1.mp4 30787_20221123205900.ts.zip out1.mp4.zip @bennettpeter @kmdewaal

[gigem]

I've been out of town and haven't been following this (nor much of any other thread) lately. However, if this is the problem I think it is, Channels DVR has a solution they use. They setup a BitReader to capture the subtitles encoded in the video and decode them themselves. They then keep track of the timecode and reattach the decoded subtitlte back to the video frame when it comes back from the video decoder. The last time I talked to him, Aman Karmani aman@tmm1.net was willing to share some (maybe all) of their code that does that. The big catch is that at the time, it used an older ffmpeg and probably needed serious porting.

[bshanteau]

@gigem If I'm not mistaken, the FFmpeg team has since resolved this problem.

[bshanteau]

@kmdewaal Would you mind explaining the necessity of creating mythffmpeg and why ffmpeg alone was not sufficient? On a quick glance I don't see what mythffmpeg does that ffmpeg doesn't.

[kmdewaal]

@kmdewaal Would you mind explaining the necessity of creating mythffmpeg and why ffmpeg alone was not sufficient? On a quick glance I don't see what mythffmpeg does that ffmpeg doesn't.

It is not really related to this ticket but a bit of background never hurts. MythTV comes with its own version of FFmpeg. This makes it possible to add fixes that are needed to let MythTV work properly and it also makes MythTV independent of a local FFmpeg installation being present or not and also independent of the version of FFMpeg that is installed. The binaries such as ffmpeg and ffplay that are generated when the MythTV version of FFmpeg is compiled are copied to the MythTV binaries directory as mythffmpeg and mythffplay. This are the names that are then used by the MythTV scripts that are used for archiving and DVD creation. This insures, as said, that all of MythTV can be installed and does work independent of whether FFmpeg is installed and which version of FFmpeg is installed. One must realize that this was all conceived at a time when digital television was new and both MythTV and FFmpeg were very much in heavy development, and one could not rely on a FFmpeg commandline option still being there one month later. Time has moved on and things are far more stable now, and there are now efforts to use the FFmpeg that is present on the system instead bundling a MythTV-specific version with MythTV.

[gigem]

@gigem If I'm not mistaken, the FFmpeg team has since resolved this problem.

@bshanteau, that would be excellent! Aman told me he tried to get it accepted before but it was rejected as being too much of a hack. And he's an ffmpeg committer! The last time I checked myself, it didn't look like anything had been done. I'll check again soon but if you have a reference handy, please send it my way.

[bshanteau]

@gigem

The case that mythfrontend doesn't show subtitles with nvdec is a bug in how MythTV is using nvdec. I expect it will get fixed by those knowledgeable in that area soonish.

I just checked and the nvdec subtitle bug has been fixed in v33.0~master.202212052334.5c8d88cf42~ubuntu22.04.1

Thanks to all!!!

[bshanteau]

@gigem Also see the FFmpeg documentation on subtitles.

[bshanteau]

@gigem And the FFmpeg Wiki entry: How to extract and remove closed captions

[bshanteau]

@gigem Finally, the Wikibook: FFMPEG An Intermediate Guide/subtitle options

[gigem]

@bshanteau, thanks for the links. I don't think this problem has been fixed for the mediacodec case, which is the one that matters most to me. I checked the ffmpeg, git logs and don't see anything having been done regarding closed captions. I've asked Aman for confirmation but he hasn't replied yet.

FWIW, I did see some other changes I'd been loosly tracking that have made it into master. Those changes use the native API for mediacodec so Java is no longer needed. It's possible that change fixes the closed captions issue as a side effect but I doubt it. Regardless, support for that should be pulled into MythTV eventually.

[tmm1]

Nothing has changed wrt to the mediacodec decoders. If someone wants to work on that I can point them in the right direction.

[bshanteau]

@tmm1 I'm willing to help, although I suspect the resolution might be as simple as grabbing the latest binaries of fmpeg and ffplay that are generated when, as @kmdewaal said above:

the MythTV version of FFmpeg is compiled are copied to the MythTV binaries directory as mythffmpeg and mythffplay

How is the latest MythTV version of FFmpeg compiled?

[tmm1]

That may help with nvdec, but not with mediacodec (android).

To extract closed caption data from mpeg2 streams without decoding, the BSF patchset that adds A53_CC support needs to be merged. There is an old version that no longer applies cleanly and must be forward-ported: https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227321.html

[bshanteau]

@tmm1 wrote:

That may help with nvdec, but not with mediacodec (android).

What is the command line option that calls mediacodec? This doesn't work: ffmpeg -hwaccel mediacodec -i ...

[gigem]

@bshanteau, mediacodec is the Android, hardware decoder and only runs on Android. I doubt there is a true, command-line way to run it. Instead, you will need to use an Android app which uses it, and probably one that uses the NDK at that. I don't know if Android, Java apps have this problem. BTW, @tmm1 is an ffmpeg developer and a Channels DVR developer. He knows what he's talking about.

[bshanteau]

@gigem I have no doubt that @tmm1 knows what he's talking about. Nevertheless, I'm struggling to understand what he means when he says:

That may help with nvdec, but not with mediacodec (android).

On first glance it sounds like he's saying that ffmpeg can be used to decode a video/audio/subtitle stream with mediacodec on an Android device. But, as you say, that's impossible, since ffmpeg doesn't run on Android.

So what is @tmm1 asking for? Is it that he wants to be able to transfer a recording file from an Android device to a Linux machine and decode it with mythffmpeg using a mediacodec decoder there? Does that mean the current version of mythffmpeg has a mediacodec decoder and ffmpeg does not? If not, what difference does it make to recompile mythffmpeg and mythffplay with the latest binaries from ffmpeg and ffplay? That would then include any fixes that the FFmpeg team has made since the current versions were compiled, including nvdec.

Unless the current versions of mythffmpeg and mythffplay can handle mediacodec and a recompile would not, I see only gains and no losses with a recompile.

[tmm1]

Since this thread is about NVDEV, maybe a new issue about mediacodec makes more sense. I don't want to distract or conflate two issues.

My understanding is that there is some mythtv android client. In that client, when you playback it uses mediacodec for hardware decoding. In this decoding setup, caption data is not available because the mediacodec decoders cannot parse and return it. Thus you must extract the caption data before decoding, then reattach it to the decoded output.

For decoding/playback, ffmpeg cli is not used but rather libavcodec is used directly.

However, since last week it is now possible to also run ffmpeg cli on android by using NDK directly instead of java. You can see the patchset here: https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_5F3D69DA7D4AADBE3995C32DB6EC685A010A@qq.com/

[gigem]

@bshanteau, I think your confusion stems from not knowing that MythFrontend runs on Android and not just Linux and Mac. For some MythTV users and developers, like me, the Android frontend running on Nvidia Shields is the primary frontend. Because of the mediacodec/subtitle issue being discussed here, we can't see subtitles when mediacodec is enabled.

The change @tmm1 is referring to is a patch he wrote some time ago for Channels DVR which fixes the issue. It's similar to the idea propesed by @ulmus-scott where the video stream is split and decoded by hardware for the video and separately by software for the subtitles. For those of us using the Android frontend, the inclusion of @tmm's patch or something similar would be greatly apprecieated. The catch is that the ffmpeg API has changed and the patch requires non-trivial porting before it can be used.

[ulmus-scott]

@tmm1 It appears you resubmitted the FFmpeg patch set https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190911185610.87081-2-ffmpeg@tmm1.net/ and it was rejected for the same reason: anonymous unions were not supported by all supported compilers. I assume this is still the case since anonymous unions are a C11 feature and FFmpeg is C99.

Is that the newest version of the patch set?

libavcodec/mpeg12dec.c also supports other subtitle embedding formats.

[tmm1]

Yes that's the newest version. I think I started to port it forward so I will find my WIP and share that.

Looks like I removed the unions already: https://github.com/FFmpeg/FFmpeg/compare/master...tmm1:FFmpeg:cbs-misc-a53

[tmm1]

Here is my WIP tree. Basically the way the BSF filters work changed pretty dramatically so a lot of the code has to be restructured and rewritten.

https://github.com/FFmpeg/FFmpeg/compare/master...tmm1:FFmpeg:cbs-misc-a53-v2

[bshanteau]

@tmm1 @gigem Thanks for the clarification.

Getting back to the subject issue, I'm trying to find out the details of what @kmdewaal describes:

there are now efforts to use the FFmpeg that is present on the system instead bundling a MythTV-specific version with MythTV

Where can I find these efforts? Is it MythTV/FFmpeg?

[bshanteau]

@kmdewaal @tmm1 @gigem Do I understand correctly that mythffmpeg is compiled from a different version and with a different configuration than the ffmpeg in the Ubuntu repository (or whatever)?

On my system (LInux Mint 21 XFCE based on Ubuntu 22.04) ffmpeg is version 4.4.2 while mythffmpeg is version 5.1. Also, the configurations are different (for instance, mythffmpeg is built with --disable-nvenc).

$ ffmpeg -version
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 70.100 / 56. 70.100
libavcodec     58.134.100 / 58.134.100
libavformat    58. 76.100 / 58. 76.100
libavdevice    58. 13.100 / 58. 13.100
libavfilter     7.110.100 /  7.110.100
libswscale      5.  9.100 /  5.  9.100
libswresample   3.  9.100 /  3.  9.100
libpostproc    55.  9.100 / 55.  9.100

$ mythffmpeg -version
ffmpeg version 5.1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
configuration: --arch=x86 --sysinclude=/usr/include --cc='ccache gcc' --cxx='ccache g++' --ld=gcc --cpu=generic --target_os=linux --pkg_config=pkg-config --prefix=/usr --libdir=/usr/lib --as=gcc --objcc=gcc --dep_cc=gcc --host_cc=gcc --host_ld=gcc --ar=ar --nm='nm -g' --ranlib='ranlib -D' --disable-cross-compile --enable-libmp3lame --enable-libx264 --disable-libx265 --enable-libvpx --disable-libxvid --enable-vdpau --enable-libxml2 --enable-libass --disable-dxva2 --enable-libbluray --disable-libfontconfig --disable-libfreetype --disable-libiec61883 --disable-crystalhd --disable-sdl2 --disable-ffplay --enable-vaapi --enable-libdrm --enable-gnutls --extra-cflags=' -w' --extra-ldflags= --extra-libs= --enable-stripping --strip='echo skipping strip' --disable-manpages --disable-podpages --disable-doc --disable-nvenc --enable-shared --disable-static --enable-gpl --enable-pic --disable-demuxer=mpegtsraw --disable-indev=dshow
libavutil      57. 28.100 / 57. 28.100
libavcodec     59. 37.100 / 59. 37.100
libavformat    59. 27.100 / 59. 27.100
libavdevice    59.  7.100 / 59.  7.100
libavfilter     8. 44.100 /  8. 44.100
libswscale      6.  7.100 /  6.  7.100
libswresample   4.  7.100 /  4.  7.100
libpostproc    56.  6.100 / 56.  6.100

[paul-h]

Yes the mythff* executables are compiled using our copy of the ffmpeg source and the configuration can be tweaked by passing options to our configure script. It should accept all the same configure options that the ffmpeg configure accepts unless something has changed.

[ulmus-scott]

@tmm1 I am open to the idea of rebasing the FFmpeg patch series, but I have a few questions first:

1. Do you know what commits broke the patch series? (This could help for updating them.)
2. Is adding code to each *_metadata_bsf.c the best approach/the proper place for this code?
3. How would I test this within FFmpeg?
4. How would we use this within MythTV?

Note: A/341 also provides for the CC data to be embedded in HEVC streams, identical to AVC via ANSI/SCTE 128-1.

---

@bshanteau https://github.com/MythTV/FFmpeg is our downstream fork of FFmpeg that we have customized over the years. It is copied into https://github.com/MythTV/mythtv/tree/master/mythtv/external/FFmpeg .

What @kmdewaal is referencing is probably my efforts to reduce our downstream changes on the way to the long term goal of using the system supplied FFmpeg, see https://github.com/MythTV/mythtv/issues/428 .

[tmm1]

1. ffmpeg/ffmpeg@01a68c12a7cdf9031c19d1f00885a84d58e2c2a6 and ffmpeg/ffmpeg@c837d9481a0035f3b547a5b417e5341e6e0d6128
2. yes i think so
3. something like -vf h264_metadata=a53_cc=extract
4. the filter would be wired up to the mediacodec decoder, something like

   #if CONFIG_H264_MEDIACODEC_DECODER
   -DECLARE_MEDIACODEC_VDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb")
   +DECLARE_MEDIACODEC_VDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb,h264_metadata=a53_cc=extract")
   #endif

[bshanteau]

@tmm1

What @kmdewaal is referencing is probably my efforts to reduce our downstream changes on the way to the long term goal of using the system supplied FFmpeg,

That would be a good thing. I hope my questions help. I know I am a novice in this stuff, so please let me know if my questions are helping or getting in the way.

Lately I've been wondering: What happens if I have both ffmpeg 4.2.4 from Ubuntu 22.04 repository and mythffmpeg 5.1 from mythbuntu/v33 with different configurations installed on my system (LInux Mint 21 XFCE)? Does that cause the libraries that each uses to get all mixed up?

And what does it mean if ffmpeg and mythffmpeg can both successfully process a recording with subtitles using the nvdec decoder from the command line but mythtv frontend's nvdec player cannot? ffmpeg -hwaccel nvdec -i 30787_20221123205900.ts -c:v h264 -f mpegts out1.mp4 mythffmpeg -hwaccel nvdec -i 30787_20221123205900.ts -c:v h264 -f mpegts out2.mp4 30787_20221123205900.ts.zip

[gigem]

@bshanteau, the system and MythTV versions of mmpeg can coexist just fine as the MythTV version uses different file and library names. The case that mythfrontend doesn't show subtitles with nvdec is a bug in how MythTV is using nvdec. I expect it will get fixed by those knowledgeable in that area soonish.

[ulmus-scott]

@tmm1

4. the filter would be wired up to the mediacodec decoder, something like

 #if CONFIG_H264_MEDIACODEC_DECODER
-DECLARE_MEDIACODEC_VDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb")
+DECLARE_MEDIACODEC_VDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb,h264_metadata=a53_cc=extract")
 #endif

Would this change be accepted by FFmpeg?

How would we attach the filter to the NVDEC decoders or the the mediacodec decoders without that FFmpeg change?

[bshanteau]

@gigem

The case that mythfrontend doesn't show subtitles with nvdec is a bug in how MythTV is using nvdec. I expect it will get fixed by those knowledgeable in that area soonish.

I just checked and the nvdec subtitle bug has been fixed in v33.0~master.202212052334.5c8d88cf42~ubuntu22.04.1

Thanks to all!!!

[bshanteau]

@gigem Nope. Bug reappeared after reboot.