Framerate mismatch in transport streams

[KungFuJesus]

• Platform: Linux mythtvbox 6.0.3-gentoo #1 SMP PREEMPT_DYNAMIC Sun Oct 23 21:42:04 EDT 2022 x86_64 Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHz GenuineIntel GNU/Linux

• MythTV version: 32/fixes

• Component: TV playback

What steps will reproduce the bug?

For specific segments of a stream from one of my local broadcasters, I get weirdly mismatched framerates. These issues seemingly occur where things get cut in. E.g. for SNL content, prerecorded segments versus the live broadcast seem to be affected by this. I don't know if it's an issue with my local broadcaster or directly a mythtv issue. I also am not sure if it's specific to interlaced content (I'm starting to think it's not, I may have seen this behavior on a 720p broadcast at one point).

It definitely seems like an issue that is very specific to the transport stream contents. Parts of it will be fine and when the issues do occur, the apparent/calculated framerate will end up averaging around 50fps (it's 29.97 fps content from an ATSC broadcast, it should be more like 60fps after being deinterlaced). Strangely, the issue goes away with VDPAU as the decoder (though, VDPAU does seem to have a separate issue in the deinterlacing for some fields occasionally, which is why I switched to nvdec. At least I get subtitles back with VDPAU).

I tried software decoding and it seemed to have even worse issues, but that may have been the software deinterlacing being too intense for the relatively weak CPU paired with this machine. The apparent framerate as reported with mpv also reports this strange 47-50fps values with hardware decoding (but somehow playback doesn't stutter through it). Interestingly using software decoding on another machine reports the apparent framerate as 23.976 fps (perhaps 3:2 pull down issues is what induces the issue?). I also somewhat wonder if only mythtv has this playback issue due to the way it now tries to synchronize audio.

I'm attaching a small example clip that should readily reproduce the issue. The very beginning, the end of a show, is fine. You see the stuttery mess on the next show with the goofy intro for the next one. Try decoding with nvdec, then vdpau. Interestingly mpv only reports the oddball framerate when the deinterlacer is enabled.
stutterissues.ts.gz

[KungFuJesus]

I initially thought this had been related to the fix for #548, but it appears like it may be unrelated to that.

[Jpilk]

Playback in master 761df3 seems fine to me with OpenGLnormal. NVDEC Normal and VDPAU Normal have minor blocking at around 22s, just after the picture frame. Deinterlacing is on during the singing segment. Encoding is mpeg-2. Overall frame rate with all decoders is around 24 fps. NVDEC shows no subtitles.

leanfront playback and subtitles are fine.

[Jpilk]

mythfrontend -v playback with NVDEC

2022-12-19 13:21:01.903340 I Player(4): SetFrameInterval Interval:16683 Speed:1 Scan:Progressive Scan (Multiplier: 2) 2022-12-19 13:21:01.912239 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:01.912579 I Player(4): SetFrameInterval Interval:33366 Speed:1 Scan:Progressive Scan (Multiplier: 1) 2022-12-19 13:21:01.928915 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:01.962283 I PlayerFPS: 46.96 Mean: 21296 Std.Dev: 9364 CPUs: 31% 31% 2022-12-19 13:21:01.987595 I Player(4): SetFrameInterval Interval:16683 Speed:1 Scan:Progressive Scan (Multiplier: 2) 2022-12-19 13:21:01.995197 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:01.996170 I Player(4): SetFrameInterval Interval:33366 Speed:1 Scan:Progressive Scan (Multiplier: 1) 2022-12-19 13:21:02.013995 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:02.071232 I Player(4): SetFrameInterval Interval:16683 Speed:1 Scan:Progressive Scan (Multiplier: 2) 2022-12-19 13:21:02.078953 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:02.080103 I Player(4): SetFrameInterval Interval:33366 Speed:1 Scan:Progressive Scan (Multiplier: 1) 2022-12-19 13:21:02.095289 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:02.154303 I Player(4): SetFrameInterval Interval:16683 Speed:1 Scan:Progressive Scan (Multiplier: 2) 2022-12-19 13:21:02.161909 I VideoOutput: Disabled all deinterlacing 2022-12-19 13:21:02.163710 I Player(4): SetFrameInterval Interval:33366 Speed:1 Scan:Progressive Scan (Multiplier: 1) 2022-12-19 13:21:02.179236 I VideoOutput: Disabled all deinterlacing

[KungFuJesus]

Perhaps it's also more noticeable to me because the frontend changes the refresh rate of the TV to match the video? Though, I think the TV is in 60 hz mode when I see it. When you leave the recording and come back, it ends up in a different spot.

Oh and you have to turn on the deinterlacer for it to be apparent. I don't know why your deinterlacer is being disabled, it's 1080i video. During that singing segment the framerate gets mismatched and weird. This is on fixes/32. The nvdec used is distributed with the drivers for 525.60.11

[Jpilk]

Extract from dev list, MarkK, Thu, 20 Feb 2020 10:43:34 +0000. I haven't investigated the current state, but he didn't like that deinterlacer.

The bigger issue is that when NVDEC deinterlacing is enabled, short backwards seeks are totally innacurate. This is an issue with how we handle the increased framerate seen when NVDEC does the deinterlacing. I'm considering switching to use the cuda-yadif deinterlacing filter - which will eliminate the framerate issue in most cases and give us full control of deinterlacing. Currently we can only enable NVDEC when the decoder is opened.

[KungFuJesus]

So reading a little bit more about this, I realized this is an NBC broadcast. Wikipedia's entry here has some interesting notes:

The ATSC specification and MPEG-2 allow the use of progressive frames coded within an interlaced video sequence. For example, NBC stations transmit a 1080i60 video sequence, meaning the formal output of the MPEG-2 decoding process is sixty 540-line fields per second. However, for prime-time television shows, those 60 fields can be coded using 24 progressive frames as a base – actually, an 1080p24 video stream (a sequence of 24 progressive frames per second) is transmitted, and MPEG-2 metadata instructs the decoder to interlace these fields and perform 3:2 pulldown before display, as in soft telecine.

So, perhaps something in this weird 1080p24 video in a 1080i60 envelope is causing nvdec to treat this weird?

[KungFuJesus]

Extract from dev list, MarkK, Thu, 20 Feb 2020 10:43:34 +0000. I haven't investigated the current state, but he didn't like that deinterlacer.

The bigger issue is that when NVDEC deinterlacing is enabled, short backwards seeks are totally innacurate. This is an issue with how we handle the increased framerate seen when NVDEC does the deinterlacing. I'm considering switching to use the cuda-yadif deinterlacing filter - which will eliminate the framerate issue in most cases and give us full control of deinterlacing. Currently we can only enable NVDEC when the decoder is opened.

Sadly it's actually the better deinterlacer of the 2 nvidia choices. It seems VDPAU occasionally screws up a few fields and you'll see the horizontal blinds effect (it actually might be on a fairly regular basis, though it's infrequent enough that you have to really look for it). I believe nvdec actually had this issue too at one point until someone fixed it, suggesting that maybe it's not in Nvidia's code. The lack of subtitles in nvdec is also annoying, but given the quality of the subtitles in ATSC for a lot of carriers, it wasn't that big of a loss for me.

I have 2x advanced on for the nvdec deinterlacer. The reported screwed up framerate in your logging is the tell tale sign the symptom is happening for me. I think my display is in 60hz mode so I don't think it's because I have the option to change the mode to match the video (the video itself is detected as 29.976 fps in the playback data). It could be the newer synchronization code with the audio fighting with the weird detected framerate post deinterlacing.

[KungFuJesus]

The mismatched framerate isn't as obvious in that scene and you do have to be watching for a bit. It's more discernible the longer into it you watch, particularly during dialog. VDPAU detects what I think is the proper framerate when everything is done deinterlacing (24 fps), but that 46ish fps framerate is choppy weird and wrong when played back with nvdec.

Also important to note other ffmpeg based players report that wrong apparent framerate when used in combination with nvdec + the deinterlacer but only mythtv gives the stutter (which makes me suspicious it's the audio synchronization).

[KungFuJesus]

Here's nvdec

Here's MPV, after toggling on the deinterlacer (without turning it on, it reported the proper 24fps framerate):

[KungFuJesus]

I also noticed this particular recording switches a lot between 29.97 fps television and 24 fps. Certain quick cuts will be 30fps while the vast majority of content will be 24fps. It definitely seems liked a weird 3:2 pulldown issue. Maybe the wrongly reported apparent framerate is just a red herring due to the doubling coming out of the deinterlacer?

[Jpilk]

My NVDEC log above was with the 470xx driver, single rate none and HQ double rate prefer driver deint. I'll try other options.

[KungFuJesus]

Yeah come to think of it - the places where I do see improperly deinterlaced content are the 30fps content. So perhaps VDPAU isn't deinterlacing at all and that's why it mostly looks ok?

I'm not entirely sure how decoders use the deinterlacing to do that 3:2 pulldown, perhaps the term deinterlacing in general is a misnomer for that. FWIW playing back in mpv without deinterlacing appears just fine except for those sequences that are cut in at 30fps (the baking it intro animation).

Toggling on the deinterlacer in real time with mpv (I kind of wish mythtv had the ability to do this) fixes it while not disrupting the 24fps content

[BrettLMiller]

It does, there are OSD playback menu options for interlacing etc.. well MythTV before MarkK changed everything.

[Jpilk]

Playback of the clip that you posted has no defects that I can see with OpenGL Normal and no deinterlacers. Subtitles work too.

The nVidia options without deinterlacing have blocking immediately after the picture-frame shot but playback is otherwise ok. No subtitles with NVDEC.

master 761df3, Fedora 36, Core2 duo, 2.66 GHz, GT 710, 470xx drivers

The on-screen playback info with HDMI input to my tv reports:

Video 1920x1080@29.97fps FPS mainly either 23.81 or 24.16 +/- 0.21

mythfrontend -nw -geometry 1920x1080+1680+0 -v playback:debug

2022-12-20 14:59:45.454979 D PlayerOverlay: GetSecondsPlayed: framesPlayed 405, honorCutList 1, pos 15923 2022-12-20 14:59:46.205949 I PlayerFPS: 23.80 Mean: 42018 Std.Dev: 8773 CPUs: 29% 24% 2022-12-20 14:59:46.455999 D PlayerOverlay: GetSecondsPlayed: framesPlayed 429, honorCutList 0, pos 16925 2022-12-20 14:59:46.456226 D PlayerOverlay: GetSecondsPlayed: framesPlayed 429, honorCutList 1, pos 16925 2022-12-20 14:59:47.406747 I PlayerFPS: 24.15 Mean: 41400 Std.Dev: 8783 CPUs: 30% 26% 2022-12-20 14:59:47.457052 D PlayerOverlay: GetSecondsPlayed: framesPlayed 453, honorCutList 0, pos 17921 2022-12-20 14:59:47.457181 D PlayerOverlay: GetSecondsPlayed: framesPlayed 453, honorCutList 1, pos 17921 2022-12-20 14:59:48.457928 D PlayerOverlay: GetSecondsPlayed: framesPlayed 477, honorCutList 0, pos 18918 2022-12-20 14:59:48.458153 D PlayerOverlay: GetSecondsPlayed: framesPlayed 477, honorCutList 1, pos 18918 2022-12-20 14:59:48.624828 I PlayerFPS: 23.81 Mean: 41996 Std.Dev: 8728 CPUs: 32% 24%

[KungFuJesus]

It does, there are OSD playback menu options for interlacing etc.. well MythTV before MarkK changed everything.

You can manually specify the scan but that doesn't seem to force on or off deinterlacing. At least as far as I can tell. The defect only appears with deinterlacing enabled with nvdec. I'm using 2x advanced.

The 24fps playback is the correct framerate and when I see that reported in the playback data (such as with VDPAU), the framerate looks fine. Perhaps I should post a youtube video of this to demonstrate the choppiness?

[Jpilk]

Thanks for the Wikipedia reference above. I hadn't noticed its continuation off the page. Making more sense now...

https://github.com/MythTV/mythtv/issues/680#issuecomment-1357749599

[KungFuJesus]

Also it's probably relevant to emphasize this is on fixes/32, I've not and cannot test this against master

[Jpilk]

I have run 'mythffprobe -hide_banner -v error -show_streams -show_format -of json' on your clip and on one beginning at the snow globe. Here's the video info for the first:

{ [mpeg2video @ 0x5626fdedc280] Invalid frame dimensions 0x0. "streams": [ repeated 18 times { "index": 0, "codec_name": "mpeg2video", "codec_long_name": "MPEG-2 video", "profile": "Main", "codec_type": "video", "codec_tag_string": "[2][0][0][0]", "codec_tag": "0x0002", "width": 1920, "height": 1080, "coded_width": 0, "coded_height": 0, "closed_captions": 0, "film_grain": 0, "has_b_frames": 1, "sample_aspect_ratio": "1:1", "display_aspect_ratio": "16:9", "pix_fmt": "yuv420p", "level": 4, "color_range": "tv", "chroma_location": "left", "field_order": "tt", "refs": 1, "id": "0x51", "r_frame_rate": "30000/1001", "avg_frame_rate": "30000/1001", "time_base": "1/90000", "start_pts": 4827109713, "start_time": "53634.552367", "duration_ts": 2795793, "duration": "31.064367", "extradata_size": 150, "disposition": { "default": 0, "dub": 0, "original": 0, "comment": 0, "lyrics": 0, "karaoke": 0, "forced": 0, "hearing_impaired": 0, "visual_impaired": 0, "clean_effects": 0, "attached_pic": 0, "timed_thumbnails": 0, "captions": 0, "descriptions": 0, "metadata": 0, "dependent": 0, "still_image": 0 }, and this is from the shorter clip:

       "pix_fmt": "yuv420p",
        "level": 4,
        "color_range": "tv",
        "chroma_location": "left",
        "field_order": "progressive",
        "refs": 1,
        "id": "0x100",
        "r_frame_rate": "60000/1001",
        "avg_frame_rate": "30000/1001",
        "time_base": "1/90000",
        "start_pts": 274149,
        "start_time": "3.046100",
        "duration_ts": 1142641,
        "duration": "12.696011",
        "extradata_size": 150,

Both quoted durations are longer than they actually play for. The field_order and r_frame_rate seem to be the significant differences.

[KungFuJesus]

While undoubtedly this could very well be something the broadcaster is doing wrong - they are doing it reliably wrong :-/. Even if I could say, email the engineers working for the station to correct it, assuming we identify what "it" is that's causing this, my guess is that they are unlikely to do anything if televisions and other things receiving this OTA broadcast seem to be immune to it.

It seems like an issue that has to be worked around in software some sort of way.

[Jpilk]

I redid some of this on the el7 box running v32 and using a dd-type cutting tool (pycut.py) with no demuxing and no changing of timestamps. No GPU available and no stuttering. Similar stream analysis.

Clip as posted

ionice -c 3 mythffprobe -hide_banner /home/john/SGs/RcSG2/11156_20221221103500.ts

Input #0, mpegts, from '/home/john/SGs/RcSG2/11156_20221221103500.ts': Duration: 00:00:26.83, start: 53633.779067, bitrate: 6627 kb/s Stream #0:0[0x51]: Video: mpeg2video (Main), yuv420p(tv, top first), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc Side data: cpb: bitrate max/min/avg: 15934400/0/0 buffer size: 7995392 vbv_delay: N/A Stream #0:10x54: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 384 kb/s Stream #0:20x56: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s

Clip starting at snowglobe

ionice -c 3 mythffprobe -hide_banner /home/john/SGs/RcSG2/11156_20221221103500.ts

Input #0, mpegts, from '/home/john/SGs/RcSG2/11156_20221221103500.ts': Duration: 00:00:19.19, start: 53639.539067, bitrate: 7015 kb/s Stream #0:0[0x51]: Video: mpeg2video (Main), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 59.94 tbr, 90k tbn, 59.94 tbc Side data: cpb: bitrate max/min/avg: 15934400/0/0 buffer size: 7995392 vbv_delay: N/A Stream #0:10x54: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 384 kb/s Stream #0:20x56: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s

No doubt mythffprobe -read_intervals xxx would do this but tl:dr

ffprobe version 4.4.1 Copyright (c) 2007-2021 the FFmpeg

Myth v32.0-2033e46576 built 15 Sep 2022

[KungFuJesus]

The issue is apparent on another frontend running from earlier git revision of fixes/32 as well, with nvidia driver 515.57, myth commit hash 23725516afe. Maybe I can capture video, upload it to youtube, and hope it doesn't get DMCA flagged?

[Jpilk]

There's always the possibility that my rebuilt seektable doesn't match yours built during recording. I don't know if that might make a difference. The easiest way I know of copying is via the .xml output of native-mode mytharchive; but never restore the full archive to a different box because the recordedids will clash.

Or you could just see if 'mythcommflag --rebuild --file xxxxx_yyyymm...ts' affects the stutter.