Constant twitching with -video-sync=display-resample

[YaLTeR]

Trying to watch a 120 FPS Twitch stream with -video-sync=display-resample results in constant video twitching. Same result on display set to 60 Hz, 120 Hz and 144 Hz. 60 FPS streams are fine. -video-sync=audio sometimes also results in constant video twitching and sometimes twitches a little and then normalizes.

mpv version and platform

Arch Linux x64

mpv 0.25.0 (C) 2000-2017 mpv/MPlayer/mplayer2 projects
 built on Sun Apr 23 12:26:53 CEST 2017
ffmpeg library versions:
   libavutil       55.58.100
   libavcodec      57.89.100
   libavformat     57.71.100
   libswscale      4.6.100
   libavfilter     6.82.100
   libswresample   2.7.100
ffmpeg version: 3.3

Reproduction steps

Open a 120 FPS Twitch stream with -video-sync=display-resample.

Expected behavior

Smooth video playback.

Actual behavior

Video constantly twitching, unwatchable.

Log file

display-resample, twitching: http://sprunge.us/LJFW audio, twitching: http://sprunge.us/CRHD audio, twitching then normal: http://sprunge.us/ZTjV

[YaLTeR]

Update: also getting the twitching on various 120 FPS videos. Let's see if I can find a small one with the problem so I can upload it and link here.

[YaLTeR]

Here's a sample file which causes twitching: https://0x0.st/Ytk.mp4 And here's a log for it: http://sprunge.us/bQiK This one causes constant twitching with both default video-sync and display-resample.

EDIT: this might be a regression because I used to be able to play back this file fine.

[haasn]

That video seems to twitch back and forth for me even without display-resample or interpolation, although interpolation being enabled does add some extra amounts of stutter on top.

I can quite clearly see the camera shaking even by framestepping through the video. Does it play fine in any other players?

[mia-0]

That’s clearly just the video, there’s even motion blur on it.

[haasn]

@lachs0r Even ignoring the parkinson's camera, turning on interpolation makes the playback choppy/stuttery for me on top of that - although that might just be because my GPU can't keep up with the 120 Hz rendering.

[YaLTeR]

Yes, there is slight camera twitching in the video. But the twitching I'm talking about is 10x more severe and back and forth in time as opposed to camera going left and right every frame. I knew it was going to cause confusion but I didn't have a better example at hand.

[YaLTeR]

Here's a better example video: https://0x0.st/YxE.mkv Log file with display-resample twitching: http://sprunge.us/SAJG

[Jj0YzL5nvJ]

The first video have a "like SVP" effect / glitch, if you see frame per frame is very obvious. I don't see anything wrong in the second video.

[YaLTeR]

Here's a recording showcasing the problem on the last example video I posted: https://0x0.st/Y30.mkv I play it back three times: First, ffplay — no issue. Second, mpv with display-resample — twitching. Third, mpv with default options — twitching, then after I switch in and out of fullscreen the twitching is gone.

[haasn]

Second video plays fine for me as well.

@YaLTeR Can you confirm whether or not it still happens with mpv --no-config --video-sync=display-resample?

[YaLTeR]

Doesn't happen with --no-config. My config:

profile=opengl-hq
hwdec=auto
ytdl-format="bestvideo[fps=60]+bestaudio/bestvideo+bestaudio/best"

Further investigating, looks like hwdec=auto is causing it because setting just --hwdec=no without --no-config fixes the issue. I guess this could be a regression in hwdec = vdpau then? Because there used to be no issue.

[Jj0YzL5nvJ]

Hardware acceleration has limits, and I'm almost sure that 60 FPS is one of them. In my case is 30 FPS (ATI Radeon HD 5750). The 2nd and 3rd video report wrong FPS (1000.000FPS) http://sprunge.us/fRgC

[YaLTeR]

I used to be able to watch 100, 120 and 144 FPS videos perfectly fine with these settings. GTX 970. Besides, if it was a hwdec limitation the twitching wouldn't disapper shortly with video-sync=audio.

[haasn]

@YaLTeR: Maybe hwdec=vdpau didn't work on these samples in the past?

[YaLTeR]

Well, I had hwdec=auto in my config for quite a while I believe and I haven't changed the PC parts in a long time. I'll see if I can set up old mpv versions for testing tomorrow.

[YaLTeR]

Tried different mpv versions and it looks like the problem appeared all the way back in v0.18.0. The last version where hwdec=vdpau and video-sync=display-resample work smoothly for all videos I posted and some others is v0.17.0. Here's a log for the last example video from that version: http://sprunge.us/eGjO

[YaLTeR]

Did bisecting, 761eeacf5407cab07a48e63d3dc215facf4d9650 is the first bad commit.

[ghost]

Can you double-check this? It seems very unlikely to trigger it.

[YaLTeR]

Double checked. Here's log for working previous commit 7be37337f4: http://sprunge.us/EeAS And here's log for broken commit 761eeacf54: http://sprunge.us/AQZT

[ghost]

Besides the version number, the only difference in that log is:

-[vo/opengl-hq] GLX chose FB config with ID 0x2c2325d0
+[vo/opengl-hq] GLX chose FB config with ID 0x682325d0

which seems suspicious, but absolutely can't be triggered by the commit you pointed out.

The previous commit (which you claim is working) seems way more likely to trigger this issue.

[YaLTeR]

¯\_(ツ)_/¯ Here's a video of me building and showcasing these two commits, maybe I'm doing something wrong: https://0x0.st/YlU.mkv

[tchebb]

I'm hitting what I believe to be the same issue, except I'm seeing it when I play a 24FPS movie and have my display's refresh rate set to 24 or 25FPS. The issue only appears with video-sync=display-*, hwdec=vdpau, and vo=opengl (or opengl-hq), and only when the video is fullscreened. (Notably, setting vo=vdpau fixes the issue.) If I set my display's refresh rate to something significantly higher than the video's frame rate (60 or 96FPS), it works fine. I'm using the nvidia 387.22 driver, and have not yet attempted to bisect.

[tchebb]

Some notes to myself while I attempt to debug this:

• The rendering on current mpv master (under the conditions described in my last comment) still has problems, but is significantly better. The twitching is gone from most of the frame, but still occasionally manifests at the very top.
• The commit that improved the rendering was 5261d1b099edf16712582790085f4d369438dfe5, which is effectively a revert of the commit @YaLTeR originally identified. I'm completely confident of this, as I can selectively revert that commit from the current master and the whole-screen twitching comes back. As noted above, though, this commit seems to have merely mitigated the symptoms rather than fixing the underlying issue.
• With or without that commit reverted, running with --opengl-glfinish eliminates the problem. This, along with the content of the relevant commits, indicates to me that the VDPAU surface is being freed too early somehow. I'm currently trying to work my way through the pipeline to figure out how to confirm and fix this.

[tchebb]

I've tracked the issue down to improper synchronization between the VDPAU and OpenGL command streams. Since each one is a separate context, there's no guarantee that a VDPAU command issued by the CPU will complete after GL commands issues prior to it.

This becomes a problem in ra_hwdec_mapper_unmap() (in video/out/gpu/hwdec.c), which releases the buffers associated with a hardware-decoded frame after the rendering GL commands are issued (but not necessarily before they complete). The call to mapper->driver->unmap() issues only GL commands (for VDPAU, at least) and so is not a problem. The call to mp_image_unrefp(), however, ultimately calls release_decoder_surface() (in video/vdpau.c), which marks the VDPAU surface as no longer in use, allowing it to be used as a target for the next frame that gets decoded. The decoding is done by ffmpeg and uses VDPAU commands (obviously). So, due to the aforementioned lack of synchronization, it happily clobbers the old frame while the GL commands are still accessing it.

The reason 761eeacf5407cab07a48e63d3dc215facf4d9650 and 5261d1b099edf16712582790085f4d369438dfe5 worsened and improved the symptoms, I believe, is because they caused the CPU to mark the surface as "not in use" earlier and later, respectively. When the surface was freed earlier, it could be grabbed by the decoder thread immediately and overwritten with a different frame before the GL commands saw it at all. When it was freed later, most of it could be used before another frame began to overwrite it. (I haven't worked out the timing in detail yet, which is why this explanation is a bit hand-wavey.) In my test environment, obviously fullscreen and --video-sync also have an effect on timing, but I haven't explored that in detail at all yet.

The fix to this probably isn't trivial. Some sort of GL fencing is obviously needed to prevent the VDPAU surface from being reused before GL is done with it. It might be possible to reuse some of the direct rendering code, which also has to do explicit GL synchronization, but the existing architecture doesn't seem like a great fit for what we need.

Note that I'm fairly new to GL and entirely new to VDPAU, and I haven't traced through the timings in detail yet, so this explanation could very well be wrong. However, it seems to fit what I'm seeing and the issue disappears when I add a hack to prevent the VDPAU surfaces from ever being released (at least until VDPAU stops working entirely due to memory exhaustion).

[haasn]

It might be possible to reuse some of the direct rendering code, which also has to do explicit GL synchronization, but the existing architecture doesn't seem like a great fit for what we need.

I wonder if we could somehow move the responsibility of calling mp_image_unrefp call to the hwdec mapper unmap() code; then the vdpau hwdec mapper code would be free to insert whatever synchronization fences it needs to ensure there's no early free.

[tchebb]

@haasn The mapper does currently keep its own reference to the mp_image, so it would be feasible for ra_hwdec_mapper_unmap() to do the synchronization. That code is API-independent, though, so it couldn't directly insert GL fences—it would have to rely on (probably new) ra_ methods for the actual fencing.

[haasn]

To be clear, I was referring to the actual vdpau mapper implementation - not the generic mapper code.

[tchebb]

Isn't that code vo-independent, though? I don't see how it could have sufficient knowledge of how the surface is used or access to the output API to insert fences.

[rob-sve]

Just wanted to add for others that might get this: I ran into this issue with twitching at the top of the screen as @tchebb described, and tried out setting vo=vdpau in my mpv.conf which seemed to solve it. While that did fix the twitching, it also made mpv freeze (as well as X11 itself momentarily) when exiting fullscreen. Then I googled around for that issue, and found #5307 where it was suggested that you disable flipping in nvidia-settings. That did not fix the freezing issue for me, but it did fix the twitching issue even after I remove the vo=vdpau setting.

So could this issue also be related to having 'Allow Flipping' enabled in nvidia-settings?

I've also previously turned on Triple Buffering in the Xorg config because I had issues with vsync in KDE Plasma.

[tchebb]

@rsvensson Thanks for the info! If my theory about what's causing this is correct, the problem can occur with any combination of settings when OpenGL/VDPAU is used, but only will occur when the GPU attempts to read the VDPAU surface after the CPU frees it. As such, it seems entirely reasonable that changing settings which affect the code path on either the CPU or GPU can make the problem appear and disappear.

[garoto]

Might wanna use auto-profiles.lua to only apply video-sync and interpolation to videos where they really matter I guess.

[interpol]
    profile-desc=cond:get("container-fps", math.huge) <= 45
    video-sync=display-resample
    interpolation=yes

[ghost]

From a very superficial glance over this issue, it seems the cause and fix is well known. Who wants to write a patch? Also why is anyone still using vdpau?