Hardware acceleration scales image incorrectly on AMD with VDPAU

[clee]

Describe the bug On an ultrawidescreen (21:9) client with a 16:9 source, the image is scaled correctly only when I choose "Force software decoding". Both "Automatic (Recommended)" and "Force hardware decoding" result in incorrect scaling, cropping off the sides of the image.

Steps to reproduce

• Windows machine running GameStream: 2560x1440
• Linux desktop connecting to Windows via Moonlight: 3440x1440
• "Video decoder" set to "Force hardware" or "Automatic (Recommended)"

Screenshots Broken scaling with hardware decoding:

Correct scaling, with forced software decoding:

Affected games All games

Moonlight settings (please complete the following information) Resolution and FPS: 1440p / 100fps, 80mbps bitrate Display mode: Full-screen (Recommended) V-Sync on, Frame pacing on

Advanced settings: Video decoder: broken scaling with "Force hardware decoding" and "Automatic (Recommended)", proper scaling with "Force software decoding" Video codec: Force H.264

Client PC details (please complete the following information)

• OS: Fedora 29 amd64
• Moonlight Version: v0.8.1-21aad643f3fc5b85d38f10f1b1ea4389c92f4258 (also happens with v0.8.0, and happened with previous release as well)
  • GPU: Radeon RX Vega 56
  • Driver: Mesa Gallium driver 18.2.8 for Radeon RX Vega (VEGA10, DRM 3.27.0, 4.19.15-300.fc29.x86_64, LLVM 7.0.0), using VDPAU because Avoiding VAAPI on AMD driver

Server PC details (please complete the following information)

• OS: Windows 10 1803, 64-bit
• GeForce Experience version: 3.16.0.140
• Nvidia GPU driver: 417.71

Moonlight Logs (please attach) vd-automatic.log vd-hardware.log vd-software.log

[clee]

Update: after consulting with @cgutman on Discord and testing FORCE_VAAPI=1, this definitely seems to be a problem with VDPAU scaling on AMD. Luckily, the VAAPI memory leak seems to be fixed for me, so FORCE_VAAPI=1 is a useful workaround until VAAPI is either manually-selectable in the UI or enabled by default on AMD cards on Linux.

[cgutman]

Can you try with this change and see if it fixes VDPAU for you?

diff --git a/app/streaming/video/ffmpeg-renderers/vdpau.cpp b/app/streaming/video/ffmpeg-renderers/vdpau.cpp
index 53fc81a..b1b01c2 100644
--- a/app/streaming/video/ffmpeg-renderers/vdpau.cpp
+++ b/app/streaming/video/ffmpeg-renderers/vdpau.cpp
@@ -335,8 +335,8 @@ void VDPAURenderer::renderFrameAtVsync(AVFrame* frame)
                                    0, nullptr,
                                    &sourceRect,
                                    chosenSurface,
-                                   &outputRect,
                                    nullptr,
+                                   &outputRect,
                                    0,
                                    nullptr);

[clee]

Moving the &outputRect after the nullptr totally fixes the VDPAU scaling for me.