Blank screen when trying to use NVENC on Nvidia card to encode stream from monitor on AMD card

[peacey]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your issue described in the documentation?

• [X] I have read the documentation

Is your issue present in the nightly release?

• [x] This issue is present in the nightly release

Describe the Bug

On a dual-GPU linux system with Nvidia and AMD, using the Nvidia NVENC encoder with KMS capture from a monitor connected to an AMD GPU results in a blank (black) stream and the following error:

Feb 13 12:21:21 servy sunshine[401261]: [2024:02:13:12:21:21]: Error: Couldn't import RGB Image: 00003004 Feb 13 12:21:21 servy sunshine[401261]: [2024:02:13:12:21:21]: Error: Could not convert image

Using the VA-API encoder with the AMD GPU while capturing from the AMD GPU works fine. However, I want to capture from my AMD card via KMS but encode with the Nvidia card. The documentation does not say there is a restriction on using the same GPU for encoder and capture, so I suspect this could be a bug?

Expected Behavior

Stream should work (no blank screen). More specifically, the stream that is captured from AMD GPU Monitor via KMS grab should be encoded with Nvidia GPU (NVENC) and sent to client without issue.

Additional Context

This problem happens with both HDR or SDR color coding trying to stream anything (simplest case I tested is the Desktop). This was only tested on Wayland using Kwin (Plasma 6 RC2). Monitor is found with sunshine, and stream works with vaapi when using same GPU for capture and encoding.

Host Operating System

Linux

Operating System Version

Arch Linux Testing

Architecture

64 bit

Sunshine commit or version

69a3edd9b01c76aa44fd5c2a29de1c3b3722cb41

Package

Linux - PKGBUILD

GPU Type

Nvidia

GPU Model

RTX 3060 for encoding, AMD RTX 7900 XTX for monitor output.

GPU Driver/Mesa Version

Nvidia 550 beta, AMD Linux 6.7.4 amdgpu driver

Capture Method (Linux Only)

KMS

Config

encoder = nvenc
fps = [10,30,60,90,120,144]
adapter_name = /dev/dri/renderD129

Apps

Desktop

Relevant log output

Feb 13 16:08:47 servy sunshine[17484]: [2024:02:13:16:08:47]: Info: Found H.264 encoder: h264_nvenc [nvenc]
Feb 13 16:08:47 servy sunshine[17484]: [2024:02:13:16:08:47]: Info: Found HEVC encoder: hevc_nvenc [nvenc]
Feb 13 16:08:47 servy sunshine[17484]: [2024:02:13:16:08:47]: Info: Executing [Desktop]
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: CLIENT CONNECTED
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found display [wayland-0]
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found interface: zxdg_output_manager_v1(31) version 3
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found interface: wl_output(59) version 4
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Resolution: 2304x1440
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Offset: 0x0
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Logical size: 2304x1440
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Name: DP-6
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found monitor: LLV 0x2160/66662160
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: DP-6: LLV 0x2160/66662160
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Screencasting with KMS
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found monitor for DRM screencasting
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found connector ID [106]
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Found cursor plane [76]
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: SDR color coding [Rec. 601]
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Color depth: 8-bit
Feb 13 16:08:48 servy sunshine[17484]: [2024:02:13:16:08:48]: Info: Color range: [MPEG]
Feb 13 16:08:49 servy sunshine[17484]: [2024:02:13:16:08:49]: Error: Couldn't import RGB Image: 00003004
Feb 13 16:08:49 servy sunshine[17484]: [2024:02:13:16:08:49]: Error: Could not convert image

[kentyman23]

I'm not sure if it should work or not, but I know I've seen cross-GPU encoding discouraged in both the Sunshine and Moonlight Discord channels. [1, 2]

cross-gpu encoding (copying frames back and forth through system memory) is hardly worth the worse performance and power efficiency.

[cgutman]

I'm not sure if it should work or not, but I know I've seen cross-GPU encoding discouraged in both the Sunshine and Moonlight Discord channels.

Yes, what you're trying to do is not recommended. Even if it can use the DMA engines to offload the copy rather than having to go through CPU, it's still worse performance, latency, and power efficiency than just encoding on the card where the frame is already present.

I suspect the issue here is that the framebuffer has vendor-specific tiling modifiers. Those vendor modifiers are only understood by the the original GPU, so the import fails on the Nvidia GPU because it doesn't understand AMD's tiling format. We could copy the frame buffer into a linear buffer on the original GPU then try to import that, but that's just adding more copies and overhead to something that's already non-ideal.

In the future, Pipewire (or even wlgrab/X11 capture backends today) might handle this modifier issue properly by providing linear buffers that are owned by the compositor itself that could be directly imported on the target GPU. However, we're not there today and that still comes with all the caveats I mentioned earlier about those copies not being free.

[Formedras]

It would be nice, though, to still be able to do this for edge cases, such as where the display card is already transcoding for other purposes (e.g. Twitch streaming) and incapable of handling another stream. Or perhaps for GPU-thermally-constrained systems where adding another stream to the GPU causes throttling, but giving it to the integrated graphics may be acceptable.

[ISAndreiva]

Another use case for this is the fact that AMD's VA-API encoder sucks and loves to overshoot bitrate limits (at least for Sunshine) while Nvidia's nvenc is perfectly stable, is said to be better-looking and an nvenc capable gpu is pretty cheap to buy and use just for encoding. Also this bug only appears if CUDA is available, without CUDA the Nvidia GPU can be used for encoding just fine but the performance is terrible, based on the warning message it seems to do GPU -> CPU -> GPU which would explain the performance.

[zhicheng233]

In a Windows environment, it seems that no matter if I force the encoder, it always chooses CPU encoding. PS: My primary GPU is an AMD Radeon RX 6500 XT, which lacks encoding functionality. The other one is an NVIDIA GeForce GT 710, but it is not connected to a monitor. In OBS, they work fine.