LizardByte / Sunshine

Self-hosted game stream host for Moonlight.
http://app.lizardbyte.dev/Sunshine/
GNU General Public License v3.0
20.12k stars 965 forks source link

NVENC acceleration is not detected on Windows #225

Closed dannytech closed 2 years ago

dannytech commented 2 years ago

Describe the Bug

On a Windows 10 host, when I start Sunshine, even as administrator, it fails to detect my NVENC encoder (a GTX 1660 Super). I've included the log below of it trying NVENC, AMF, and software encoders, and falling back to software. I've also tried forcing Sunshine to use NVENC, but then it just crashes on startup because it can't find the NVENC encoder.

[2022:06:24:16:23:35]: Info: Trying encoder [nvenc]
[2022:06:24:16:23:35]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:35]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:35]: Info: Color range: [JPEG]
[2022:06:24:16:23:35]: Error: Could not open codec [h264_nvenc]: Invalid argument
[2022:06:24:16:23:35]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:35]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:35]: Info: Color range: [JPEG]
[2022:06:24:16:23:35]: Error: Could not open codec [h264_nvenc]: Invalid argument
[2022:06:24:16:23:35]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:35]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:35]: Info: Color range: [JPEG]
[2022:06:24:16:23:35]: Error: Could not open codec [h264_nvenc]: Invalid argument
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Error: Could not open codec [h264_nvenc]: Invalid argument
[2022:06:24:16:23:36]: Info: Encoder [nvenc] failed
[2022:06:24:16:23:36]: Info: Trying encoder [amdvce]
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Error: Could not open codec [h264_amf]: Unknown error occurred
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Error: Could not open codec [h264_amf]: Unknown error occurred
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Error: Could not open codec [h264_amf]: Unknown error occurred
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Error: Could not open codec [h264_amf]: Unknown error occurred
[2022:06:24:16:23:36]: Info: Encoder [amdvce] failed
[2022:06:24:16:23:36]: Info: Trying encoder [software]
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[x2022:06:24:16:23:36]: Info: Color coding [Rec. 709]2
6[42022:06:24:16:23:36]: Info: Color range: [JPEG]
 [error]: high profile doesn't support a bit depth of 10
[2022:06:24:16:23:36]: Error: Could not open codec [libx264]: Invalid argument
[2022:06:24:16:23:36]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 6140 MiB
Feature Level      : 0x0000B100
Capture size       : 2560x1440
Offset             : 0x0
Virtual Desktop    : 2560x1440
[2022:06:24:16:23:36]: Info: Color coding [Rec. 601]
[2022:06:24:16:23:36]: Info: Color range: [JPEG]
[2022:06:24:16:23:37]: Warning: software: h264: replacing nalu prefix data
[2022:06:24:16:23:37]: Info:
[2022:06:24:16:23:37]: Info: //////////////////////////////////////////////////////////////
[2022:06:24:16:23:37]: Info: //                                                          //
[2022:06:24:16:23:37]: Info: // Ignore any errors mentioned above, they are not relevant //
[2022:06:24:16:23:37]: Info: //                                                          //
[2022:06:24:16:23:37]: Info: //////////////////////////////////////////////////////////////
[2022:06:24:16:23:37]: Info:
[2022:06:24:16:23:37]: Info: Found encoder software: [libx264]
[2022:06:24:16:23:37]: Info: Configuration UI available at [https://localhost:47990]
[2022:06:24:16:23:37]: Info: Registered Sunshine Gamestream service

Expected Behavior

I wouldn't expect AMF to work so that's fine, but NVENC should be succeeding on h264_nvenc (and HEVC, if it checks for that), and I also see that libx264 had some errors, although when I actually stream it seems to work fine with software encoding (besides noticeably high latency).

Additional Context

I am running inside a virtual machine with the GPU passed through, should that matter. However, I have tested other programs and they work fine with this setup. Parsec, for example, is able to use the Nvidia GPU for encoding, so the NVENC is definitely accessible:

image

I also installed Sunshine using the new installer rather than the portable Zip, but I don't believe this could cause any issues.

Sunshine Host Operating System and Version

Windows 10 19043.1766

Architecture

64-bit

Sunshine Version

0.14.0

GPU Type

Nvidia

GPU Model

GeForce GTX 1660 Super

GPU Driver/Mesa Version

516.40

Capture Method (Linux Only)

No response

perroboc commented 2 years ago

I'm using windows 11 and have the same GPU (1660 Super), and everything works fine. So it must be something to do with the GPU passthrough you're using.

I tested this using the portable zip file:

Start:

[2022:07:01:15:43:10]: Info:
Device Description : NVIDIA GeForce GTX 1660 SUPER
Device Vendor ID   : 0x000010DE
Device Device ID   : 0x000021C4
Device Video Mem   : 5992 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 8147 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2022:07:01:15:43:10]: Debug: Source format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2022:07:01:15:43:10]: Info: Color coding [Rec. 601]
[2022:07:01:15:43:10]: Info: Color range: [JPEG]                                                                                              

End:

[2022:07:01:15:43:11]: Warning: nvenc: hevc: replacing nalu prefix data
[2022:07:01:15:43:11]: Info:
[2022:07:01:15:43:11]: Info: //////////////////////////////////////////////////////////////
[2022:07:01:15:43:11]: Info: //                                                          //
[2022:07:01:15:43:11]: Info: // Ignore any errors mentioned above, they are not relevant //
[2022:07:01:15:43:11]: Info: //                                                          //
[2022:07:01:15:43:11]: Info: //////////////////////////////////////////////////////////////
[2022:07:01:15:43:11]: Info:
[2022:07:01:15:43:11]: Debug: ------  h264 ------
[2022:07:01:15:43:11]: Debug: PASSED: supported
[2022:07:01:15:43:11]: Debug: REF_FRAMES_RESTRICT: supported
[2022:07:01:15:43:11]: Debug: REF_FRAMES_AUTOSELECT: supported
[2022:07:01:15:43:11]: Debug: SLICE: supported
[2022:07:01:15:43:11]: Debug: CBR: supported
[2022:07:01:15:43:11]: Debug: DYNAMIC_RANGE: unsupported
[2022:07:01:15:43:11]: Debug: VUI_PARAMETERS: supported
[2022:07:01:15:43:11]: Debug: NALU_PREFIX_5b: supported
[2022:07:01:15:43:11]: Debug: -------------------
[2022:07:01:15:43:11]: Debug: ------  hevc ------
[2022:07:01:15:43:11]: Debug: PASSED: supported
[2022:07:01:15:43:11]: Debug: REF_FRAMES_RESTRICT: supported
[2022:07:01:15:43:11]: Debug: REF_FRAMES_AUTOSELECT: supported
[2022:07:01:15:43:11]: Debug: SLICE: supported
[2022:07:01:15:43:11]: Debug: CBR: supported
[2022:07:01:15:43:11]: Debug: DYNAMIC_RANGE: unsupported
[2022:07:01:15:43:11]: Debug: VUI_PARAMETERS: supported
[2022:07:01:15:43:11]: Debug: NALU_PREFIX_5b: unsupported
[2022:07:01:15:43:11]: Debug: -------------------
[2022:07:01:15:43:11]: Info: Found encoder nvenc: [h264_nvenc, hevc_nvenc]
[2022:07:01:15:43:11]: Info: Configuration UI available at [https://localhost:47990]
[2022:07:01:15:43:12]: Info: Registered Sunshine Gamestream service
github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 30 days with no activity. Comment or remove the stale label, otherwise this will be closed in 5 days.

github-actions[bot] commented 2 years ago

This issue was closed because it has been stalled for 5 days with no activity.

luizribeiro commented 1 year ago

@dannytech, were you ever able to figure this out? I'm running into exactly the same issue, but with a 3070. I also do GPU passthrough, so that could indeed be related.

da-anda commented 1 year ago

same issue, also GPU passthrough. Native gamestream worked fine, but since it's subject of being shut down, I am looking for an replacement like sunshine.

da-anda commented 1 year ago

could it be related to the fact that the GPU has no monitor attached since the error message shown during enumeration of nvenc is ERROR: Failed to locate an output device and does not necessarily have anything to do with PCI passthrough?

dannytech commented 1 year ago

No, in my case and I would assume everyone else's, there was a monitor attached (this is usually required in order to take advantage of game streaming). PCI passthrough definitely seems to be the problem. I never got it working.

da-anda commented 1 year ago

well, it at least did the trick for me. Just played a couple seconds Portal RTX via sunshine running inside a win10 VM. The VM is running on a KVM/Quemu based hypervisor with all libvirt drivers installed.

da-anda commented 1 year ago

did a new installation of Windows and once again am facing this issue. And SteamLink streaming unfortunately sucks, so is also not an alternative. Could this issue be re-opened please as it is still not fixed? Issue is that Sunshine fails to enumerate the available encoders inside a virtual machine with GPU passthrough.

KuleRucket commented 1 year ago

It probably got closed because nobody has been able to reproduce and fix it. You could try running it with verbose logging and post the results to see if that reveals any more information.

da-anda commented 1 year ago

created a new ticket for the VM passthrough issue since it might be a different cause than the one from this ticket. #844 Will try to grab a verbose log. Oddly, after I had installed the nightly over the current stable I had just tried, it worked again until I did a reboot.

ReenigneArcher commented 1 year ago

FYI - Issues are automatically closed after 100 days with no activity. A single comment will reset the timeline... but comments after the issue are closed will not automatically re-open it.

da-anda commented 1 year ago

@KuleRucket I tried verbose logging (min_log_level=0, also tried verbose as it's mentioned in the docs) but it didn't change the log output on startup, so no further insights here

fidoboy commented 6 months ago

The issue still happens as today. Using also a 1660 SUPER. First time launching throws:

Error: NvEnc: encoding format is not supported by the gpu

After the first time error, restarting sunshine it works normally again. But first time connecting from Moonlight always fails with error mentioned.