search

LizardByte / Sunshine

Self-hosted game stream host for Moonlight.
http://app.lizardbyte.dev/Sunshine/
GNU General Public License v3.0
19.75k stars 954 forks source link

Sunshine can't use nvenc or cuda on fedora 35 with latest nvidia drivers and cuda installed. #95

Closed hradec closed 2 years ago

hradec commented 2 years ago

Describe the Bug

I've tried to run sunshine (booth binary and building it myself) on fedora35, and it constantly fails to initialize nvenc, cuda and libva, with "unknown errors", despite the distro installation having the latest nvidia drivers and cuda installed. (downloaded from nvidia website and installed, not from any repository!)

At the end, it reverts to use libx264. :(

I've installed ffmpeg from rpmfusion dnf repositories, which seems to be version 4.4.1. (I'm not sure if this ffmpeg has nvenc support.)

this is the log I get when running sunshine (booth the binary downloaded from github and the one I've built spit out the same log!):

[2022:03:13:21:41:17]: Error: Failed to create session: This hardware does not support NvFBC
[2022:03:13:21:41:17]: Error: Couldn't expose some/all drm planes for card: /dev/dri/card0
[2022:03:13:21:41:17]: Info: Detecting connected monitors
[2022:03:13:21:41:17]: Info: //////////////////////////////////////////////////////////////////
[2022:03:13:21:41:17]: Info: //                                                              //
[2022:03:13:21:41:17]: Info: //   Testing for available encoders, this may generate errors.  //
[2022:03:13:21:41:17]: Info: //   You can safely ignore those errors.                        //
[2022:03:13:21:41:17]: Info: //                                                              //
[2022:03:13:21:41:17]: Info: //////////////////////////////////////////////////////////////////
[2022:03:13:21:41:17]: Info: Trying encoder [nvenc]
[2022:03:13:21:41:17]: Info: Screencasting with X11
[2022:03:13:21:41:18]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:18]: Info: Color range: [JPEG]
[2022:03:13:21:41:18]: Error: Could not open codec [h264_nvenc]: Function not implemented
[2022:03:13:21:41:19]: Info: Screencasting with X11
[2022:03:13:21:41:19]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:19]: Info: Color range: [JPEG]
[2022:03:13:21:41:19]: Error: Failed to create a CUDA device: Unknown error occurred
[2022:03:13:21:41:19]: Info: Screencasting with X11
[2022:03:13:21:41:19]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:19]: Info: Color range: [JPEG]
[2022:03:13:21:41:19]: Error: Failed to create a CUDA device: Unknown error occurred
[2022:03:13:21:41:19]: Info: Screencasting with X11
[2022:03:13:21:41:19]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:19]: Info: Color range: [JPEG]
[2022:03:13:21:41:19]: Error: Failed to create a CUDA device: Unknown error occurred
[2022:03:13:21:41:19]: Info: Encoder [nvenc] failed
[2022:03:13:21:41:19]: Info: Trying encoder [vaapi]
[2022:03:13:21:41:19]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
libva info: VA-API version 1.13.0
[2022:03:13:21:41:20]: Info: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)

[2022:03:13:21:41:20]: Error: Couldn't initialize va display: unknown libva error
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
libva info: VA-API version 1.13.0
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Info: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)

[2022:03:13:21:41:20]: Error: Couldn't initialize va display: unknown libva error
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
libva info: VA-API version 1.13.0
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Info: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)

[2022:03:13:21:41:20]: Error: Couldn't initialize va display: unknown libva error
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
libva info: VA-API version 1.13.0
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Info: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)

[2022:03:13:21:41:20]: Error: Couldn't initialize va display: unknown libva error
[2022:03:13:21:41:20]: Info: Encoder [vaapi] failed
[2022:03:13:21:41:20]: Info: Trying encoder [software]
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 709]
x264 [error]: high profile doesn't support a bit depth of 10
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Error: Could not open codec [libx264]: Invalid argument
[2022:03:13:21:41:20]: Info: Screencasting with X11
[2022:03:13:21:41:20]: Info: Color coding [Rec. 601]
[2022:03:13:21:41:20]: Info: Color range: [JPEG]
[2022:03:13:21:41:20]: Warning: software: h264: replacing nalu prefix data
[2022:03:13:21:41:20]: Info: 
[2022:03:13:21:41:20]: Info: //////////////////////////////////////////////////////////////
[2022:03:13:21:41:20]: Info: //                                                          //
[2022:03:13:21:41:20]: Info: // Ignore any errors mentioned above, they are not relevant //
[2022:03:13:21:41:20]: Info: //                                                          //
[2022:03:13:21:41:20]: Info: //////////////////////////////////////////////////////////////
[2022:03:13:21:41:20]: Info: 
[2022:03:13:21:41:20]: Info: Found encoder software: [libx264]
[2022:03:13:21:41:20]: Info: Open the Web UI to set your new username and password and getting started
[2022:03:13:21:41:20]: Info: File /root/.config/sunshine/sunshine_state.json doesn't exist
sendto: Network is unreachable
sendto: Network is unreachable
sendto: Network is unreachable
sendto: Network is unreachable
[2022:03:13:21:41:20]: Info: Adding avahi service Sunshine
[2022:03:13:21:41:20]: Info: Configuration UI available at [https://localhost:47990]
[2022:03:13:21:41:21]: Info: Avahi service Sunshine successfully established.
[2022:03:13:21:41:22]: Error: Couldn't discover any UPNP devices

Expected Behavior

Sunshine should use nvenc since nvidia hardware is present and drivers/cuda are installed.

Additional Context

No response

Sunshine Host Operating System and Version

Fedora 35

Architecture

x86-64

Sunshine Version

0.13.0 (github binary) / git master branch (custom built)

GPU Type

Nvidia

GPU Model

Nvidia Geforce GTX960

GPU Driver/Mesa Version

Nvidia Driver 510.47.03

Capture Method (Linux Only)

Nvidia Driver 510.47.03

hradec commented 2 years ago

Is there any tips from anyone else running fedora35 that I should know? maybe I need to install ffmpeg from another repository? or do I need to install nvenc sdk from nvidia as well?

hradec commented 2 years ago

regarding the message This hardware does not support NvFBC, should I run the patch from https://github.com/keylase/nvidia-patch to enable it?

hradec commented 2 years ago

I've installed ffmpeg from rpmfusion dnf repositories, which seems to be version 4.4.1. (I'm not sure if this ffmpeg has nvenc support.)

I've ran ffmpeg -codecs | grep nvenc to double check if ffmpeg does support nvenc, and it seems it does indeed:

ffmpeg version 4.4.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --enable-chromaprint --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libilbc --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librav1e --enable-librtmp --enable-librubberband --enable-libsmbclient --enable-version3 --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libglslang --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-avfilter --enable-avresample --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m h264_qsv h264_cuvid ) (encoders: libx264 libx264rgb h264_nvenc h264_qsv h264_v4l2m2m h264_vaapi nvenc nvenc_h264 )
 DEV.L. hevc                 H.265 / HEVC (High Efficiency Video Coding) (decoders: hevc hevc_qsv hevc_v4l2m2m hevc_cuvid ) (encoders: libx265 nvenc_hevc hevc_nvenc hevc_qsv hevc_v4l2m2m hevc_vaapi )

It does list h264_nvenc, nvenc, nvenc_h264, nvenc_hevc and hevc_nvenc as available. So I guess the problem is not ffmpeg?

I've also ran a desktop capture using ffmpeg and nvenc like this:

ffmpeg -loglevel debug \
        -f x11grab -s 1920x1080 -framerate 60 -i :1.0 \
        -c:v h264_nvenc -preset:v llhq  \
        -rc:v vbr_minqp -qmin:v 19  \
        -f mpegts - | nc -l -p 9000

and was able to confirm it indeed does supports nvenc: image

So, I'm not sure why sunshine is failling to use nvenc and reverting to libx264, since the current installed ffmpeg does support nvenc, and nvenc is indeed properly installed since ffmpeg alone is able to use it.

Any insights would be greatly appreciated!

hradec commented 2 years ago

regarding the message This hardware does not support NvFBC, should I run the patch from https://github.com/keylase/nvidia-patch to enable it?

I went ahead and gave it a try, and seems running sudo patch.sh followed by sudo path-fbc.sh did the trick indeed! After running then, this is the output I got from sunshine: 

[2022:03:14:01:02:45]: Info: Found [1] outputs
[2022:03:14:01:02:45]: Info: Virtual Desktop: 1920x1080
[2022:03:14:01:02:45]: Info: XrandR: available
[2022:03:14:01:02:45]: Info: -- Output --
[2022:03:14:01:02:45]: Info:   Resolution: 1920x1080
[2022:03:14:01:02:45]: Info:   Offset: 0x0
[2022:03:14:01:02:45]: Error: Couldn't destroy session handle: Unable to cleanup NvFBC
[2022:03:14:01:02:45]: Error: Failed to gain CAP_SYS_ADMIN
[2022:03:14:01:02:45]: Error: Couldn't expose some/all drm planes for card: /dev/dri/card0
[2022:03:14:01:02:45]: Info: Detecting connected monitors
[2022:03:14:01:02:46]: Error: Could not create Sunshine Mouse: Permission denied
Aborted (core dumped)
[rhradec@ws-003 Sunshine]$ sudo LD_PRELOAD=/lib64/libstdc++.so.6  build/sunshine
[2022:03:14:01:02:50]: Info: Found [1] outputs
[2022:03:14:01:02:50]: Info: Virtual Desktop: 1920x1080
[2022:03:14:01:02:50]: Info: XrandR: available
[2022:03:14:01:02:50]: Info: -- Output --
[2022:03:14:01:02:50]: Info:   Resolution: 1920x1080
[2022:03:14:01:02:50]: Info:   Offset: 0x0
[2022:03:14:01:02:50]: Error: Couldn't destroy session handle: Unable to cleanup NvFBC
[2022:03:14:01:02:50]: Error: Couldn't expose some/all drm planes for card: /dev/dri/card0
[2022:03:14:01:02:50]: Info: Detecting connected monitors
[2022:03:14:01:02:51]: Info: //////////////////////////////////////////////////////////////////
[2022:03:14:01:02:51]: Info: //                                                              //
[2022:03:14:01:02:51]: Info: //   Testing for available encoders, this may generate errors.  //
[2022:03:14:01:02:51]: Info: //   You can safely ignore those errors.                        //
[2022:03:14:01:02:51]: Info: //                                                              //
[2022:03:14:01:02:51]: Info: //////////////////////////////////////////////////////////////////
[2022:03:14:01:02:51]: Info: Trying encoder [nvenc]
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Could not open codec [h264_nvenc]: Function not implemented
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Could not open codec [hevc_nvenc]: Function not implemented
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 709]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Could not open codec [h264_nvenc]: Function not implemented
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:51]: Info: Color coding [Rec. 709]
[2022:03:14:01:02:51]: Info: Color range: [JPEG]
[2022:03:14:01:02:51]: Error: Could not open codec [hevc_nvenc]: Function not implemented
[2022:03:14:01:02:51]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:51]: Info: Screencasting with NvFBC
[2022:03:14:01:02:52]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:52]: Info: Color range: [JPEG]
[2022:03:14:01:02:52]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:52]: Info: Screencasting with NvFBC
[2022:03:14:01:02:52]: Info: Color coding [Rec. 601]
[2022:03:14:01:02:52]: Info: Color range: [JPEG]
[2022:03:14:01:02:52]: Warning: nvenc: hevc: replacing nalu prefix data
[2022:03:14:01:02:52]: Error: Couldn't destroy session handle: The context is not bound
[2022:03:14:01:02:52]: Info: 
[2022:03:14:01:02:52]: Info: //////////////////////////////////////////////////////////////
[2022:03:14:01:02:52]: Info: //                                                          //
[2022:03:14:01:02:52]: Info: // Ignore any errors mentioned above, they are not relevant //
[2022:03:14:01:02:52]: Info: //                                                          //
[2022:03:14:01:02:52]: Info: //////////////////////////////////////////////////////////////
[2022:03:14:01:02:52]: Info: 
[2022:03:14:01:02:52]: Info: Found encoder nvenc: [h264_nvenc, hevc_nvenc]
sendto: Network is unreachable
sendto: Network is unreachable
sendto: Network is unreachable
sendto: Network is unreachable
[2022:03:14:01:02:52]: Info: Configuration UI available at [https://localhost:47990]
[2022:03:14:01:02:52]: Info: Adding avahi service Sunshine
[2022:03:14:01:02:53]: Info: Avahi service Sunshine successfully established.
[2022:03:14:01:02:54]: Error: Couldn't discover any UPNP devices

So the Found encoder nvenc: [h264_nvenc, hevc_nvenc] indeed confirms it could find nvenc now, despite the Error: Could not open codec [h264_nvenc]: Function not implemented and Error: Could not open codec [hevc_nvenc]: Function not implemented messages.

I guess nvenc can only be used when screencasting with NvFBC?!

Shouldn't nvenc be usable with X11 screenscast as well, the same way ffmpeg does it?

Anyhow, if someone else runs into problem to use nvenc with sunshine with an nvidia gpu in fedora 35, running patch.sh and patch-fbc.sh from https://github.com/keylase/nvidia-patch fixed the issue for me!

ReenigneArcher commented 2 years ago

@hradec thanks for posting your solution!

KuleRucket commented 2 years ago

"[2022:03:14:01:02:51]: Error: Could not open codec [h264_nvenc]: Function not implemented" - I believe this message is misleading. When I turn on verbose logging, the debug messages show: [h264_nvenc @ 0x55b4dc6eed40] 10 bit encode not supported [h264_nvenc @ 0x55b4dc6eed40] Provided device doesn't support required NVENC features [h264_nvenc @ 0x55b4dc6eed40] Nvenc unloaded [2021:12:22:10:11:41]: Error: Could not open codec [h264_nvenc]: Function not implemented

So as I understand from that, h264 is still supported just not with HDR encoding.

hradec commented 2 years ago

When I turn on verbose logging, the debug messages show: [h264_nvenc @ 0x55b4dc6eed40] 10 bit encode not supported

how do you turn verbose on?

ReenigneArcher commented 2 years ago

When I turn on verbose logging, the debug messages show: [h264_nvenc @ 0x55b4dc6eed40] 10 bit encode not supported

how do you turn verbose on?

add this to your sunshine.conf

min_log_level = verbose

hradec commented 2 years ago

@ReenigneArcher thanks!!

KuleRucket commented 2 years ago

Is there anything left to fix here? The last log looks pretty normal to me so maybe this can be closed?

ReenigneArcher commented 2 years ago

Is there anything left to fix here? The last log looks pretty normal to me so maybe this can be closed?

Closing now. I can re-open if needed.