Sunshine with VAAPI, is using the wrong encoding format with HEVC Main10

vulturm commented 1 year ago

Is there an existing issue for this?

[x] I have searched the existing issues

Is your issue described in the documentation?

[X] I have read the documentation

Is your issue present in the nightly release?

[X] This issue is present in the nightly release

Describe the Bug

Hello,

The title is the response the I've received after I've initially reported this issue to the official mesa bug tracker.

I've been successfully using Sunshine 0.21 on my desktop in combination with Moonlight on a raspberry pi4 with HEVC and it works great!

However, when trying to force sunshine to use HEVC Main10 HDR, I get the Unsupported format: yuv420p10le. error, even though vainfo command on linux reports HEVC Main10 profile as supported:

vainfo: VA-API version: 1.18 (libva 2.18.2)
vainfo: Driver version: Mesa Gallium driver 24.0.0-devel for AMD Radeon RX 6700 XT (radeonsi, navi22, LLVM 16.0.6, DRM 3.54, 6.5.8-200.fc38.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

The response I got from the issue reported to mesa is that "Only P010 format is supported for 10bit encoding, but Sunshine tries to use YUV420P10 without checking supported formats. So this is Sunshine issue, nothing wrong on Mesa side."

And,

"Just changing AV_PIX_FMT_YUV420P10 to AV_PIX_FMT_P010 in this file should be enough to make it work."

I am reporting this as a bug and not as PR, since I am not too familiar with the code or if it also has other implications changing the format. Or, if a detection logic would be better to be implemented to query for supported formats first and try to use one that drivers reports as supported.

It would be nice if such a small change could make HDR streaming work on linux.

Thank you in advance!

Expected Behavior

No error and HDR works.

Additional Context

Initial mesa issue report.

Host Operating System

Linux

Operating System Version

Fedora 38

Architecture

64 bit

Sunshine commit or version

0.21

Package

Linux - rpm

GPU Type

AMD

GPU Model

GPU: Navi 22 (Radeon RX 6700 XT)

GPU Driver/Mesa Version

Mesa 24.0.0-0.3.20231103.15.28b79b2 (git)

Capture Method (Linux Only)

KMS

Config

output_name = 0
hevc_mode = 3
encoder = vaapi
min_threads = 8
av1_mode = 1
min_log_level = 2
resolutions = [
    352x240,
    480x360,
    858x480,
    1280x720,
    1920x1080,
    2560x1080,
    3440x1440,
    1920x1200,
    3860x2160,
    3840x1600
]
capture = kms
fps = [10,30,60]

Apps

No response

Relevant log output

[2023:11:04:00:24:26]: Warning: [hevc_vaapi @ 0x7feb9407d940] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2023:11:04:00:24:26]: Info: Screencasting with KMS
[2023:11:04:00:24:26]: Info: Found monitor for DRM screencasting
libva info: VA-API version 1.18.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_18
libva info: va_openDriver() returns 0
[2023:11:04:00:24:26]: Info: SDR color coding [Rec. 709]
[2023:11:04:00:24:26]: Info: Color depth: 10-bit
[2023:11:04:00:24:26]: Info: Color range: [JPEG]
libva info: VA-API version 1.18.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_18
libva info: va_openDriver() returns 0
[2023:11:04:00:24:26]: Error: [AVHWFramesContext @ 0x7feb943764c0] Unsupported format: yuv420p10le.
[2023:11:04:00:24:26]: Warning: Encoder [vaapi] does not support HEVC Main10 on this system

robinAwallace commented 1 year ago

Im also encountering this issue with the same GPU on ubuntu 22.04 :disappointed:

njdom24 commented 11 months ago

Also seeing this on my 5700 XT with wlroots capture method

JFingerle commented 11 months ago

Im also getting this error with a 5700 XT.

nettoneko commented 11 months ago

Can reproduce with 6700 XT

LizardByte / Sunshine