RPI5: Vulkan renderer + DRM can lead to segfault

[svnhub]

Under very specific circumstances Moonlight will consistently segfault on startup because it feeds incorrect data to a subsystem.

The following environment is required:

• A DRM/PRIME or similar hardware accelerated decoder WITH support for 10-bit decoding, for example upstream FFMPEG 7.0.2
• Vulkan-renderer based display WITHOUT support for 10-bit presentation but with support for receiving decoded frame references directly from hardware via DMABUFF or mmap

An example of such environment would be an RPI5 with Wayland in 8-bit mode and a full modern Vulkan+Mesa setup.

Steps to reproduce

1) Start moonlight under Wayland 2) SDL correctly detects that the Vulkan display driver does not support 10bit 3) SDL selects the Vulkan display driver in 8bit mode 4) Moonlight proceeds to test 10bit HEVC via FFMPEG that uses V4L2 stateless hardware decoding to deliver hwfmt=rpi4_10 frames via DMABUFF directly into PlVlk+Wayland. 5) Program segfaults

Moonlight Logs

Attached to this issue are two log files. The first one shows the faulty startup situation and a stacktrace from the segfault. fails-with-10bit.txt The line in session.cpp eventually leading to the the segfault is here: https://github.com/moonlight-stream/moonlight-qt/blob/67e89d9e161dba70fee807a499ce90736793db12/app/streaming/session.cpp#L382

The second log file shows a successful execution of a modified moonlight client where 10bit testing is skipped if the display does not support 10bit. It completes startup and works as expected: fixed-with-hack.txt

Other notes

This was tested with Moonlight 6.0.1 and current git as well as qt5/qt6, FFMPEG 6 and 7 and way too many SDL and Mesa versions.

[cgutman]

It is not invalid to play 10-bit content on an 8-bit display. Libplacebo contains tone mapping code to convert from HDR to SDR. In the case that the Vulkan driver doesn't support the required formats, libplacebo should return a failure which will cause Moonlight to try the next renderer.

In the case of your hack, it may just be working because it's avoiding Vulkan and libplacebo entirely. Moonlight only constructs a libplacebo renderer for HDR content by default, so skipping 10-bit content will skip Vulkan completely. You can test to see if it's really 10-bit or Vulkan/libplacebo by running with the environment variable PREFER_VULKAN=1 set.

Please provide:

• Name and version of the distro you're using
• Version of libplacebo that you're using
• If possible, please compile libplacebo with debug symbols by passing -Dbuildtype=debug to your meson setup command, so we can see line numbers and variables in the stack trace

[svnhub]

You are quite right, it does indeed seem to use a different (EGL?) renderer in 8-bit mode and PREFER_VULKAN=1 crashes it even when skipping the 10-bit tests.

The following stacktrace is from a Gentoo Linux (based on the linux/arm64/23.0/desktop profile) with:

• libplacebo directly from git (today)
• moonlight 6.0.1 (without the hack)
• ffmpeg 7.0.2
• SDL2 2.30.7
• qtbase 6.7.2-r4

segfault_wfull-frames.txt

I think there is some compiler optimizations unrolling things (otherwise I have no idea how that line could produce a segfault), so I will recompile with slightly less aggressive optimization options and see if it gets more clear.