Wayland/Vulkan: WaylandBackend.cpp:1568 Assertion `u8bitFormat != DRM_FORMAT_INVALID`

[tskaar]

Is there an existing issue for this?

• [X] I have searched the existing issues

Are you using any gamescope patches or a forked version of gamescope?

• [X] The issue occurs on upstream gamescope without any modifications

Current Behavior

Launching gamescope on a Wayland compositor that's running with a Vulkan renderer crashes Gamescope and the following assert triggers:

gamescope: ../src/Backends/WaylandBackend.cpp:1568: virtual void gamescope::CWaylandBackend::GetPreferredOutputFormat(uint32_t*, uint32_t*) const: Assertion `u8BitFormat != DRM_FORMAT_INVALID' failed.

Doing a git bisect shows that 01101093445af20ea25ad386fd872d5f92266c2c is the bad commit.

Workarounds:

1. Using OpenGL as backend for the Wayland compositor
2. Using the SDL backend, requires #1548

Compositor(s):

• Jay with Vulkan backend
• River with Vulkan backend
• River with OpenGL backend (works)

Steps To Reproduce

1. Launch River/Sway with WLR_RENDERER=vulkan
2. Open terminal
3. Launch gamescope

Hardware information

- Distro: Arch Linux
- CPU: AMD 7735HS
- GPU0: Radeon 680M
- GPU1: Radeon RX 7600S
- Driver Version: mesa-git (24.3.0_devel.197058.e69aba2cded-1)

Software information

- Desktop environment: Jay + River
- Session type: Wayland
- Gamescope version: git upstream (3.15.9.r21.g7dd1bcd9-1)
- Gamescope launch command(s): n/a

Which gamescope backends have the issue you are reporting?

• [X] Wayland (default for nested gamescope)
• [ ] DRM (default for embedded gamescope, i.e. gamescope-session)
• [ ] SDL
• [ ] OpenVR

Logging, screenshots, or anything else

Bisect log:

git bisect start
# status: waiting for both good and bad commits
# bad: [7dd1bcd9102a17e039970ccd9a324a9fe8365d6d] rendervulkan: Fix a CVulkanCmdBuffer leak that could result in screenshot request failures
git bisect bad 7dd1bcd9102a17e039970ccd9a324a9fe8365d6d
# status: waiting for good commit(s), bad commit known
# good: [b2505fe4dac917aadd1ab05473d863830a7b5859] Fix a CVulkanCmdBuffer leak that could result in screenshot request failures
git bisect good b2505fe4dac917aadd1ab05473d863830a7b5859
# good: [8045786067f7a7fd9fd3fff98c29588f314218a9] layer: Add hacks for Talos Principle and Serious Sam's bad swapchain usage
git bisect good 8045786067f7a7fd9fd3fff98c29588f314218a9
# bad: [ca7bf9ff4a1987f358a27c173d46fad7a4645658] readme: Fix Meson deprecated command warning
git bisect bad ca7bf9ff4a1987f358a27c173d46fad7a4645658
# bad: [d87f6d9b9d98ea8064e4063c01d483895fef87f0] Add clipboard support from SDL backend
git bisect bad d87f6d9b9d98ea8064e4063c01d483895fef87f0
# good: [7a273743368aaa6368cb0d31d4786d78b4b00dc6] rendervulkan: Fix scanout-able check
git bisect good 7a273743368aaa6368cb0d31d4786d78b4b00dc6
# bad: [a2b45e15009f80ca1ad346840f2574bbe060fa94] Fix how reshade effects are cleared
git bisect bad a2b45e15009f80ca1ad346840f2574bbe060fa94
# bad: [01101093445af20ea25ad386fd872d5f92266c2c] rendervulkan: Attempt at fixing modifierless devices having issues
git bisect bad 01101093445af20ea25ad386fd872d5f92266c2c
# first bad commit: [01101093445af20ea25ad386fd872d5f92266c2c] rendervulkan: Attempt at fixing modifierless devices having issues