[amdgpu] Segmentation fault with --va-derive=on

[ctrlcctrlv]

[fred@大.狸.agency ~/.cache/yay/kmsvnc-git/src/kmsvnc]$ sudo gdb ./build/kmsvnc -ex 'run -p 5901  -b 10.2.2.2 -4 -d /dev/dri/card0  --va-derive=on '
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/kmsvnc...
Starting program: /home/fred/.cache/yay/kmsvnc-git/src/kmsvnc/build/kmsvnc -p 5901  -b 10.2.2.2 -4 -d /dev/dri/card0  --va-derive=on
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
No keyboard layout set from environment variables, use US layout by default
See https://xkbcommon.org/doc/current/structxkb__rule__names.html
drm driver is amdgpu
Plane 40 CRTC 0 FB 0 Type primary
Plane 46 CRTC 0 FB 0 Type primary
Plane 52 CRTC 0 FB 0 Type primary
Plane 58 CRTC 0 FB 0 Type primary
Plane 64 CRTC 95 FB 154 Type primary
Using plane 64 to locate framebuffers
Template framebuffer is 154: 2944x1080 fourcc:875713112 mod:144115188622392067 flags:2
handles 1 1 1 0
offsets 0 13565952 13639680 0
pitches 11776 3072 3072 0
format XR24, modifier AMD:GFX10_RBPLUS,GFX9_64K_R_X,DCC,DCC_RETILE,DCC_INDEPENDENT_64B,DCC_INDEPENDENT_128B,DCC_MAX_COMPRESSED_BLOCK=64B,DCC_CONSTANT_ENCODE,PIPE_XOR_BITS=4,PACKERS=4
[New Thread 0x7fffeb3ff6c0 (LWP 295719)]
[New Thread 0x7fffeabfe6c0 (LWP 295720)]
[New Thread 0x7fffea3fd6c0 (LWP 295721)]
[New Thread 0x7fffe9bfc6c0 (LWP 295722)]
vaapi vendor Mesa Gallium driver 23.3.0-devel for AMD Radeon RX 6900 XT (navi21, LLVM 16.0.6, DRM 3.54, 6.3.7-1-amd-staging-drm-next-git-gb4d88db12014)

Thread 1 "kmsvnc" received signal SIGSEGV, Segmentation fault.
0x000055555555c974 in va_init () at /home/fred/.cache/yay/kmsvnc-git/src/kmsvnc/va.c:297
297                if (va->image->format.fourcc == format_to_try[i].fmt->fourcc) {
(gdb) p format_to_try[i]
$1 = {va_fourcc = 1111970392, fmt = 0x0, is_alpha = 0 '\000', va_rt_format = 131072}
(gdb) p i
$2 = 2
(gdb) p format_to_try
$3 = {{va_fourcc = 1480738642, fmt = 0x55555577fad0, is_alpha = 0 '\000', va_rt_format = 131072}, {va_fourcc = 1094862674, fmt = 0x55555577fa40, is_alpha = 1 '\001', va_rt_format = 131072}, {va_fourcc = 1111970392, fmt = 0x0, 
    is_alpha = 0 '\000', va_rt_format = 131072}, {va_fourcc = 1111970369, fmt = 0x55555577fa70, is_alpha = 1 '\001', va_rt_format = 131072}, {va_fourcc = 1481787202, fmt = 0x55555577faa0, is_alpha = 0 '\000', va_rt_format = 131072}, {
    va_fourcc = 1095911234, fmt = 0x55555577fa10, is_alpha = 1 '\001', va_rt_format = 131072}, {va_fourcc = 1380401752, fmt = 0x0, is_alpha = 0 '\000', va_rt_format = 131072}, {va_fourcc = 1380401729, fmt = 0x0, is_alpha = 1 '\001', 
    va_rt_format = 131072}, {va_fourcc = 808669784, fmt = 0x0, is_alpha = 0 '\000', va_rt_format = 2097152}, {va_fourcc = 808669761, fmt = 0x0, is_alpha = 1 '\001', va_rt_format = 2097152}, {va_fourcc = 808665688, fmt = 0x0, 
    is_alpha = 0 '\000', va_rt_format = 2097152}, {va_fourcc = 808665665, fmt = 0x0, is_alpha = 1 '\001', va_rt_format = 2097152}}
(gdb)

[isjerryxiao]

Some vaapi drivers have faulty vaDeriveImage implementation. In this case i suspect dereferencing va->image->format.fourcc produced the segfault (the actual vaDeriveImage call returned VA_STATUS_SUCCESS). Anyway, please don't use va derive.

[ctrlcctrlv]

i now get a black screen with X for cursor but no crash