x11vnc -connect … -threads crashes in Fedora ≥ 40

[ppisar]

"/src/x11vnc -display :0 -connect localhost:8000 -threads" crashes during establishing a session in various ways. E.g. like this:

25/06/2024 16:03:01 Making connection to client on host localhost port 8000
25/06/2024 16:03:01   0 other clients
25/06/2024 16:03:01 Normal socket connection
25/06/2024 16:03:01 Disabled X server key autorepeat.
25/06/2024 16:03:01   to force back on run: 'xset r on' (3 times)
25/06/2024 16:03:01 incr accepted_client=1 for ::1:8000  sock=12
25/06/2024 16:03:01 reverse_connect: localhost:8000/::1 OK
25/06/2024 16:03:01 reverse_connect: turning on auth for ::1
25/06/2024 16:03:01 client progressed=0 in 15/10 0.000006 s
25/06/2024 16:03:01 Client Protocol Version 3.8
25/06/2024 16:03:01 Protocol version sent 3.8, using 3.8
25/06/2024 16:03:01 Send channel security type 'none'
25/06/2024 16:03:01 rfbProcessClientSecurityType: executing handler for type 1
25/06/2024 16:03:01 rfbProcessClientSecurityType: returning securityResult for client rfb version >= 3.8
25/06/2024 16:03:01 rfbProcessClientNormalMessage: unknown message type 87
25/06/2024 16:03:01  ... closing connection
25/06/2024 16:03:01 client_count: 0
25/06/2024 16:03:01 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEC6)
25/06/2024 16:03:01 rfbProcessClientNormalMessage: read: Bad file descriptor
25/06/2024 16:03:01 Restored X server key autorepeat to: 1
25/06/2024 16:03:01 viewer exited.
25/06/2024 16:03:01 client_count: 0
25/06/2024 16:03:01 viewer exited.
double free or corruption (!prev)
extra[1] signal: 6
Aborted (core dumped)

This is a built from the latest commit a61f364e385d98677c04070a994397b287385219, configured with:

./configure --build=x86_64-redhat-linux --host=x86_64-redhat-linux --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --runstatedir=/run --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-avahi --with-colormultipointer --with-crypto --with-dpms --with-fbdev --without-fbpm --without-macosx-native --with-ssl --with-uinput --with-x --without-xcomposite --with-xdamage --with-xfixes --with-xinerama --with-xkeyboard --with-xrandr --with-xrecord --without-xtrap 'CFLAGS=-O0 -g'

The crash is probably triggered by a compiler or some of the build-time dependencies. I was unable to pin point what's the exact trigger. I only can tell that it does not happen in Fedora 39, but happens since Fedora 40 (GCC 14.1.1, glibc 2.39, libX11 1.8.9). It's x86_64 platform.

It does not happen without -threads option. It looks like a newer compiler is more aggressive in optimizations and the threaded implementation does not properly accesses to shared memory ("unknown message type 87", "read: Bad file descriptor", "double free or corruption").

The "double free or corruption" message and SIGABRT come from glibc memory allocator. Valgrind did not show anything relevant, gdb crashed for a different reason).

By the way, an official Fedora build log https://kojipkgs.fedoraproject.org//packages/x11vnc/0.9.16%5E20230801gita61f364/2.fc40/data/logs/x86_64/build.log which uses more hardening options than CFLAGS='-O0 -g', shows many warnings.

The original bug report from a Fedora user https://bugzilla.redhat.com/show_bug.cgi?id=2294197.