wayvnc crashes due to SIGILL while calling nvnc_open(...)

[primeos]

wayvnc version: 0.1.0 neatvnc version: 0.1.0 GCC version: 9.2.0

When I try to run wayvnc on a test VM it crashes after receiving SIGILL (illegal instruction). On my old desktop PC (AMD Radeon HD 6950) wayvnc crashes as well. Any ideas what could be going wrong or what I could be missing / why this seems to happen only on my setup?

On my test VM I get the following gdb output using virtio with 3D acceleration:

(gdb) run
Starting program: /nix/store/0srxrcwx5k9x5amdcq0nf1gsn1xynh9k-wayvnc-0.1.0/bin/wayvnc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/1ncwrl8bplq3xhmj8pxfkx4y0i90vmnx-glibc-2.30/lib/libthread_db.so.1".
libEGL warning: DRI2: failed to create dri screen
DEBUG: EGL: eglInitialize: DRI2: failed to create screen
DEBUG: OpenGL ES 3.1 Mesa 19.3.3

Program received signal SIGILL, Illegal instruction.
nvnc_open (address=address@entry=0x409013 "127.0.0.1", port=port@entry=5900) at ../src/server.c:768
768     ../src/server.c: No such file or directory.
(gdb) bt
#0  nvnc_open (address=address@entry=0x409013 "127.0.0.1", port=port@entry=5900) at ../src/server.c:768
#1  0x00000000004059b5 in init_nvnc (self=self@entry=0x7fffffffbcd0, addr=addr@entry=0x409013 "127.0.0.1",
    port=port@entry=5900) at ../src/main.c:422
#2  0x0000000000404e29 in main (argc=<optimized out>, argv=<optimized out>) at ../src/main.c:788

Or when I switch to QXL:

(gdb) run
Starting program: /nix/store/0srxrcwx5k9x5amdcq0nf1gsn1xynh9k-wayvnc-0.1.0/bin/wayvnc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/1ncwrl8bplq3xhmj8pxfkx4y0i90vmnx-glibc-2.30/lib/libthread_db.so.1".
libEGL warning: No hardware driver found, falling back to software rendering
libEGL warning: pci id for fd 4: 1b36:0100, driver (null)

DEBUG: OpenGL ES 3.1 Mesa 19.3.3

Program received signal SIGILL, Illegal instruction.
nvnc_open (address=address@entry=0x409013 "127.0.0.1", port=port@entry=5900) at ../src/server.c:768
768     ../src/server.c: No such file or directory.
(gdb) bt
#0  nvnc_open (address=address@entry=0x409013 "127.0.0.1", port=port@entry=5900) at ../src/server.c:768
#1  0x00000000004059b5 in init_nvnc (self=self@entry=0x7fffffffbcc0, addr=addr@entry=0x409013 "127.0.0.1",
    port=port@entry=5900) at ../src/main.c:422
#2  0x0000000000404e29 in main (argc=<optimized out>, argv=<optimized out>) at ../src/main.c:788

Sway and other wlroots based compositors run fine with both drivers (but SIGILL should be unrelated to the GPU / libEGL warnings anyway).

[primeos]

I missed https://github.com/any1/neatvnc/blob/v0.1.0/meson.build#L25, that's probably the reason as AVX isn't available in my VM.

[primeos]

Indeed, it does work without -mavx. Does using the AVX extension result in significant performance gains or are they only minor?

Anyway, I'll close this for now.

[any1]

You can try enabling whichever SIMD architecture is supported (e.g. sse4, sse3, sse2) and see if you notice any difference in CPU load when running some animation, e.g. weston-simple-egl or glxgears.