Segmentation fault when preferred encoding set as ZRLE

[sdikshit786]

Hello, I have faced one issue when I connects for the first time to my vncserver with ZRLE set as preferred encoding at the client end, the vncserver crashes with SIGSEGV.

Steps: Started VNC server VNCServer started listening on port 5900 Start the vncviewer(TigerVNC) Under options --> Preferred encoding untick the Auto Select option and select ZRLE Save and connect After sometime the vncserver crashes

Expected Behavior It should not crash.

Logs/Backtraces registers: x0-x9 : 0000ffff74000ed0 000000002cb22c58 0000000000000100 0000000000000000 0000ffff68000f80 0000000000000740 0000000000000040 0000000000000040 0000000000000000 0000000000001e00, x10-x19 : 00000000000079e4 0000000000000000 0000ffff7f8e01c0 0000000000000040 0000ffff74000fd0 0000ffff740010d0 0000ffff68000fc0 0000000000001d00 000000306c55ea70 0000ffff70000b20, x20-x29 : 0000ffff68004f90 0000ffff68000f80 0000000000000040 0000000000000740 0000000000000040 0000000000000000 0000ffff70008b20 0000000000000780 0000000000000040 0000ffff7cdcedd0 x30 = 0xffff7f8f2ec0, sp = 0xffff7cdcedd0, pc = 0xffff7f8e0200, pstate = 0x20000000, orig_x0 = 0xffff6803b000, syscallno = 0xffffffff fault address: (null) callstack: [0xffff7f8e0200] (unresolved) rfbSendRectEncodingRRE+0xfc0 /usr/lib64/libvncserver.so.1.0.0 [0xffff7f8f2ec0] (unresolved) rfbSendRectEncodingZRLE+0x320 /usr/lib64/libvncserver.so.1.0.0 [0xffff7f8d45a4] (unresolved) rfbSendFramebufferUpdate+0x834 /usr/lib64/libvncserver.so.1.0.0 [0xffff7f8cd49c] (unresolved) rfbDefaultPtrAddEvent+0x1fc /usr/lib64/libvncserver.so.1.0.0 [0x306c576f78] start_thread (pthread_create.c:463) /lib64/libpthread-2.27.so

(rfbDefaultPtrAddEvent+0x1fc)[0xffff7f8cd49c]
resolves to 0xc2a0 + 0x1fc = 0xc49c : libvncserver/main.c:498 clientOutput thread calls rfbSendFramebufferUpdate

(rfbSendFramebufferUpdate+0x834)[0xffff7f8d45a4] resolves to 0x12d70 + 0x834 = 0x135a4 : libvncserver/rfbserver.c:3129 rfbSendFramebufferUpdate() --> rfbSendRectEncodingZRLE

(rfbSendRectEncodingZRLE+0x320)[0xffff7f8f2ec0] resolves to 0x31b70 + 0x320 = 0x31e90 : libvncserver/zrleencodetemplate.c:112

--> ZRLE_ENCODE-->GET_IMAGE_INTO_BUF--> (cl->translateFn)(cl->translateLookupTable, &cl->screen->serverFormat,\ &cl->format, fbptr, (char)buf, \ cl->scaledScreen->paddedWidthInBytes, tw, th); }
(*cl->translateFn) is calling rfbTranslateWithRGBTables32to8

(rfbSendRectEncodingRRE+0xfc0)[0xffff7f8e0200] resolves to 0x1e240 + 0xfc0 = 1f200 : libvncserver/tabletranstemplate.c:104 rfbTranslateWithRGBTables32to8 --> static void rfbTranslateWithRGBTablesINtoOUT

I am using vnc_0.9.10 Please help me to find out what went wrong

[bk138]

Please post the LibVNCServer version you are using and if it's an old one (do you mean LibVNCServer 0.9.10 ?) then upgrade to the latest release and try again.

[bk138]

Feel free to post more info once you got it. For the time being, closing this.