LibVNC / libvncserver

LibVNCServer/LibVNCClient are cross-platform C libraries that allow you to easily implement VNC server or client functionality in your program.
GNU General Public License v2.0
1.08k stars 483 forks source link

Make sure cursorMutex is properly locked before we try to access cursor member #533

Closed sobomax closed 1 year ago

sobomax commented 1 year ago

This fixes random crashes in the cursor updating routines with VirtualBox 6.1.x running Ubuntu 22.04 guest when remote cursor is enabled (TightVNC client).

The crash is as follows:


11/08/2022 11:58:59 Client Protocol Version 3.8
11/08/2022 11:58:59 Protocol version sent 3.8, using 3.8
11/08/2022 11:58:59 rfbProcessClientSecurityType: executing handler for type 2
11/08/2022 11:59:01 Enabling NewFBSize protocol extension for client 192.168.23.190
11/08/2022 11:59:01 Enabling LastRect protocol extension for client 192.168.23.190
11/08/2022 11:59:01 Enabling cursor position updates for client 192.168.23.190
11/08/2022 11:59:01 Enabling full-color cursor updates for client 192.168.23.190
11/08/2022 11:59:01 Using compression level 1 for client 192.168.23.190
11/08/2022 11:59:01 Using tight encoding for client 192.168.23.190
11/08/2022 11:59:01 Pixel format for client 192.168.23.190:
11/08/2022 11:59:01   32 bpp, depth 24, little endian
11/08/2022 11:59:01   true colour: max r 255 g 255 b 255, shift r 16 g 8 b 0
11/08/2022 11:59:04 Sending rfbEncodingNewFBSize for resize to (800x600)
11/08/2022 11:59:04 Sending rfbEncodingNewFBSize for resize to (1920x1080)

Thread 34 received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
[Switching to LWP 105544 of process 39230]
0x0000000803325fb4 in rfbSendCursorShape (cl=0x803c2eb00) at /var/tmp/usr/ports/net/libvncserver/work/libvncserver-LibVNCServer-0.9.13/libvncserver/cursor.c:161
161                 bitmapByte = bitmapData[i * bitmapRowBytes + j];
(gdb) print i
$1 = 0
(gdb) print bitmapRowBytes
$2 = 4
(gdb) print j
$3 = 0
(gdb) print bitmapData
$4 = (uint8_t *) 0x0
(gdb) print pCursor
$5 = (rfbCursorPtr) 0x85ba37a30
(gdb) print *pCursor
$6 = {cleanup = 0 '\000', cleanupSource = 0 '\000', cleanupMask = 0 '\000', cleanupRichSource = 0 '\000', source = 0x0, mask = 0x0, width = 32, height = 32, xhot = 3, yhot = 3, foreRed = 0, foreGreen = 0,
  foreBlue = 0, backRed = 0, backGreen = 0, backBlue = 0, richSource = 0x0, alphaSource = 0x0, alphaPreMultiplied = 0 '\000'}
bk138 commented 1 year ago

Thanks for the contribution!