search

LibVNC / libvncserver

LibVNCServer/LibVNCClient are cross-platform C libraries that allow you to easily implement VNC server or client functionality in your program.
GNU General Public License v2.0
1.1k stars 482 forks source link

x11vnc getting segmentation fault when large text selected in the noVNC client using Edit option #484

Closed brahma1989 closed 2 years ago

brahma1989 commented 2 years ago

If you'd like to put out an incentive for fixing this bug, you can do so at https://issuehunt.io/r/LibVNC/libvncserver

Describe the bug when I select large text say 1500 lines using cursor my x11vnc server gets segmentation fault. x11vnc version: 0.9.13 libvncserver version: 0.9.13 here is the stack trace : (gdb) bt

0 0x76ea6b8c in rfbStatLookupMessage (cl=cl@entry=0x1fa5460, type=type@entry=3)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:196

1 0x76ea6d10 in rfbStatRecordMessageSent (cl=cl@entry=0x1fa5460, type=type@entry=3, byteCount=byteCount@entry=262152, byteIfRaw=byteIfRaw@entry=262152)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:256

2 0x76ea279c in rfbSendServerCutText (rfbScreen=,

str=0x222010 <primary_str> "\n[root@ci-9cb65498f0d8 log]# cat -n 3000 messages\ncat: 3000: No such file or directory\n     1\tOct 25 05:10:02 localhost rsyslogd: [origin software=\"rsyslogd\" swVersion=\"8.24.0\" x-pid=\"63417\" x-info=\"h"..., str@entry=0x7ecd19fc "xw\a\002", len=len@entry=262144)
at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/rfbserver.c:3750

3 0x00081dd0 in selection_send (ev=ev@entry=0x7ecd1aa8) at selection.c:508

4 0x000bd678 in check_xevents (reset=reset@entry=0) at xevents.c:1471

5 0x0007fc18 in watch_loop () at screen.c:4611

6 0x000291e8 in main (argc=, argv=) at x11vnc.c:5990

(gdb) frame 0

0 0x76ea6b8c in rfbStatLookupMessage (cl=cl@entry=0x1fa5460, type=type@entry=3)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:196

196 if (ptr->type==type) { (gdb) frame 1

1 0x76ea6d10 in rfbStatRecordMessageSent (cl=cl@entry=0x1fa5460, type=type@entry=3, byteCount=byteCount@entry=262152, byteIfRaw=byteIfRaw@entry=262152)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:256

256 ptr = rfbStatLookupMessage(cl, type); (gdb)

To Reproduce

Expected Behavior

it should be able to paste the large text at least 256 KB without any server crash. server must be pasting large text to client clipboard. Logs/Backtraces

(gdb) bt

0 0x76ea6b8c in rfbStatLookupMessage (cl=cl@entry=0x1fa5460, type=type@entry=3)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:196

1 0x76ea6d10 in rfbStatRecordMessageSent (cl=cl@entry=0x1fa5460, type=type@entry=3, byteCount=byteCount@entry=262152, byteIfRaw=byteIfRaw@entry=262152)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:256

2 0x76ea279c in rfbSendServerCutText (rfbScreen=,

str=0x222010 <primary_str> "\n[root@ci-9cb65498f0d8 log]# cat -n 3000 messages\ncat: 3000: No such file or directory\n     1\tOct 25 05:10:02 localhost rsyslogd: [origin software=\"rsyslogd\" swVersion=\"8.24.0\" x-pid=\"63417\" x-info=\"h"..., str@entry=0x7ecd19fc "xw\a\002", len=len@entry=262144)
at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/rfbserver.c:3750

3 0x00081dd0 in selection_send (ev=ev@entry=0x7ecd1aa8) at selection.c:508

4 0x000bd678 in check_xevents (reset=reset@entry=0) at xevents.c:1471

5 0x0007fc18 in watch_loop () at screen.c:4611

6 0x000291e8 in main (argc=, argv=) at x11vnc.c:5990

(gdb) frame 0

0 0x76ea6b8c in rfbStatLookupMessage (cl=cl@entry=0x1fa5460, type=type@entry=3)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:196

196 if (ptr->type==type) { (gdb) frame 1

1 0x76ea6d10 in rfbStatRecordMessageSent (cl=cl@entry=0x1fa5460, type=type@entry=3, byteCount=byteCount@entry=262152, byteIfRaw=byteIfRaw@entry=262152)

at /usr/src/debug/libvncserver/0.9.13-r0/libvncserver-LibVNCServer-0.9.13/libvncserver/stats.c:256

256 ptr = rfbStatLookupMessage(cl, type); (gdb) Your environment (please complete the following information):

Additional context

i see this segmentation fault as well in frequently.

1 0x76ae1ae8 in poll_for_reply () at ../../libxcb-1.12/src/xcb_in.c:457

2 0x76ae23c0 in wait_for_reply () at ../../libxcb-1.12/src/xcb_in.c:515

3 0x76ae24ac in xcb_wait_for_reply () at ../../libxcb-1.12/src/xcb_in.c:546

4 0x76dc4d70 in _XReply (dpy=dpy@entry=0x1768470, rep=rep@entry=0x7ef48040, extra=2129952832, extra@entry=0, discard=1995091968, discard@entry=1)

at ../../libX11-1.6.3/src/xcb_io.c:602

5 0x76dbb374 in XQueryPointer (dpy=dpy@entry=0x1768470, w=w@entry=246, root=root@entry=0x7ef480e4, child=child@entry=0x7ef480e8,

root_x=root_x@entry=0x7ef480ec, root_y=0x7ef480f0, root_y@entry=0x4c7d0 <check_x11_pointer+144>, win_x=0x7ef480f4, win_x@entry=0x7ef480ec,
win_y=0x7ef480f8, win_y@entry=0x7ef480f0, mask=mask@entry=0x7ef480fc) at ../../libX11-1.6.3/src/QuPntr.c:48

6 0x000c60a8 in XQueryPointer_wr (display=0x1768470, w=246, root_return=0x7ef480e4, root_return@entry=0x7ef480dc, child_return=0x7ef480e8,

child_return@entry=0x7ef480e0, root_x_return=root_x_return@entry=0x7ef480ec, root_y_return=0x7ef480f0, root_y_return@entry=0x7ef480e8,
win_x_return=0x7ef480f4, win_x_return@entry=0x7ef480ec, win_y_return=0x7ef480f8, win_y_return@entry=0x7ef480f0, mask_return=0x7ef480fc,
mask_return@entry=0x7ef480f4) at xwrappers.c:1423

7 0x0004c7d0 in check_x11_pointer () at cursor.c:2000

8 0x000809d0 in check_cursor_changes () at screen.c:4137

9 watch_loop () at screen.c:4719

10 0x000291e8 in main (argc=, argv=) at x11vnc.c:5990

(gdb) bt 0 full (More stack frames follow...) (gdb) frame 0

0 0x76c5feac in __GI___libc_free (mem=) at malloc.c:3123

3123 ar_ptr = arena_for_chunk (p); (gdb) frame 1

1 0x76ae1ae8 in poll_for_reply () at ../../libxcb-1.12/src/xcb_in.c:457

457 free(head);

brahma1989 commented 2 years ago

This issue occurs when you launch a vnc console in a browser either in Chrome or Mozilla. the x11vnc server crashes when you select a large text more than 40kb of size in console, the reason for the crash is a buffer overflow in web socket encoding code, basically, the connection between the NoVnc client to x11vnc server to be established using web socket APIs which are part of libvncserver library.

the server has to push/send the text to be selected to all the connected clients and it performs data encoding before pushing the text to the NoVnc client which is using web socket encoding. The x11vnc server allocates 256 kb of buffer to hold the select text but web socket encoding allocates only 40kb of buffer to hold the data that is encoded, that's where the buffer overflow occurs when the server pushes more than 40kb of data to be encoded, which corrupts the memory of data structures eventually causing the server to down when it tries to access the bad memory.

To maintain the same buffer lengths in both x11vnc and web socket encode, this below patch allocates 256kb of buffer in web socket code which solves the x11vnc server crash issue.

diff -ruN a/libvncserver/ws_decode.h b/libvncserver/ws_decode.h --- a/libvncserver/ws_decode.h 2021-11-02 11:28:23.023362582 +0530 +++ b/libvncserver/ws_decode.h 2021-11-05 12:09:16.134032729 +0530 @@ -28,6 +28,7 @@

define WS_HYBI_MASK_LEN 4

define ARRAYSIZE(a) ((sizeof(a) / sizeof((a[0]))) / (size_t)(!(sizeof(a) % sizeof((a[0])))))

+#define WS_MAX_ENCODE_BUF (262144L)

struct ws_ctx_s; typedef struct ws_ctx_s ws_ctx_t; @@ -112,7 +113,7 @@

struct ws_ctx_s { char codeBufDecode[2048 + WSHLENMAX]; / base64 + maximum frame header length /

maxnet commented 2 years ago

To maintain the same buffer lengths in both x11vnc and web socket encode, this below patch allocates 256kb of buffer in web socket code which solves the x11vnc server crash issue.

Eh, what about the possibility that other applications that are using libvncserver do not have the same 256 kb limit as x11vnc, and are sending even larger amounts than that? You would be leaving a buffer overflow in them...

Isn't there any other solution? E.g. changing rfbWriteExact() so that it calls itself multiple times with smaller chunks, if websockets are used and len>UPDATE_BUF_SIZE https://github.com/LibVNC/libvncserver/blob/master/libvncserver/sockets.c#L812

I don't think it matters if things gets split over multiple websocket frames, correct?