VK_PACKET ignored by VNC viewer for Windows

svenssonaxel commented 1 week ago

Describe the bug MS Windows has a special virtual keycode (VK) called VK_PACKET that indicates that the scan code is to be interpreted as a character code instead. This keycode is frequently used by applications that inject key events. It appears that VNC viewer ignores this keycode and sends no key events to the server. This bug likely also caused #1818.

To Reproduce Steps to reproduce the behavior:

In Keepass 2, create an entry with password "a'a^a`a~a.
In Windows, select Swedish keyboard layout.
Connect to a remote server using Tiger VNC client and, on the remote server, open a text editor.
Use Keepass 2 autotype feature to type the password into the VNC client.
Notice the text aaaaa appearing in the remote server's text editor.

Expected behavior The text "a'a^a`a~a should appear.

Client (please complete the following information):

OS: Windows 10 Pro
VNC client: TigerVNC
VNC client version: 1.12.0

Server (please complete the following information):

OS: Debian
VNC server: x11vnc
VNC server version: 0.9.16
Server downloaded from: NixOS
Server was started using: x11vnc -N -forever -loop100 -usepw -nevershared -clear_keys -v

Additional context

My guess is that Keepass 2 will use VK_PACKET on any keyboard layout when auto-typing characters not found in that keyboard layout.
You may add -pipeinput "reopen:cat - > /tmp/keys.in.fifo" to the x11vnc arguments to get a text stream of the key events received by the server.

CendioOssman commented 4 days ago

VK_PACKET is a bit problematic in that it doesn't give any physical key information. But I suppose we could try a best effort approach.

Could you see if you could generate a debug log so we get some more details about the events?

svenssonaxel commented 4 days ago

VK_PACKET is a bit problematic in that it doesn't give any physical key information. But I suppose we could try a best effort approach.

IIUC,

VK_PACKET events carry character code information (UTF-16 code units) instead of scan codes.
vncviewer's job is to translate key events to Linux keysym codes for use in RFB
Unicode to keysym translation is little more than table lookup and on fail, add an offset.

So I don't really see how lack of physical key information would provide any obstacle for supporting VK_PACKET on the client side.

Could you see if you could generate a debug log so we get some more details about the events?

I have tried for a very long time to build vncviewer.exe, to no avail. BUILDING.txt contains 14-year-old instructions that today are rather unhelpful. I've failed, thoroughly, to build it using MinGW both under Linux and Cygwin. I've also tried following the GHA workflow locally under MSYS2, and even that failed. Unless BUILDING.txt can be updated, I will not be able to produce a windows executable.

What I can do, is provide you with an event log from listenkey.exe -i [1] when auto-typing a test string on the Windows client, and the corresponding xev output on the Linux server.

The test string is a"b'c^d`e~f＋g𐀂h. It corresponds to the following character code sequence: U+0061, U+0022, U+0062, U+0027, U+0063, U+005e, U+0064, U+0060, U+0065, U+007e, U+0066, U+ff0b, U+0067, U+010002, U+0068.

Here is the listenkey.exe -i output:


{"type":"keydown","win_scancode":   30,"win_virtualkey": 65,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411751,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   30,"win_virtualkey": 65,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411751,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   34,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411783,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   34,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411783,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   48,"win_virtualkey": 66,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411829,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   48,"win_virtualkey": 66,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411829,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   39,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411861,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   39,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411861,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   46,"win_virtualkey": 67,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411908,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   46,"win_virtualkey": 67,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411908,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   94,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411954,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   94,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952411954,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   32,"win_virtualkey": 68,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412001,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   32,"win_virtualkey": 68,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412001,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   96,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412048,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   96,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412048,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   18,"win_virtualkey": 69,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412095,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   18,"win_virtualkey": 69,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412095,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":  126,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412158,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":  126,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412158,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   33,"win_virtualkey": 70,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412204,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   33,"win_virtualkey": 70,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412204,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":65291,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412251,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":65291,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412251,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   34,"win_virtualkey": 71,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412298,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   34,"win_virtualkey": 71,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412298,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":55296,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412361,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":55296,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412361,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":56322,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412408,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":56322,"win_virtualkey":231,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412408,"win_eventname":"WM_KEYUP"}
{"type":"keydown","win_scancode":   35,"win_virtualkey": 72,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412454,"win_eventname":"WM_KEYDOWN"}
{"type":"keyup"  ,"win_scancode":   35,"win_virtualkey": 72,"win_extended":false,"win_injected":true ,"win_lower_il_injected":false,"win_altdown":false,"win_time": 952412454,"win_eventname":"WM_KEYUP"}

As you can see, U+ff0b is inserted into the scan code field as-is, while U+010002 is split into two key presses, one for each UTF-16 code unit (0xd800, 0xdc02).

Printing the test string on Linux using xdotool into an xev window produces the following:


KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712663, (2659,395), root:(2662,423),
    state 0x0, keycode 38 (keysym 0x61, a), same_screen YES,
    XLookupString gives 1 bytes: (61) "a"
    XmbLookupString gives 1 bytes: (61) "a"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712667, (2659,395), root:(2662,423),
    state 0x0, keycode 38 (keysym 0x61, a), same_screen YES,
    XLookupString gives 1 bytes: (61) "a"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712674, (2659,395), root:(2662,423),
    state 0x0, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712674, (2659,395), root:(2662,423),
    state 0x1, keycode 48 (keysym 0x22, quotedbl), same_screen YES,
    XLookupString gives 1 bytes: (22) """
    XmbLookupString gives 1 bytes: (22) """
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712678, (2659,395), root:(2662,423),
    state 0x1, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712678, (2659,395), root:(2662,423),
    state 0x0, keycode 48 (keysym 0x27, apostrophe), same_screen YES,
    XLookupString gives 1 bytes: (27) "'"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712681, (2659,395), root:(2662,423),
    state 0x0, keycode 56 (keysym 0x62, b), same_screen YES,
    XLookupString gives 1 bytes: (62) "b"
    XmbLookupString gives 1 bytes: (62) "b"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712685, (2659,395), root:(2662,423),
    state 0x0, keycode 56 (keysym 0x62, b), same_screen YES,
    XLookupString gives 1 bytes: (62) "b"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712690, (2659,395), root:(2662,423),
    state 0x0, keycode 48 (keysym 0x27, apostrophe), same_screen YES,
    XLookupString gives 1 bytes: (27) "'"
    XmbLookupString gives 1 bytes: (27) "'"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712693, (2659,395), root:(2662,423),
    state 0x0, keycode 48 (keysym 0x27, apostrophe), same_screen YES,
    XLookupString gives 1 bytes: (27) "'"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712696, (2659,395), root:(2662,423),
    state 0x0, keycode 54 (keysym 0x63, c), same_screen YES,
    XLookupString gives 1 bytes: (63) "c"
    XmbLookupString gives 1 bytes: (63) "c"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712700, (2659,395), root:(2662,423),
    state 0x0, keycode 54 (keysym 0x63, c), same_screen YES,
    XLookupString gives 1 bytes: (63) "c"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712704, (2659,395), root:(2662,423),
    state 0x0, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712705, (2659,395), root:(2662,423),
    state 0x1, keycode 15 (keysym 0x5e, asciicircum), same_screen YES,
    XLookupString gives 1 bytes: (5e) "^"
    XmbLookupString gives 1 bytes: (5e) "^"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712709, (2659,395), root:(2662,423),
    state 0x1, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712709, (2659,395), root:(2662,423),
    state 0x0, keycode 15 (keysym 0x36, 6), same_screen YES,
    XLookupString gives 1 bytes: (36) "6"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712713, (2659,395), root:(2662,423),
    state 0x0, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (64) "d"
    XmbLookupString gives 1 bytes: (64) "d"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712717, (2659,395), root:(2662,423),
    state 0x0, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (64) "d"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712723, (2659,395), root:(2662,423),
    state 0x0, keycode 49 (keysym 0x60, grave), same_screen YES,
    XLookupString gives 1 bytes: (60) "`"
    XmbLookupString gives 1 bytes: (60) "`"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712727, (2659,395), root:(2662,423),
    state 0x0, keycode 49 (keysym 0x60, grave), same_screen YES,
    XLookupString gives 1 bytes: (60) "`"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712730, (2659,395), root:(2662,423),
    state 0x0, keycode 26 (keysym 0x65, e), same_screen YES,
    XLookupString gives 1 bytes: (65) "e"
    XmbLookupString gives 1 bytes: (65) "e"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712734, (2659,395), root:(2662,423),
    state 0x0, keycode 26 (keysym 0x65, e), same_screen YES,
    XLookupString gives 1 bytes: (65) "e"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712740, (2659,395), root:(2662,423),
    state 0x0, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712740, (2659,395), root:(2662,423),
    state 0x1, keycode 49 (keysym 0x7e, asciitilde), same_screen YES,
    XLookupString gives 1 bytes: (7e) "~"
    XmbLookupString gives 1 bytes: (7e) "~"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712744, (2659,395), root:(2662,423),
    state 0x1, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712745, (2659,395), root:(2662,423),
    state 0x0, keycode 49 (keysym 0x60, grave), same_screen YES,
    XLookupString gives 1 bytes: (60) "`"
    XFilterEvent returns: False

KeyPress event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712748, (2659,395), root:(2662,423),
    state 0x0, keycode 41 (keysym 0x66, f), same_screen YES,
    XLookupString gives 1 bytes: (66) "f"
    XmbLookupString gives 1 bytes: (66) "f"
    XFilterEvent returns: False

KeyRelease event, serial 52, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712752, (2659,395), root:(2662,423),
    state 0x0, keycode 41 (keysym 0x66, f), same_screen YES,
    XLookupString gives 1 bytes: (66) "f"
    XFilterEvent returns: False

MappingNotify event, serial 52, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 52, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 52, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 52, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

KeyPress event, serial 56, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712758, (2659,395), root:(2662,423),
    state 0x0, keycode 148 (keysym 0x100ff0b, UFF0B), same_screen YES,
    XLookupString gives 3 bytes: (ef bc 8b) "＋"
    XmbLookupString gives 3 bytes: (ef bc 8b) "＋"
    XFilterEvent returns: False

MappingNotify event, serial 56, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 56, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 56, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 56, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

KeyRelease event, serial 60, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712764, (2659,395), root:(2662,423),
    state 0x0, keycode 148 (keysym 0x0, NoSymbol), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 60, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712769, (2659,395), root:(2662,423),
    state 0x0, keycode 42 (keysym 0x67, g), same_screen YES,
    XLookupString gives 1 bytes: (67) "g"
    XmbLookupString gives 1 bytes: (67) "g"
    XFilterEvent returns: False

KeyRelease event, serial 60, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712774, (2659,395), root:(2662,423),
    state 0x0, keycode 42 (keysym 0x67, g), same_screen YES,
    XLookupString gives 1 bytes: (67) "g"
    XFilterEvent returns: False

MappingNotify event, serial 60, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 60, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 60, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 60, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

KeyPress event, serial 64, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712778, (2659,395), root:(2662,423),
    state 0x0, keycode 148 (keysym 0x1010002, U00010002), same_screen YES,
    XLookupString gives 4 bytes: (f0 90 80 82) "𐀂"
    XmbLookupString gives 4 bytes: (f0 90 80 82) "𐀂"
    XFilterEvent returns: False

MappingNotify event, serial 64, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 64, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 64, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

MappingNotify event, serial 64, synthetic NO, window 0x0,
    request MappingKeyboard, first_keycode 148, count 1

KeyRelease event, serial 68, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712784, (2659,395), root:(2662,423),
    state 0x0, keycode 148 (keysym 0x0, NoSymbol), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 68, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712791, (2659,395), root:(2662,423),
    state 0x0, keycode 43 (keysym 0x68, h), same_screen YES,
    XLookupString gives 1 bytes: (68) "h"
    XmbLookupString gives 1 bytes: (68) "h"
    XFilterEvent returns: False

KeyRelease event, serial 68, synthetic NO, window 0x1c00001,
    root 0x50d, subw 0x0, time 1184712795, (2659,395), root:(2662,423),
    state 0x0, keycode 43 (keysym 0x68, h), same_screen YES,
    XLookupString gives 1 bytes: (68) "h"
    XFilterEvent returns: False

As you can see, U+010002 is here sent as one keysym only, 0x1010002. (Ignore keysym and XLookupString info in KeyRelease events. The effective keysym of a KeyRelease event is the keysym of the latest preceding KeyPress event with a matching keycode.)

It appears to me the only challenge on the vncclient.exe side would be the state management for UTF-16 code units. Note that UTF-16 code unit pairs that synthesize into one unicode code point are simply identifiable with conditions ((first_scancode & 0xfc00) == 0xd800) and ((second_scancode & 0xfc00) == 0xdc00). After that, translation to unicode should be as simple as code_point = (((first_scancode & 0x3ff) << 10) | (second_scancode & 0x3ff)) + 0x10000;. I assume there is already code to translate from unicode code point to keysym to be sent over RFB.

[1] listenkey.exe obtainable from https://github.com/svenssonaxel/keyboa/archive/refs/tags/0.2.0-alpha.191002.tar.gz

CendioOssman commented 2 days ago

So I don't really see how lack of physical key information would provide any obstacle for supporting VK_PACKET on the client side.

We need to track which keys are pressed to maintain a sane state for the server. We can't track symbols since a single key can generate different symbols. I wonder if MapVirtualKey() does something sensible for VK_PACKET.

I have tried for a very long time to build vncviewer.exe, to no avail. BUILDING.txt contains 14-year-old instructions that today are rather unhelpful. I've failed, thoroughly, to build it using MinGW both under Linux and Cygwin. I've also tried following the GHA workflow locally under MSYS2, and even that failed. Unless BUILDING.txt can be updated, I will not be able to produce a windows executable.

We already have debug output, you just need to enable it. No need to rebuild TigerVNC:

https://github.com/TigerVNC/tigervnc/wiki/Debug-Logs#client-1

That said, the build instructions should be current. Where did you get stuck?

It appears to me the only challenge on the vncclient.exe side would be the state management for UTF-16 code units

Yeah, that looks like it would require some more state management. I wonder if it is worth the hassle.

CendioOssman commented 2 days ago

Hmm... To make things more confusing, WM_KEYUP only has 8 bits available for "scan code", which is quite insufficient for the 16 bits VK_PACKET needs.

Looking at the Wine code, it suggest that VK_PACKET overwrites a few other fields as well. But that would need to be tested.

CendioOssman commented 2 days ago

If you manage to get a build going, please see how this patch works:

diff --git a/vncviewer/Viewport.cxx b/vncviewer/Viewport.cxx
index e29c877cf..f66ef81a5 100644
--- a/vncviewer/Viewport.cxx
+++ b/vncviewer/Viewport.cxx
@@ -971,6 +971,15 @@ int Viewport::handleSystemEvent(void *event, void *data)
       keyCode = 0x38;
     }

+    // Windows will send VK_PACKET if it doesn't have a physical key
+    // associated with the event (e.g. from a mobile device virtual
+    // keyboard). In such cases, the "scan code" is actually a UTF-16
+    // code point.
+    if (vKey == VK_PACKET) {
+      isExtended = false;
+      keyCode = 0;
+    }
+
     // Windows doesn't have a proper AltGr, but handles it using fake
     // Ctrl+Alt. However the remote end might not be Windows, so we need
     // to merge those in to a single AltGr event. We detect this case
@@ -1030,7 +1039,10 @@ int Viewport::handleSystemEvent(void *event, void *data)
     if (keyCode == 0xb7)
       keyCode = 0x54;

-    keySym = win32_vkey_to_keysym(vKey, isExtended);
+    if (vKey == VK_PACKET)
+      keySym = ucs2keysym(msg->lParam >> 16);
+    else
+      keySym = win32_vkey_to_keysym(vKey, isExtended);
     if (keySym == NoSymbol) {
       if (isExtended)
         vlog.error(_("No symbol for extended virtual key 0x%02x"), (int)vKey);

svenssonaxel commented 1 day ago

We need to track which keys are pressed to maintain a sane state for the server. We can't track symbols since a single key can generate different symbols. I wonder if MapVirtualKey() does something sensible for VK_PACKET.

You're talking about the server, while this ticket is only about the client. Even so, IIUC the server would receive keysyms without any physical key info over the RFB protocol. If the server is Windows, you could then use VK_PACKET to send key events for keysyms that do not appear in the current keyboard layout. Again, this is a separate question, but if you're interested I have example code here for injecting key events in Windows to write any text no matter the keyboard layout.

We already have debug output, you just need to enable it. No need to rebuild TigerVNC:

Thank you. See vncviewer.log for an example when auto-typing the same example string I mentioned earlier.

That said, the build instructions should be current. Where did you get stuck?

See Dockerfile-debian.txt which you can run as docker build -f Dockerfile-debian.txt .. As commented in that file, when using the documented build command, ZLIB cannot be found, and although I can solve this, I cannot solve all problems.

Yeah, that looks like it would require some more state management. I wonder if it is worth the hassle.

I'm willing to submit a PR if I can build and test. Keepass and vncviewer are pretty much the only windows programs I use, so it'd be nice if they can talk as intended :-)

Hmm... To make things more confusing, WM_KEYUP only has 8 bits available for "scan code", which is quite insufficient for the 16 bits VK_PACKET needs.

My test above using listenkey.exe suggests otherwise, featuring 16-bit scan code values for both down and up events. Even if this is somehow not the case, there is still significant value in supporting the special case where each keydown event using VK_PACKET is immediately followed by a corresponding keyup event (this is the case with keepass auto-type, as well as most use cases I can think of). Even if you can only read 8 bits you can assume the rest.

Looking at the Wine code, it suggest that VK_PACKET overwrites a few other fields as well. But that would need to be tested.

If you manage to get a build going, please see how this patch works:

Would be happy to! I'd need help as explained above.

svenssonaxel commented 1 day ago

If you manage to get a build going, please see how this patch works:

I have a Windows build working now, using nix. This patch seems to change no behaviour. I'll look into the code eventually if you don't beat me to it.

CendioOssman commented 21 hours ago

Could you share a debug log with the patch applied? There should hopefully be some more details.

svenssonaxel commented 8 hours ago

@CendioOssman Managed to fix it and opened a PR with a patch that works for me.

TigerVNC / tigervnc

VK_PACKET ignored by VNC viewer for Windows #1847