Feature: VNC Compatibility on Linux Coming Soon

[rbreaves]

Sorry I have been a bit inactive lately @joshgoebel and @RedBearAK and I do plan to catch up soon on the outstanding tickets, sadly though I will be out for most of this weekend.

I have been making some real progress though on making Kinto compatible with VNC servers, whether or not this can be leveraged and layered with xrdp I do not know. Currently though I am having issues with getting Super_L or the Win key to detect properly but physical keyboard input is happening now over VNC using this sort of method.

Note: apologies for the lack of no cursor that needs to be added

sudo /usr/bin/x11vnc -forever -display :0 -auth guess -rfbport 5900 -pipeinput UINPUT:direct_key=/dev/input/event1 -usepw

You can then also stop the xkeysnail service and run it in another tab with this.

sudo /usr/local/bin/xkeysnail ~/.config/kinto/kinto.py
# Can specify device this way too
sudo /usr/local/bin/xkeysnail --device=/dev/input/event1 ~/.config/kinto/kinto.py

I feel like this is literally the most game changing thing I have done since writing Kinto itself. My x11vnc command is not optimized at all just yet for speed or anything like that, but I know it can be sped up significantly after doing some testing with VNC in general under vmware. Bringing this to metal though or direct to the VMs themselves I think will be a pretty big development.

It is funny that the pipeinput parameter was there all along and I never noticed it at all until I saw a single reference to a mouse input redirection.

At any rate this error needs resolving.

09/06/2022 23:48:47 3*Super_L, calling: set_xdamage_mark()

More info can be found here. https://github.com/LibVNC/x11vnc/issues/193

[rbreaves]

A bit more debug info posted here https://github.com/LibVNC/x11vnc/issues/193#issuecomment-1151958727

[joshgoebel]

-pipeinput UINPUT:direct_key=/dev/input/event1

Is this the relevant part? So... (trying to understand)... is what this does to take the networked keyboard events and instead of passing them directly it creates a uinput device and dumps them into that, then you're going to the keymapper with that uinput as input into a second uinput?

Or am I totally getting this wrong? And now I'm wondering how the keyboard events ever got injected in the first place if it's not thru uinput, perhaps just using the X (higher) layer for keyboard events instead?

[rbreaves]

Is this the relevant part? Yes.

And instead of sending the input through x11 in some synthetic manner that goes through the normal xkb and xkbmap stuff it yea sends it direct to the existing uinput device which is being shared with the existing physical keyboard as it dumps its output into the same data stream or whatever terminal should be used for /dev/input/event1. Uinput is the only thing xkeysnail understands after all.

As far as xkeysnail is concerned your remote keyboard is your already existing keyboard sitting in front of your computer since all that input is ending up in the same place.. clearly though the Super key is not being transmitted correctly.. might require a patch to x11vnc.

[rbreaves]

Also finding out more info.. the Super_L input.. while detected by x11vnc and resulting in info like this...

10/06/2022 00:38:12 client 3 network rate 167.0 KB/sec (4469.4 eff KB/sec)
10/06/2022 00:38:12 client 3 latency:  2.0 ms
10/06/2022 00:38:12 dt1: 0.1362, dt2: 0.6899 dt3: 0.0020 bytes: 137810
10/06/2022 00:38:12 link_rate: LR_UNKNOWN - 2 ms, 167 KB/s
10/06/2022 00:38:58 3*Super_L, calling: set_xdamage_mark()
10/06/2022 00:38:58 4*Super_L, calling: check_xrecord_reset()
10/06/2022 00:38:58 4*Super_L, calling: check_xrecord_reset()
10/06/2022 00:38:59 5*Super_L, calling: push_black_screen(0)
10/06/2022 00:39:02 3*Super_L, calling: set_xdamage_mark()
10/06/2022 00:39:03 3*Super_L, calling: set_xdamage_mark()

Note: screen really did go black here 10/06/2022 00:38:59 5*Super_L, calling: push_black_screen(0)

showkey doesn't register the Super_L key at all like it should.... I could hit Alt and Ctrl fine, but Super showed nothing unless I was at my physical keyboard.

sudo showkey --keycodes
kb mode was ?UNKNOWN?
[ if you are trying this under X, it might not work
since the X server is also reading /dev/console ]

press any key (program terminates 10s after last keypress)...
keycode  28 release
keycode  56 press
keycode  56 release
keycode  56 press
keycode  56 release
keycode  56 press
keycode  56 release
keycode  56 press
keycode  56 release
keycode  29 press
keycode  29 release
keycode  29 press

[rbreaves]

The problem could also be unique going from a Windows host to Linux for all I know.. Might work under Linux to Linux, or Mac to Linux.

[rbreaves]

Or am I totally getting this wrong? And now I'm wondering how the keyboard events ever got injected in the first place if it's not thru uinput, perhaps just using the X (higher) layer for keyboard events instead?

And yea by default vnc translates your remotes keyboard input into an xinput device which you can see listed when running xinput list but yea its input is as good as a ghost as far as xkeysnail is concerned as it just yet another keyboard input method under Linux which it knows nothing about. This has been a major thorn in my side ever since I migrated away from my custom C solution with xkbmap and over into xkeysnail.. just now understanding it all enough to begin to address it properly.

So far I have LITERALLY only found 2 or 3 references on Google of anyone at all using this pipeinput method and only one of them was actually using the direct_key feature. One other referenced using it as well, direct_key but didn't say direct_key or explain how which is what sent me on this wild goose chase and reading the x11vnc options readme a bit more carefully lol.

You can also specify the input device specifically via xkeysnail - if you know the proper location. Most likely it'll be event1 I think.

sudo systemctl stop xkeysnail
sudo /usr/local/bin/xkeysnail --device=/dev/input/event1 ~/.config/kinto/kinto.py

[rbreaves]

A bit more info..

# Get list of devices and uinput paths
ls -l /dev/input/by-id
# direct_key for keyboard
# direct_abs for mouse

sudo /usr/bin/x11vnc -forever -display :0 -auth guess -rfbport 5900 -cursor none -nodragging -noxdamage -pipeinput UINPUT:direct_key=/dev/input/event1,direct_abs=/dev/input/event7 -overlay -noxfixes -dk -usepw

Translation from 0xffeb - Super_L value to the keysym value of keycode 133. Is registered by x11vnc but nothing else, xev app will should most keys from remote session - except Super_L.

[rbreaves]

Fixed it!!!!

https://github.com/LibVNC/x11vnc/pull/195

Now the only other immediate issue that I see needing resolving has to do with 2 finger scrolling no longer working while using pipeinput...

For anyone that wants to build x11vnc from my branch with the fix then that is located here. https://github.com/rbreaves/x11vnc/tree/193-Fix-Super

[RedBearAK]

Is this going to translatable in any way to other remote desktop software besides VNC? RDP, AnyDesk, NoMachine, things like that?

[rbreaves]

I have not yet tested it against xRDP to say for 100% certain but I expect it resolves it there - xRDP just loops back to your vnc server. Granted some changes to a KB ini file may need to happen to pass through Super properly even over xrdp.. https://github.com/neutrinolabs/xrdp/issues/2169#issuecomment-1059792757

Anydesk & NoMachine are a no as I believe they use X11 instead of uinput, uinput for direct input is required - at least for the keyboard, not the mouse.. in fact the mouse w/ scrolling only works over X11.

So I have merged my 2 fixes for keyboard over uinput & mouse w/ scrolling over X11 here for anyone that wants to compile it & test it out.

https://gist.github.com/rbreaves/87059f92f030ee4e068f911ddb56a4dd

Closing this ticket even though it’s experimental atm & may need some tweaking before an official merge into x11vnc can happen. Mainly parameterization to allow keyboard over uinput & mouse over X11.

[rbreaves]

Please continue to post any questions either here, the fork or PRs.