search

sibson / vncdotool

A command line VNC client and python library
Other
451 stars 120 forks source link

No symbol characters when automating actions in QEMU/KVM VMs. #269

Open nick-george opened 1 year ago

nick-george commented 1 year ago

Please include the following information:

vncdotool version: vncdotool==1.2.0 VNC server and version: QEMU VNC Server, version 6.2+dfsg-2ubuntu6.12 (ubuntu 22.04) Steps to reproduce

Which erroneous result did you get instead Screenshot from 2023-08-29 08-08-34

Additional information This is my locale, do you think it could be somehow involved? 

LANG=en_AU.UTF-8
LANGUAGE=en_AU:en
LC_CTYPE="en_AU.UTF-8"
LC_NUMERIC="en_AU.UTF-8"
LC_TIME="en_AU.UTF-8"
LC_COLLATE="en_AU.UTF-8"
LC_MONETARY="en_AU.UTF-8"
LC_MESSAGES="en_AU.UTF-8"
LC_PAPER="en_AU.UTF-8"
LC_NAME="en_AU.UTF-8"
LC_ADDRESS="en_AU.UTF-8"
LC_TELEPHONE="en_AU.UTF-8"
LC_MEASUREMENT="en_AU.UTF-8"
LC_IDENTIFICATION="en_AU.UTF-8"
LC_ALL=

I'm aware of the 'shift-' operations, but I was hoping not to have to write my own parser for every character sent. Looking through the issue history for this project, it seems the capitalisation issues were solved years ago, so I figure it's an issue with QEMU in particular? Have you got an suggestions on how I can fix?

Entering the same characters in a vnc viewer works fine.

Many thanks!

sibson commented 1 year ago

Is it possible to try with an en_US locale? I don't think we've really solved the capitalization issue, just put in enough work-arounds that it mostly works. I've always suspected locale as being the culprit but never had a good repo.

pmhahn commented 1 year ago

I don't know how often I have answered such questions in the past, but the VNC / RFB protocol is not ideal for being used with virtual machines. Daniel P. Berrangé is one of the QEMU core developers and has several related blog posts on this issue; read them if you're interested in the gory details.

The summary is this:

  1. you have 3 mappings from key location (keycode) to key symbol (keysym), back, and forward again.
  2. First is the PC you're sitting on and where you're running your VNC client, e.g. vncdotool. The keyboard hardware sends a keycode like (2nd row, 2nd key from the left). The operating system on your PC translates this to 1 or ! or whatever is configured there.
  3. VNC / RFB works with keysyms, so that information is submitted over the wire to the VNC server, which is QEMU. QEMU does hardware emulation, so it needs to translate the keysym back into a keycode. Similar to your client OS the OS inside the VM expects a keycode. That back-translation is not unique, as the keysym 1 may have also been generated by the key from they numeric key block. This is the 2nd configuration, which can be specified by qemu -k … and should match the keyboard layout of your client. (if multiple people with different layouts access the same VM, basically you have lost; actually not as you can re-configure QEMU on the fly.)
  4. The 3rd and last conversion happens by the OS inside your VM, which then again converts from keycode to keysym.
  5. You get the most consistent experience if all 3 settings are the same, e.g. all US-English or German. If one of them is different, you might get strange results.
  6. To solve this issue QEMU introduced the QEMU Extended Key Event Message as an extension to RFB: In addition to the keysym it also includes the (original) keycode from your VNC client and sends them together to QEMU, which then ignores the keysym and simply passes the unmodified keycode into the VM: the only configuration left is then the keyboard layout configured inside your VM.
  7. BUT: When you use vncdotool you give it keysyms as arguments on the command line, e.g. 123. So vncdotool has to convert this into a sequence of key-presses, QEMU does its back-translation, and your OS inside the VM gets some keycodes.

vncdotool currently has no option to configure its keyboard layout (first conversion from above) and is hard-coded to en-US: vncdotool/client.py.

There are other open issues related to that, e.g. #139 #209

nick-george commented 1 year ago

Many thanks for your detailed response @pmhahn and @sibson. It's a lot to digest.

The discussion around it could be simplified to just vncdotool. If I use key shift-2, I get a different result to using "type @". When I use type @, I get a 2, but shift-2 works. This means that vncdotool is sending different keycodes for both scenarios. I want vncdotool to send the same data either way. The en_AU and en_US keyboard layouts are essentially the same (I'm not aware of any differences).

Can I easily troubleshoot how vncdotool converts from the user-supplied input to the virtual key-presses? Using the -v -v -v option, I can see it saying it's sending a '@', but from your explanation, it's actually sending raw key presses?

I understand that I could write a parser that tokenises each character of input I wish to send, then sets the appropriate shift-X code to make it work (for US keyboards only), but it seems there must be a better way.

In the end, I've decided that I probably can't use this tool to do what I want to achieve (automatic configuration of a machine), as there are probably too many failure scenarios to account for (Window positions etc).

Feel free to close the ticket. It would be awesome if the content of your comment above was included in the main docs for the project.

Cheers, Nick

pmhahn commented 11 months ago

For an en-US keyboard you're lucky and can use vncdo --force-caps …: For the characters listed in

SPECIAL_KEYSUS = '~!@#$%^&*()+{}|:"<>?'

vncdo will then automatically send shift-X key presses and releases instead, which should work better with Qemu.

Can I easily troubleshoot how vncdotool converts from the user-supplied input to the virtual key-presses? Using the -v -v -v option, I can see it saying it's sending a '@', but from your explanation, it's actually sending raw key presses?

Just this week I observed a similar issue with a very old version of Qemu 2.6 from ~2016; I just opened https://github.com/sibson/vncdotool/pull/270 which includes a patch to improve logging the key-codes sent; feel free to give my branch a try.

sibson commented 10 months ago

Thanks @pmhahn for the great explanation. I suspect VM is one of the more common use cases these days, and it would make sense to implement the QEMU extension. Not sure what that means for other VMs, but anything that improves things so it just works for most people is a good thing.