Closed Rhys-T closed 1 year ago
gujjwal00
commented
1 year ago I looked into how Option key is supposed to work in macOS, and Wikipedia suggests that it should be treated as "Meta" key instead of "Alt". As a quick test, please try the following APK: app-debug.zip. It will send Meta_L & Meta_R instead of Alt_L & Alt_R for Alt key. If Option key works as expected with this APK, then we know whats happening here.
Rhys-T
commented
1 year ago According to the Keyboard Viewer, that's hitting the correct keys now: Win is still Command, but Alt is now Option. However:
I've tried this with "Send legacy key events" on and off, and it behaves identically either way. Is there anything else I need to try? Do I need to open a separate issue for this, since it's not directly related to 'which key is which'?
gujjwal00
commented
1 year ago According to the Keyboard Viewer, that's hitting the correct keys now: Win is still Command, but Alt is now Option.
Great, that was the main hurdle.
If I use the Hacker's Keyboard Alt key, it works for keyboard shortcuts (e.g. ⌘⌥H → Hide Others, ⌥⌫ → delete an entire word), but not special characters (e.g. ⌥Z → Ω) - it just acts like I wasn't pressing Alt/Option at all, and types the letter as is. 'Dead keys', like accents, do sort of work (e.g. ⌥E E → é), but before pressing the final letter, I don't get the underlined accent by itself (e.g. ´), like I do when typing directly on the Mac.
Accents are expected to work like that. See here for some background. Basically, they are composed by AVNC itself, instead of being sent to server.
I am not sure why ⌥Z → Ω is not working. What is the the expected character for this combination?
Using the Alt key from AVNC's virtual key strip, I get almost the same thing, except the dead keys don't work either - it just ignores the Alt/Option key like it does for any other letter. Keyboard shortcuts still work, nothing else does.
Alt from virtual keys should work, but I have never tested it with accents. One thing worth noting is that this Alt simulates Left Alt key, and accents might require Right Alt key to be pressed.
I've tried this with "Send legacy key events" on and off, and it behaves identically either way.
That should not have any effect here,
Is there anything else I need to try? Do I need to open a separate issue for this, since it's not directly related to 'which key is which'?
No need as accents are working as expected IMO.
About how to implement this change: There is no way for AVNC to detect if server is running on macOS, but we can let the user specify it. Server database in AVNC already has a column for this purpose, but its not used yet.
I am also hoping to implement configurable key-mappings someday, but its a lot of work and it probably won't be per-server.
Rhys-T
commented
1 year ago Great, that was the main hurdle.
So it sounds like basically VNC has Alt, Super, and Meta keys (inherited from X11), and Apple is treating the first two as Command, and Meta as Option? Even though Apple acts like Alt = Option in every other context? Weird, but at least it can be worked around.
Accents are expected to work like that. See here for some background. Basically, they are composed by AVNC itself, instead of being sent to server.
Ah - didn't realize that. So if I understand correctly, AVNC is trying to make keys - and by extension, Alt-key combos and dead keys - enter whatever characters they would have typed on the local Android device?
(Weirdly, I can't actually get any dead keys to do anything useful on Android apps that aren't AVNC. I've tried using the Alt keys in both Hacker's Keyboard and Unexpected Keyboard, typing into both Firefox and Samsung Notes. If I press e.g. Alt-E, it enters a ´, but it's selected/highlighted, and typing a vowel afterwards just overwrites the accent.)
I am not sure why
⌥Z → Ωis not working. What is the the expected character for this combination?
Yeah, I probably didn't write that very clearly. What I meant was that on macOS (with the U.S. keyboard layout), ⌥ Option-Z maps to the Ω (capital omega) character. (I don't know what it's supposed to do on Android - it's not typing anything for me.) It was just an example of a non-accent character that uses the ⌥ Option key on macOS.
With ⌥ Option and ⇧ Shift:
Alt from virtual keys should work, but I have never tested it with accents. One thing worth noting is that this Alt simulates Left Alt key, and accents might require Right Alt key to be pressed.
As far as I know, both Option keys should work the same on macOS[^rightoption] - there isn't a separate AltGr key or anything like that, at least for the U.S. layout. I'm not sure what the difference is with the virtual keys.
[^rightoption]: Unless an app is specifically checking for left/right Option, like for game controls or something. And from what I understand, a lot of times that doesn't even work, and they both report as left Option (and similarly for the other modifier keys).
["Send legacy key events"] should not have any effect here,
I'm honestly not entirely clear on what that option does - it just sounded like it could affect the ability to type non-ASCII characters, so I figured it wouldn't hurt to try toggling it, in case this was just user error.
No need as accents are working as expected IMO.
Okay. Personally, I was expecting it to just pass the keys through as is, and let macOS decide how to handle them. But that's probably just a 'me' thing. If someone is used to how these keys work on Android with regards to entering characters like that, it probably makes sense to try and keep them working that way.
About how to implement this change: There is no way for AVNC to detect if server is running on macOS, but we can let the user specify it. Server database in AVNC already has a column for this purpose, but its not used yet.
The only way I could come up with to detect macOS is to look for things like Apple's security types, or the (officially nonexistent as far as I know) 'RFB 003.889' version of the protocol that their server claims to support. But I don't know how robust that would actually be. I guess it's probably better to make it an explicit setting.
I am also hoping to implement configurable key-mappings someday, but its a lot of work and it probably won't be per-server.
No problem. Like I said, I don't really have any use case for full remapping right now - I was just trying to figure out how to get the Option key working.
gujjwal00
commented
1 year ago So it sounds like basically VNC has Alt, Super, and Meta keys (inherited from X11), and Apple is treating the first two as Command, and Meta as Option? Even though Apple acts like Alt = Option in every other context? Weird, but at least it can be worked around.
Yes. You can look through Modifier Keys for for some history. Now, Apple says here that Alt key is equivalent to Option, but the VNC server on macOS seems to require Meta key for triggering Option. There is probably some historical reason for this behavior.
Ah - didn't realize that. So if I understand correctly, AVNC is trying to make keys - and by extension, Alt-key combos and dead keys - enter whatever characters they would have typed on the local Android device?
Yes, because if AVNC sends 'dead-key' & 'character' separately, accents don't work at all. So if AVNC receives a dead key (e.g. ` , it will wait for the non-dead key (e.g. a). When such key is entered, AVNC will combine it with previous dead key to produce a single character à, which will be sent to the server. But this composition is only triggered for dead keys, and all other key combinations are sent straight through.
Yeah, I probably didn't write that very clearly. What I meant was that on macOS (with the U.S. keyboard layout), ⌥ Option-Z maps to the Ω (capital omega) character. (I don't know what it's supposed to do on Android - it's not typing anything for me.) It was just an example of a non-accent character that uses the ⌥ Option key on macOS.
Thanks for explaining, I will look more into this case.
I'm honestly not entirely clear on what that option does - it just sounded like it could affect the ability to type non-ASCII characters, so I figured it wouldn't hurt to try toggling it, in case this was just user error.
Truth be told, this option should not exist at all. You are not the first one to be confused by this, and I doubt anyone except me understands what it actually does.
So you have probably know, or figured out by now, that VNC protocol works on "key symbols", i.e. what you see printed on key caps. Instead of defining a brand new table of all key symbols and corresponding integer values (think Unicode), VNC protocol uses "X Key Symbols", i.e. symbols defined by X System Protocol. Originally, X protocol assigned custom values to most symbols (this was probably before Unicode). X protocol also has newer symbol scheme with straight forward mapping to all Unicode symbols. So the older symbol mappings which are superseded by Unicode are now "legacy".
But many VNC servers don't handle the newer symbol values. So I added an option to control this behavior. IMO, this option should be hard-coded to true, and the checkbox should be removed from UI.
Okay. Personally, I was expecting it to just pass the keys through as is, and let macOS decide how to handle them. But that's probably just a 'me' thing. If someone is used to how these keys work on Android with regards to entering characters like that, it probably makes sense to try and keep them working that way.
In addition to the issue mentioned above, another problem is that Android doesn't report the actual dead keys in event. It reports their corresponding "printing" keys. See the difference between Unicode characters 0x02CB & 0x0300.
The only way I could come up with to detect macOS is to look for things like Apple's security types, or the (officially nonexistent as far as I know) 'RFB 003.889' version of the protocol that their server claims to support. But I don't know how robust that would actually be. I guess it's probably better to make it an explicit setting.
Really great observation, thank you for finding it :+1: . I too looked around and only macOS seems to use it. This will make it straightforward to implement the workaround, without user having to do anything.
Rhys-T
commented
1 year ago Yeah, I was thinking in terms of (virtual) keycodes ("that key that's typically three to the right from the tab key") but RFB is actually using keysyms ("the E key, wherever it is on this keyboard", but with whatever modifications Shift, Alt, etc. have made to it, so almost a character, but not modified by things like Control…).[^qemu]
[^qemu]: ~There is apparently an extension for doing things in terms of keycodes, but it seems to be QEMU-specific, and limited to using PC/XT keycodes, so not really relevant to this situation.~ Edit: Never mind, you're already implementing that extension.
Android not giving you the combining character for a dead key sort of makes sense, maybe? At least on macOS, there's no guarantee that a dead key even directly corresponds to a combining character/accent. As an extreme example, if you set the layout to "Unicode Hex Input", you can hold Option and enter a four-hex-digit Unicode code point. The first three digits are all dead keys in that case, but they don't correspond to specific characters at all. The "ABC - Extended" layout (formerly "U.S. Extended") also has Option-Shift-; - labeled as '№', but with lots of unrelated characters behind it that aren't made from combining characters. I'd guess that what you're getting is probably the preview/placeholder character that shows up in the text until the final key is pressed.
Just to be clear, what I was expecting was that if I pressed Alt-E in AVNC, macOS would see:
and do whatever that would do on a keyboard attached directly to that Mac, whether that meant a dead key, a special character, or even a keyboard shortcut in the current app[^vscode]. But that may not be a reasonable expectation, given how VNC represents keys. (There are two different operating systems involved, with different ideas about how keyboards should work, trying to hold a conversation about keys in a third operating system's language. It's amazing that it works at all. 😀 And would a typical user consider it 'reasonable' for keys to suddenly work differently just because they were using a VNC app? What's the 'reasonable' thing to do if the client is using QWERTY and the server is set to Dvorak?) And even if it's possible to implement that, it may well need to be a separate feature request from this one.
[^vscode]: Option by itself, without Command, doesn't normally get used for shortcuts, but it can happen, especially with cross-platform apps. For example, Visual Studio Code defaults to using Option-Z to toggle word wrap.
For what it's worth, if I use Apple's Screen Sharing app to connect from one Mac to another, it seems to work like that, and use the server's interpretation of key sequences no matter what the client would have done. But I think it may be kind of 'faking it', because it starts to fall apart if the two Macs are set to different keyboard layouts. For instance, if I set the server to Dvorak, or ABC - Extended, and type a key sequence that's a dead key on the server, but not on the client, nothing appears - unless the previous key sequence was a dead key on the client.[^ws] So Screen Sharing may not be the best role model here.
[^ws]: I'd look at it in Wireshark to try and get more details on what it's actually doing, but I don't see any way to get Wireshark to look at encrypted VNC traffic, nor any equivalent to SSLKEYLOGFILE to get the keys to let it do so.
Thank you so much for looking into this!
Edit: Just saw that AVNC has support for 'raw key events'. Don't see anything about it in the settings, so I'm guessing that it just automatically turns on if the server says to use it - and that Apple's server doesn't implement that extension.
gujjwal00
commented
1 year ago Just to be clear, what I was expecting was that if I pressed Alt-E in AVNC, macOS would see:
* Option pressed * E pressed * E released * Option released
That's what AVNC does! I use xev on Linux to see generated events, so if something similar is available on macOS, you can check it yourself (sometimes VNC servers inject artificial Shift or Caps Lock).
and do whatever that would do on a keyboard attached directly to that Mac, whether that meant a dead key, a special character, or even a keyboard shortcut in the current app2. But that may not be a reasonable expectation, given how VNC represents keys. (There are two different operating systems involved, with different ideas about how keyboards should work, trying to hold a conversation about keys in a third operating system's language. It's amazing that it works at all. grinning And would a typical user consider it 'reasonable' for keys to suddenly work differently just because they were using a VNC app? What's the 'reasonable' thing to do if the client is using QWERTY and the server is set to Dvorak?) And even if it's possible to implement that, it may well need to be a separate feature request from this one.
Yes, key handling is quite messy in VNC. Input is a very complex area once you start getting into details, and RFB protocol doesn't get into gory details (somewhat understandable as things were probably not as complex back then). So different client/servers end up different key handling, and then we end up implementing workarounds, which further changes client/server behavior (and this is how I ended up adding Send legacy key events).
Edit: Just saw that AVNC has support for 'raw key events'. Don't see anything about it in the settings, so I'm guessing that it just automatically turns on if the server says to use it - and that Apple's server doesn't implement that extension.
Yes, It will be automatically used when available. But the main issue here is that even if server support raw key events, it won't be used for keys entered on a software keyboard. It needs scan codes to work, and scan codes are only available for key events coming for a hardware keyboard. Events from software keyboards simply contains the character you touched. (And accents are still composed on client-side for now.)
I'd look at it in Wireshark to try and get more details on what it's actually doing, but I don't see any way to get Wireshark to look at encrypted VNC traffic, nor any equivalent to
SSLKEYLOGFILEto get the keys to let it do so
I don't know if macOS supports normal connection, but you can force AVNC to use non-encrypted connection by setting Security to VncAuth in Advanced server options. Even then you may have to try multiple times because VNC parser in Wireshark tends to be thrown off by unknown encodings.
You can also use the Settings => Tools => Key test in AVNC to see what events are actually generated by Android.
Rhys-T
commented
1 year ago First off, it looks like the problem with Alt-Z = Ω not working may actually have been macOS's fault - when I connect from one Mac to another using Screen Sharing, I can't get Option-Z to produce a character at all. So there may be nothing for AVNC to do about that one.
[Even] if server support raw key events, it won't be used for keys entered on a software keyboard. It needs scan codes to work, and scan codes are only available for key events coming for a hardware keyboard.
Ah, okay - that's even more raw than I was thinking! I was picturing some sort of "not hardware specific, but still referring to a key position rather than a character" keycodes, like USB HID usage values, or macOS's 'virtual keycodes' kVK_*, or the code property on a JavaScript keydown/keyup event. That's what the Mac-to-Mac case seemed to be doing, when it worked - if I set the server to Dvorak, the keys would act like Dvorak, even though the client Mac was still set to QWERTY (and vice versa). But VNC doesn't seem to have anything like that, at least with any of the extensions documented here. I'm not really sure how it's being implemented, unless it's abusing keysyms to mean positions rather than characters, or something like that…
[That sequence of events is] what AVNC does!
…I didn't see the E key highlight in the keyboard viewer when I was testing, so I assumed there was no event for it. But it turns out it's never highlighting non-modifiers - the keypress is just too fast (at least with a virtual keyboard that doesn't let AVNC see the 'down' and 'up' events separately) - and I just didn't think to watch it when I was trying more 'normal' keys. D'oh!
However, I'm still not quite understanding the sequences of events being generated in some of these cases…
I use
xevon Linux to see generated events, so if something similar is available on macOS, you can check it yourself (sometimes VNC servers inject artificial Shift or Caps Lock).
I can run xev on macOS under Xquartz, and I get some very weird results - but I think part of that is because Xquartz is adding yet another layer of keyboard input translation to confuse things. I tried the Karabiner-EventViewer app that comes with Karabiner-Elements, but it doesn't show any key events from VNC, so I'm now testing using a simple event tap in Hammerspoon.
I'm using the build of AVNC from above, and testing the following key sequences:
Alt-E E ('◌́' dead key, then 'e', to select 'é', on both Android and macOS)
Here's what I'm seeing. At this point I'm not sure what's being caused by Android vs Hacker's Keyboard vs AVNC vs the protocol itself vs macOS.
This seems to be kind of spiraling out of control, so feel free to close it if you decide it's more trouble than it's worth. I'm really not trying to be a nuisance about this! The changes you made in that build still make the Option key significantly more usable than it was.
gujjwal00
commented
1 year ago This seems to be kind of spiraling out of control, so feel free to close it if you decide it's more trouble than it's worth. I'm really not trying to be a nuisance about this! The changes you made in that build still make the Option key significantly more usable than it was.
You are right, lets focus on the Option key for this issue. You have done a lot of testing, and provided a lot of useful details. I will look more into this if/when I can access a Mac device.
For now, please test the following APK: app-debug.zip
It applies the Meta-key workaround automatically when RFB 003.889 is detected.
Ah, okay - that's even more raw than I was thinking! I was picturing some sort of "not hardware specific, but still referring to a key position rather than a character" keycodes, like USB HID usage values, or macOS's 'virtual keycodes'
kVK_*, or thecodeproperty on a JavaScriptkeydown/keyupevent. That's what the Mac-to-Mac case seemed to be doing, when it worked - if I set the server to Dvorak, the keys would act like Dvorak, even though the client Mac was still set to QWERTY (and vice versa). But VNC doesn't seem to have anything like that, at least with any of the extensions documented here. I'm not really sure how it's being implemented, unless it's abusing keysyms to mean positions rather than characters, or something like that…
Actually, Android doesn't even say what values are returned by [getScanCode()](https://developer.android.com/reference/android/view/KeyEvent.html#getScanCode()). But after testing and reading through Android source code, I found that this function returns Linux kernel key codes. VNC protocol uses slightly modified XT scan codes, so AVNC uses a table to translate Linux key codes to XT key codes and sends them to server. Its possible that Mac viewer is using some special extension.
Using Alt from virtual key bar is unlikely to work for accents. Hacker's Keyboard has no idea that Alt is pressed, so it doesn't generate special events to indicate dead keys. It will generate simple events for character C/Z/E.
Rhys-T
commented
1 year ago You are right, lets focus on the Option key for this issue. You have done a lot of testing, and provided a lot of useful details. I will look more into this if/when I can access a Mac device.
Sounds good.
For now, please test the following APK: app-debug.zip It applies the Meta-key workaround automatically when
RFB 003.889is detected.
Just tried it, and it seems to work correctly. With the Mac, Alt becomes Option and Win/Meta/Super becomes Command. With Termux+TigerVNC, they show up as Alt_L and Super_L respectively, according to xev -event keyboard, and add 0x8 (Mod1Mask) and 0x40 (Mod4Mask) respectively to the state of subsequent key events.
Its possible that Mac viewer is using some special extension.
Could be - like I said, I can't tell anything useful from Wireshark, and I'm not qualified to try and reverse-engineer Apple's implementations, so I can't say for sure. I was only guessing that it wasn't an extension because I'd expect an extension to not get confused that way when the two Macs disagreed on which keys were dead.
Using Alt from virtual key bar is unlikely to work for accents. Hacker's Keyboard has no idea that Alt is pressed, so it doesn't generate special events to indicate dead keys. It will generate simple events for character C/Z/E.
I would have expected it to generate events for that key, and have macOS - which can see that Alt key being pressed - handle decisions about which character to type, if any. But I freely admit I'm not an expert on any of this, especially on the Android side, and it sounds like VNC (and possibly Android) just doesn't have the language to express what I'm looking for (at least without Apple extensions that nobody else seems to have reverse-engineered), so if it can't be done that way, I'm willing to accept that.
If all else fails, I can still use macOS's Keyboard Viewer through the 'mouse' part of AVNC to hit the problematic key sequences.
Thanks for your help!
gujjwal00
commented
1 year ago I would have expected it to generate events for that key, and have macOS - which can see that Alt key being pressed - handle decisions about which character to type
This is the core issue, resulting from using 'key symbols'. Servers often ignore previous modifier keys if the second character is not as expected. There is no explicit specification in VNC protocol for how 'key-combinations' should be handled. When I was implementing input handling, I tested multiple servers. Almost all of them had subtle differences (sometimes across versions of same server) in how they would interpret key events sent by AVNC.
As an example, lets say you press Shift+A. This can be sent in few ways:
Shift followed by aShift followed by AAI have found the 2nd option to be most reliable (also, think about pressing A when Caps Lock is ON). You can see multiple workarounds implemented by AVNC to achieve some sensible result: 1, 2.
BTW, v2.2.2 is now available with fix for Option key.
Rhys-T
commented
1 year ago I forgot to go back and test this once v2.2.2 reached F-Droid, but it seems to be working there now. Thanks!
When I connect to a macOS server (Screen Sharing/Apple Remote Desktop), Alt and ⊞ Windows/Meta/❖ Super/whatever currently both get mapped to the ⌘ Command key, and there is no way to press the ⌥ Option key. This happens whether I use AVNC's own virtual keys, or the modifier keys in Hacker's Keyboard. (I haven't tried it with a physical keyboard yet.)
It seems to be a problem on the macOS end - the same thing happens with both MultiVNC and bVNC. Remmina also has the same issue, but that thread has instructions for getting it working by customizing the key mapping settings.[^forgot] I'm not familiar enough with how the VNC/RFB protocol handles modifier keys to understand what exactly that map is saying[^whynotswap], but that at least shows it's possible for non-Apple clients to hit both keys.
Would it be possible to add this mapping to AVNC, either as a setting under "Key mappings" or automatically whenever it detects that it's connected to an Apple server?
I don't necessarily need full arbitrary modifier remapping like Remmina has - just the ability to make those two keys work for Macs would be enough. If it's implemented as a setting, I'd like to be able to change it for individual servers, since not everything I'm connecting to is a Mac. (If you'd prefer, I can split off "Allow overriding key mappings per server" as its own feature request.)
As far as which key should be which… the standard mapping seems to be Win/Super → Command, Alt → Option. That would probably have the best chance of making the keys work 'normally', if the keyboard being used was an Apple one. But I don't really have a strong preference either way. (When I use a PC-style keyboard on a Mac, I usually map them the other way around, in order to keep Command and Option in the same physical positions as on an Apple keyboard.)
[^forgot]: I had actually run into that issue and fix before, but it's been long enough since I used Remmina that I forgot about it. [^whynotswap]: Or why the default configuration somehow makes them both Command, rather than just swapping them, yet still looks like two separate modifier keys to Linux servers.