Non-US keyboard layouts

[ditron]

A nice feature for me and all others with non-US keyboards is to add a keyboard layout option list and/or trying to auto detect locale settings from the browser.

[kanaka]

That's a great idea. Except I have no experience in that area and no easy way to validate it even if I did implement it. Any examples of good implementations? Or even better would be working patches :-)

[ghost]

http://guacamole.svn.sourceforge.net/viewvc/guacamole/trunk/web/javascript/

[kanaka]

A while ago Colin Dean of xvpsource.org sent me a patch that attempted to add Non-US keyboard layout support.

Here is the original patch: https://gist.github.com/763752

Here is the patch with some modifications I made to it: https://gist.github.com/2e7f20be852a4527df63

The patch switches from using keydown and keyup events to using the keypress event to get the charCode value which is then mapped through a Unicode table to get the keysym value.

Unfortunately, after playing around with it I discovered a couple of issues that prevent it from being a solution for noVNC:

• Using the keypress event means that we can't distinguish and send separate keydown and keyup events. This means the key events sent to the server don't truely reflect what the user is doing on the client side.
• To receive the keypress event we have to allow the keydown event to propagate normally. If we stop the keydown event, the keypress event will never fire. However, stopping the event at keypress time is too late to prevent certain browser hot-key actions from firing. For example, if the user presses Ctrl-A and we don't stop the keydown event from propagating, then everything on the page is selected. This is an annoyance, but there are other cases that are worse such as Ctrl-W.

The implication is that in order to do the right thing we will have to determine the right keysym from the keyCode of the keydown event and not rely on the charCode of the keypress event.

Most of Colin's patch probably won't be used (although the u2x11 script probably will be), but it was invaluable to me to figure out what really needs to be done.

[kanaka]

Some useful links:

• quirksmode writeup of keystroke issues: http://www.quirksmode.org/js/keys.html
• quirksmode page on events in general: http://www.quirksmode.org/dom/w3c_events.html
• more cross-browser info mentioned by that quirksmode page: http://www.hallvord.com/opera/keyevents.htm
• detecting shift/ctrl/alt keys: http://www.cross-browser.com/x/examples/shift_mode.php
• more on detecting modifier keys: http://www.javascripter.net/faq/ctrl_alt.htm
• still more on modifiers with specific info for older browsers: http://www.webreference.com/js/column11/modifierkeys.html

It sounds like the altKey, shiftKey and ctrlKey properties of the key events may be fairly cross-browser (in the browser versions we care about) which is very good if true.

[kanaka]

Also, I added tests/keyboard.html which prints keyboard event data (keyCode, charCode, which, altKey, shiftKey, ctrlKey, translated keysym).

[kanaka]

One thought I've had for how to generate keyboard layout data is that we may be able to create a page that generates fake keyboard events to determine what the keycode+modifiers to charCode data is (i.e. generate keydown event and trap the keypress event),

It's possible this is fast enough (and reliable enough) in all browsers we need to support that we might be able to do this on page load in noVNC. But at least we could have a publicly accessible page where people could generate and then submit their layout data.

But at the very least we might be able to use the technique with just a few key events to binary search for the keyboard layout that the user is using. Assuming there is no better way to determine that (I haven't searched in detail for how JavaScript can determine keyboard layout).

[ghost]

i would do both: a. use key sequences to guess the keyboard b. let them select

[moobyfr]

if the keyboard layout could be configurable, it could resolve many problems... for non-us keyboard, it can be very hard to type

[moobyfr]

I tried to integrate the patch from Colin Dean, as noVNC is mostly no usuable for typing text. The git is https://github.com/moobyfr/noVNC Fell free to comment if it works for you. I tested on french layout, and is ok for normal keys + modifiers. The only issue is AltGr key

[kanaka]

moobyfr, the change in your fork is not actually functional. You have only added the x11.js file without using it. Perhaps you forget to push the rest of the changes.

Also, as I mentioned in the 3rd comment there are two major issues with Colin's approach: the keydown and keyup messages are no longer detected and sent separately, unintended browser events are no longer disabled (e.g. Ctrl-W will close the browser tab rather than sending a Ctrl-W).

I am very willing to accept additional keyboard layouts, however the layout must key off off keydown and keyup events and not key off of the keypress event as with Colin's patch. If you would like to develop a French layout that keys off of keydown/keyup then I would gladly pull it into the project.

[moobyfr]

I've spent some time to look for the way to go. The keydown/keyup is the way to intercept ~all keys. But there is something bad in the management with javascript for key events. keyCode contain a key reference for a qwerty keyboard, and the real char is in CharCode. So to be realistic, noVNC should transmit keyCode to say which key was pressed. The basic idea is, as you said , and I tried, is to map keyCode to other values depending to the key pressed. But layout are more than just keys moved: on a azerty keyboard, two differents key can return the same value, and there is no way to distinguish them only with keydown/keyup: Example, the "-" and "_" on a qwerty is on the same key, and differentied with the shift modifier. the keyCode value is 189. On a azerty layout, these two symbols are on two differents keys. the keyCode returned by the browser for each key is 189! So only with the keyCode (and modifiers), it's not possible to make a table to manage differents layout. That's why the charCode has to be used. I've looked in guacamole code, the same is made: trap keydown to know that a key has been pressed, and use keypress, if needed, to get the real charCode to send.

Javascript doesn't give us all the power :/

[kanaka]

noVNC has to use keyCode and not charCode for two reasons: first so that we can send the keydown and keyup separately (for some keys on some layouts we may have to work around this due to keyCode limitations as you describe), but the second and bigger issue is that using charCode requires the key event to propagate/bubble which means that certain key sequences are seen by the browser and cause unintended behavior such as tab, space, enter, Ctrl-W, Ctrl-Tab, etc. If Guacamole is using charCode then they have this problem.

Here is a concrete example of the second problem. On most browsers, Ctrl-Minus will shrink the font size and Ctrl-Plus will increase the font size. In order to prevent this from happening in noVNC (and just send the Ctrl-Minus/Plus on to the server), we stop the keyDown event and don't allow it to propagate. However, another effect of stopping the event is that there is no keyPress event generated. I don't know of any way to both generate a keyPress event and prevent bubbling into default browser actions.

That is really terrible that on certain keyboard layouts multiple keys generate the same keyCode in Javascript.

If anybody has a sudden stroke of brilliance about how to solve the keyboard layout problems without bad side-effects, I'm all ears. In the meantime, we may just have to provide a way (button) for the user to send a '_' character when using an azerty keyboard.

Do we know of other keys that have the same overlapping key layout abomination?

[kanaka]

I may ask this question on stackoverflow.

[kanaka]

I've asked the question on stackoverflow here: http://stackoverflow.com/questions/5306132/translate-javascript-keycode-into-charcode-for-non-u-s-keyboard-layout-i-e-aze

[moobyfr]

more informations: 3x 2 keys have same keyCode: ";" and ":" are differents keys, and "!" and shift + "&" Some more fun: 6 other keys send a 0 value for keyCode: (in the qwerty mapping for having a better idea of the mapping) : the key 2 7 9 0 and the two keys right of "p" and "m" And the best: the key 4 (" ' " on a azerty ) only show keypress events, and no up/down!!

I have another proposition: add an option to noVNC to use the up/down event only for qwerty users, or the keypress for international users with less functionalities...

[kanaka]

moobyfr, thanks for your work on this. It's a really messy area and I just use a standard U.S. keyboard layout so it's difficult for me to determine exactly where the issues lie.

It looks like it's going to have to be something like a fallback for non-US layouts. Although, again, the biggest issue is not getting the mapping correct, the biggest problem is preventing unintended default browser behaviors. So it's likely that keypress will only be used for non-US keys when no modifier key (other than shift) is pressed at the same time. When a modifier (other than shift) is pressed we still can't use keypress on any key layout.

[kanaka]

Okay, it's a huge challenge but I think I've made some forward progress here:

https://github.com/kanaka/noVNC/commit/c96f900336b57401a7e2d1678f80cf88662ab566

In summary:

• I've added a keyPress handler and I report a key down for normal (safe) key combinations when the keyPress event happens.
• For special (browser action) key combinations I report the key down in the keyDown event and suppress further action.
• In order to report the key up properly I add a key event to a list in keyDown, give that element a keysym value in keyPress, and then pop that item off the list and use the keysym value during keyUp to report the key up.
• I've also incorporated Colin Deans character code to keysym lookup table for values above 255.

This isn't a 100% solution since key combinations that cause browser events (tab, enter, backspace, Ctrl-, Alt-, etc) are not translated before being sent, but for normal typing we should be able to get keyboard mapping correct now.

I'm going to leave the bug open for a while since I have not been able to test this much on alternate keyboards so I'm sure there are issues, but I think the framework is in place to be able to solve most of the issues.

[kanaka]

Also, see this commit and the comments with it: https://github.com/moobyfr/noVNC/commit/c0615b935cc1e0e5cc9be222405185bbf829294a

[kanaka]

I'm closing this issue. I don't have a great way of testing this myself. I can switch keyboard layouts and poke around a bit and things seem to be working fairly well, but I'm not a real user of non-US layouts so I'm relying on community feedback. If anybody listening uses non-US keyboards and is having issues, please re-open. Thanks.

[tenchman]

I'm having problems here with a german keyboard layout and at least the backslash key. Instead of a backslash noVNC sends a left bracket [.

The codes send with different Browsers:

• IE9: FFE3, FFE9, 005B down and 005B, FFE3, FFE9 up
• FF4: FFE3, FFE9, 005B, 005C down and 005C, FFE3 up
• Opera 11.11: FFE3, FFE9, 00DF, 005C down and 005C, FFE3 up

Unfortunately i have not the permissions to reopen this issue.

[kanaka]

@tenchman, I've reopened the bug. I can't seem to replicate your problem on Linux (Ubuntu Maverick) with Chrome 12, firefox 4.0.1, or Opera 11.11. I'll have to try with browsers under Windows at some point (I assume that is where you are seeing this?)

Just to make sure, is this the keyboard layout you are using? http://en.wikipedia.org/wiki/Keyboard_layout#Germany_and_Austria_.28but_not_Switzerland.29 And are you pressing Alt_Gr and then the key two over from the backspace?

[moobyfr]

the current version of novnc can't handle correctly AltGr key. In USA, right ctrl + alt can do the same, but these keys don't exist on european keyboards

[kanaka]

@moobyfr: hmm, interesting. I only have a US keyboard and have been switching layouts to test. Can you see what key code AltGr generates on a real French keyboard? @tenchman, if you press and release just AltGr on a German keyboard, what does noVNC report?

[tenchman]

Yes, this is exactly the keyboard i'm referring to and yes, it's a Windows issue. Same keyboard on Linux with FF, Opera, Chrome works like a charm. The same key sends 0x5c under Linux and 0x5b when pressed under Windows. (Note the funny sequence for FF4 under Windows in my previous comment)

This is what noVNC reports when pressing only AltGr

IE9

onKeyDown kC: 17 cC:undefined w:undefined keyPress down 1, keysym: 65507 (key: 17, which: undefined) onKeyDown kC: 18 cC:undefined w:undefined keyPress down 1, keysym: 65513 (key: 18, which: undefined) onKeyUp kC: 17 cC:undefined w:undefined keyPress down 1, keysym: 65507 (key: 17, which: undefined) onKeyUp kC: 18 cC:undefined w:undefined keyPress down 1, keysym: 65513 (key: 18, which: undefined)

Firefox and Opera

onKeyDown kC: 17 cC:undefined w:undefined keyPress down 1, keysym: 65507 (key: 17, which: undefined) onKeyDown kC: 18 cC:undefined w:undefined keyPress down 1, keysym: 65513 (key: 18, which: undefined) onKeyUp kC: 17 cC:undefined w:undefined keyPress down 1, keysym: 65507 (key: 17, which: undefined)

[moobyfr]

I used :http://kanaka.github.com/noVNC/noVNC/tests/keyboard.html on firefox and chrome, there are events but all 0 ....

2: Dn: key:0 char:0 which:0 id:undefined ksym:0 alt:false shift:false ctrl:false 3: Pr: key:0 char:0 which:0 id:undefined ksym:0 alt:false shift:false ctrl:false 4: Up: key:0 char:0 which:0 id:undefined ksym:0 alt:false shift:false ctrl:false

On chrome there is no KeyPress Even notified

[moobyfr]

I was not precise in latest comment: in chrome there is at least the keyDown and keyUp event, with 0 as value. AltGr is a modifier, so just relying on up/down is enough, I made a patch and a pull request to integrate it .

[neufeind]

Hmm, any chance to get this keymap-thing resolved? We need non-us-layouts, really :-)

[kanaka]

@neufeind, I would LOVE to resolve this. However, as I noted, I don't use an international layout and the issue is tricky. The current solution is better that it was originally, but there are still problems (for example with the AltGr key). If somebody steps up and provides a really good fix then they will earn eternal glory (well, at least a mention in the credits).

I think the ultimate solution may be to exactly characterize the problem and report bugs to the various browser vendors because I think there is no good solution without the browser providing more accurate/precise information during key events. I don't have the time, or knowledge/access to international keyboard layouts so this will need to be driven by somebody else (although I will certainly give assistance to whoever steps up).

[moobyfr]

while I was looking for the bugreport about altGr key in Chromium, I found the following bug report in guacamole, which handle the keys through javascript:

http://guac-dev.org/trac/ticket/136?cversion=0&cnum_hist=9#comment:11

The comment 11 explain the processing of keydown before keypress, and these keypress are ignored. An workaround has been implemented too in theirs client.

BTW, firefox 16a1 now report a real value (225) for the AltGr Key! (keyIdentifier is still 0)

[kanaka]

Interesting technique. noVNC does try and use keyPress rather than keyDown when it can but using a 0 timeout would allow it to be more aggressive about that. Anybody want to take a crack at it. Especially someone with an internal layout that can reproduce the problem?

[moobyfr]

I made some more tests and looked the code again: the code is already using this method keyPress preferred over keyDown. In fact it is working, but it depend of the vnc server :/

The key "à" has keysym 224 and AltGr "à" gives me the char @ which has keysym 64 -> I press "AltGr" Key and the Key "à", it gives me the char @ (keysym 64)

• If I use vnc4server I get the input @ on my xterm.
• If I use the buildin VNC server in qemu, I get the char "à"

I had a look in the vnc code from qemu: when a keysym comes in, it's converted in a name which is converted to a scancode.

To find out the scancode the way is keysym 64 -> "at" (@) . The "at" is looked into the fr keymap, wich says :

agrave 0x0b 0 0x0b shift at 0x0b altgr

As no modifier was sent, the scancode for agrave is used.

[kidk]

I'm also having a lot of trouble with this. I'll try to put in some debugging time tomorrow to find which characters aren't working. I'm using a azerty belgian keyboard.

Weird thing is that when I try to setup ubuntu with the belgian keyboard settings and type "azerty" it comes uit "qwerty". When I use US keyboard settings it comes out 'azerty", don't know if this is a problem with noVNC. But I'll try to find out.

[moobyfr]

in the case of a belgian layout, there should be no problem with at least with the main ascii characters accessible with only one keypress

Emmanuel

IUT R.Schuman Université de Strasbourg

[kidk]

My mistake, seems to be a problem with the vnc server.

[jalfd]

Pressing and then releasing AltGr (using Danish keyboard layout, but I'd assume it's the same for any layout where AltGr exists), Firefox (16.0.2, Windows) reports the following:

onKeyPress down, keysym: 65507 (onKeyDown key: 17, which: 17) onKeyPress down, keysym: 65513 (onKeyDown key: 18, which: 18) onKeyPress up, keysym: 65507 (onKeyPress key: 17, which: 17)

which, if I'm reading your code correctly, means that it registers keypresses on ctrl and alt, and then releasing only ctrl. The behavior we're seeing is pretty much that: it's as if the alt key sticks.

Other browsers release both keys properly when AltGr is released.

Fun stuff. I must say, after looking at input.js, I'm impressed that you've got it working as well as you have. This is the only keyboard issue we've come across so far.

[moobyfr]

Are you using a Firefox on Windows ? In Linux version, the AltGr key has it's own code. In windows it seems that AltGr is handled by simulating alt and ctrl :/

On Tue, Nov 6, 2012 at 10:01 AM, jalfd notifications@github.com wrote:

Pressing and then releasing AltGr (using Danish keyboard layout, but I'd assume it's the same for any layout where AltGr exists), Firefox (16.0.2) reports the following:

onKeyPress down, keysym: 65507 (onKeyDown key: 17, which: 17) onKeyPress down, keysym: 65513 (onKeyDown key: 18, which: 18) onKeyPress up, keysym: 65507 (onKeyPress key: 17, which: 17)

which, if I'm reading your code correctly, means that it registers keypresses on ctrl and alt, and then releasing only ctrl. The behavior we're seeing is pretty much that: it's as if the alt key sticks.

Other browsers release both keys properly when AltGr is released.

Fun stuff. I must say, after looking at input.js, I'm impressed that you've got it working as well as you have. This is the only keyboard issue we've come across so far.

— Reply to this email directly or view it on GitHubhttps://github.com/kanaka/noVNC/issues/21#issuecomment-10103412.

[takaha]

I have a noVNC that can handle emulated AltGr key events on any browsers including Firefox for windows in my sandbox. I can upload it here if you want.

[jalfd]

@moobyfr: yes, sorry, this is on Windows. I should've mentinoed that. (And just edited my comment)

[kanaka]

@takaha BTW, I commented on your pull request with some suggested changes before I merge it in.

@jalfd seems like a possible bug in Firefox. If you have time to create a simple test case, you might consider doing that and filing a bug with Mozilla on this. Make sure to report back here if you do so (test case and bug link). Of course, in creating a simple test case you might also learn something noVNC could be doing to work around it.

[takaha]

I uploaded a snapshot of my sandbox: https://github.com/kanaka/noVNC/pull/212

This supports non-us keyboard layouts. Japanese and European keyboards will work with noVNC on all popular browsers. I hope non-us keyboard users will give me feedback.

Thanks.

[jalfd]

Yeah, it's definitely a bug in Firefox, and I filed a bug here: https://bugzilla.mozilla.org/show_bug.cgi?id=809410 -- no news on it yet, however.

However, I played around a bit more, and found that there are a few hints that might be used to detect the situation. The location field is actually set, so the precise key sequence reported by Firefox is "left CTRL down, right ALT down, left CTRL up". That can at least help distinguish from the more common case where you actually press left CTRL and ALT. Further, the timestamp is set to the exact same value for the CTRL and ALT keypresses (which would never happen if they were pressed individually) Anyway, I've been a bit stretched for time lately, so that's all I have for now. :) I'll try to play around with it some more when I have a bit of time :)

[laggarcia]

From my latest patches with noVNC, I think this is still an issue... I am basically using an English (US international with dead keys) keyboard and seems that the accents are not being correctly handled. Is there any update on this? I might have some ideas but I would like to see what is the current status in case there was any updates on the last 1.5 year.

[jalfd]

There has been a lot of progress. Keyboard handling has been completely rewritten, and I'm not currently aware of any bugs that aren't due to browser limitations at the moment.

What specifically doesn't work for you? Which browser and OS are you using?

[laggarcia]

I am using noVNC to access a guest running on top of QEMU. QEMU is the VNC server in this case. I have a Fedora 20 machine running Gnome 3 and Firefox 31. My machine is configured to use "English (US international with dead keys)" keyboard. When I connect to the VNC session through noVNC, dead keys doesn't work, which means I cannot type things like 'ê' or 'ç'. I heard some people having similar issues with French keyboard layouts (although I don't have the details). I tried to use Chinese input method as well without much success.

[ruffy91]

I am using noVNC to connect to my proxmox VMs. I cannot use keys like /, {, }, and | as they get interpreted as other keys. The machine I'm connecting to is using the USA layout and is running Debian 6. The machine I'm using noVNC on is using Windows 8.1 with swiss german layout.

[DirectXMan12]

What keys do /, {, }, etc register as? There's a lot of interesting logic that goes into how noVNC interprets keypresses (see https://github.com/kanaka/noVNC/wiki/KeyEvents for more information). If you open up "input.html", what key presses are displayed?

cc @jalfd

[thardeck]

@ruffy91 It is probably the AltGr key issue again as reported at #406. You can work around it in noVNC but it seems to be mainly an issue of the keyboard conversion of the VM.

[jalfd]

@laggarcia Sorry, I didn't notice your response until now. it sounds like you're running into issue #350. Currently, Firefox on Linux is borked when it comes to dead keys. (Firefox bug 1077411)

@ruffy91 as DirectXMan12 said, we need a bit more information before we can look into this. Try launching input.html in your browser and see how the relevant keys are processed, and let us know the result.

[clnperez]

To use the input.html page, do I need to do anything special befhorehand? I'm seeing (in Chrome and Firefox on Fedora 20) a page with two boxes, but the Canvas input box won't accept input.

[jalfd]

Huh, looks like someone broke it!

You're not supposed to do anything special, just load the page, and press various keys on the keyboard. But yeah, I get javascript errors when I try to load it now. I'll try to take a look one of the next couple of days.