altGraph misinterpreted on Swiss German keyboard

db39 commented 1 year ago

Description

TinyPilot displays the correct input in the key history, but the actual characters typed on the machine are incorrect. i.e. altGr + q should input @. Forum thread for reference.

Possibly related to this previous issue.

I managed to reproduce the issue on my own devices (both set to Swiss German keyboard layout).

What's the behavior that you expect?

When pressing Alt Graph + q , I expect for TinyPilot to write @.

What's happening instead?

TinyPilot instead writes q. The behaviour is identical across other keys that require Alt Graph too.

What are the steps to reproduce this behavior?

Set the client keyboard layout to Swiss German
Set the target machine's keyboard layout to Swiss German
Press right Alt / Alt Graph and then the desired key
See the correctly interpreted key in key history, but not on the target machine's screen.

Screenshots

mtlynch commented 1 year ago

@db39 - Can we use the standard bug template here? It'd be good to have the full explanation and repro steps in the bug itself rather than linking out to the forum (although it's definitely helpful to keep the forum link).

## Description

Briefly summarize the problem you're experiencing.

## What's the behavior that you expect?

_Your answer_

## What's happening instead?

_Your answer_

## What are the steps to reproduce this behavior?

1. Open TinyPilot in Chrome
2. Click on '....'
3. See error

## Screenshots

_If applicable, add screenshots to help explain your problem._

## Logs

Please paste the URL you see when you run `/opt/tinypilot/dev-scripts/dump-logs`

db39 commented 1 year ago

Quick update - using the right alt key on the on-screen keyboard works in this situation. The problem seems to be with Alt Graph from a physical keyboard.

db39 commented 1 year ago

After doing some debugging, I've found that the right alt modifier isn't being applied to the HID code when being written.

For context, here's the debug output when using right shift + 2:

hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x20 0x00 0x1f 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]
hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]
hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]

0x20 is the correctly applied right shift modifier.

And the debug output for AltGraph / right alt + 2:

hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x1f 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]
hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]
hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]

The first line doesn't have the modifier (0x40), even though the key is held, and recognized before hitting the 2 key. It should be:

hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x40 0x00 0x1f 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]
hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]
hid.write       DEBUG [SENSITIVE] writing to HID interface /dev/null: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 [/SENSITIVE]

(this is the output you get when you use TinyPilot's on-screen right-alt key + 2).

AltGraph hits keystroke.right_shift_modifier when pressed independently, but not when used in conjunction with another key.

The frontend recognizes the correct key (@ on a Swiss German layout), but altRight isn't set:

altLeft: false
altRight: false
code: "Digit2"
ctrlLeft: false
ctrlRight: false
key: "@"
metaLeft: false
metaRight: false
shiftLeft: false
shiftRight: false

At this point I've hit a bit of a roadblock and I don't know where to look next. I've looked at the keyboard code on the frontend and backend, but I can't find an obvious culprit. I'd love some guidance towards what to look at or test next.

mtlynch commented 1 year ago

@db39 - Can you post a screenshot of what you see if you type altRight + @ on the client browser into keycode.info.

TinyPilot's JS makes it seem like the browser sets altRight: false when you're pressing altRight + @:

altLeft: false
altRight: false    <<< I'd expect this to be true
code: "Digit2"
ctrlLeft: false
ctrlRight: false
key: "@"
metaLeft: false
metaRight: false
shiftLeft: false
shiftRight: false

db39 commented 1 year ago

Keycode doesn't show that any meta / modifier keys have been pressed either:

keycode

mtlynch commented 1 year ago

Yeah, this is a tough one to solve then. The browser is reporting incorrect information: it says altKey: false, but really it's true. I'm sure there's some reason why the browser has to do that, but it breaks our assumptions that the browser is giving accurate information about which modifier keys are pressed.

Out of curiosity, does this reproduce across browsers?

This is not on our short-term queue, but in the future, my strategy for fixing this would be to have special logic on a per-locale basis to override what the browser claims for certain characters (e.g., if locale is de_CH and the key is @, then we override and set leftAlt: true).

db39 commented 1 year ago

Out of curiosity, does this reproduce across browsers?

Yeah, the behaviour is identical on both Firefox and Chrome (with TinyPilot and keycode.info).

db39 commented 3 months ago

It looks like this behavior is still causing issues for users that rely on AltGr. And it isn't only for specific languages - I believe any layout that uses AltGr has this issue.

Right now, the only workaround is to switch the client machine's keyboard language/layout to English (US), or use TinyPilot's on-screen keyboard. Both of which aren't ideal.

RastaTaz commented 3 months ago

Hi, I'm the guy who experienced the issue for Fench keyboard that @db39 mentionned.

By adding following code to /opt/tinypilot/app/static/js/keycodes.js:

export function keystrokeToCanonicalCode(keystroke) {  
console.log(keystroke);  
console.log(keystroke.code);  
console.log(keystroke.key);
(...)

I can indeed confirm that the browser doesn't set "altKey" to "true", see screenshot:

I can also state the the browser send the good "key" when pressing the "6" along with AltGr (I was going for the "|" character):

Let me add that all the inputs are doubled (twice for "keydown" and twice for "keyup" , screenshot when pressing "AltGr + 6"

Don't know the latter is important or not... Regarding the fact that "altKey" is not set to "true": I can't remember where I read that it is normal/expected behaviour when using "AltGr" key: I'll try to find again it for reference...

I also tried to add to add a condition like the work arounds for Firefox in the same .js file, but without success:

  if (keystroke.key === "AltGraph") {
    return "AltRight";
  }

My assumption is that the issue may not be in the recieved keyboard event, but maybe in the event sent to the server behind the Tinypilot as I can't see any management of the "AltGraph" modifier in the rest of the .js files in /opt/tinypilot/app (js_to_hid.py & text_to_hid.py)... But I'm no developper, just a geek !

db39 commented 3 months ago

Thanks for all of those details, @RastaTaz!

Regarding the fact that "altKey" is not set to "true": I can't remember where I read that it is normal/expected behaviour when using "AltGr" key: I'll try to find again it for reference...

If you find the reference for that, that would be great!

I can't see any management of the "AltGraph" modifier

From my understanding, when you type AltGr + 6 for |, the browser sees this keycode (54):

The problem is that the value of altKey (in the event dump) is false. And TinyPilot doesn't see an AltGr key press here, just the code for |. TinyPilot would only be able to override the altKey value if it knew that | required AltGr. To do that, we'd need to add some logic for locale information.

I hope that clarifies the issue - please let me know if you have any questions.

RastaTaz commented 1 month ago

Hi guys ! Just to let you know: I had a try and installed PiKVM v3 from https://github.com/pikvm/pikvm on TinyPilot Voyager 2a hardware and:

it seems to "just work", although I didn't dig too much into it yet, but certainly will: see next point
I don't have the keyboard issue and can type this so wanted/used "|" character into the console of my linux server !
There are some interesting features not available on TinyPilot such as virtual drives (USB key & CDROM), terminal to KVM itself in the WebUI, ATX commands via an additional hardware extension wired into the ATX connectors of POWER & RESET buttons of server motherboard (very interesting !)

Don't know which framework/technology they're using (again, I'm not a dev, just a sys admin) but they pulled it off ! => PiKVM being open-source, this may give you some pointers/tips on how to achieve this on TinyPilot too I hope.

cghague commented 3 weeks ago

Thanks for the update, @RastaTaz! It's great you've found a way to use your TinyPilot device while we develop a solution for input using AltGr. It's always exciting to see other projects running on TinyPilot hardware!

TinyPilot Pro does support virtual media and can emulate both USB and CD-ROM drives. ATX interfaces are a novel feature, but we aim for a more "plug-and-play" approach, such as using console ports.

We'll keep this discussion updated as the investigation continues!

tiny-pilot / tinypilot