Request: raw keyboard events

[MagicalTux]

A widget should be able to capture raw keyboard events, including keydown/keyup events.

For example the GameBoy emulator at github.com/andydotxyz/fynegameboy will check if a key repeats within 200ms to know if it's being held, which is not a good thing.

Instead there should be a way to access raw events where the Device allows it, and this process should be nothing more than a fallback.

[andydotxyz]

Apologies but the code you linked is not the Fyne GUI but a telnet adapter. See instead https://github.com/andydotxyz/fynegameboy/blob/2d31152a860878e3ae63ba2b8929ed94e36650a1/fyne/driver.go#L266 where the key up and key down events are used to get the hardware events you describe.

[MagicalTux]

Great this is working. Now I just need to create a table to convert the fyne keycode name back to a scancode.

Some keys are not supported on 105 keys European keyboard or 109 keys Japanese keyboards, any chance the raw scancode could simply be added to the event and it be triggered even if the scancode wasn't found in the name table?

[andydotxyz]

We don't want to expose the scancode as that is so hardware dependent - there are a bunch of tables inside the GLFW driver (internal/driver/glfw/) that converts codes / names to named keys in the Fyne API. Are we missing the key names, or are we not managing to look them up correctly?

[MagicalTux]

I think some of the key names are missing, but also the keypad keys and keyboard keys are mixed together and impossible to tell apart (for example French keyboards require to use Shift to access digits on the keyboard side).

Maybe there is a way to access directly glfw when on desktop and bypass fyne?

[andydotxyz]

I'm not quite sure why getting access to the keyboard scancodes is more desirable than fixing the Fyne code. I would like to understand this more so we can properly fix the issue. We want to avoid exposing scancodes as much as possible because the hardware aspect is only relevant on desktop apps - but most apps just want to know if "7" was pressed, or the Enter key, for example - where scancodes would be a distraction as both can be triggered by 2 or 3 different codes...

[MagicalTux]

Yes, it's OK for most cases to just know 7 was pressed, as long as you're in the US or UK. Most European countries do not use the same keymap and keys which are equal on us keymaps aren't equal anymore.

The missing keys are (lowercase followed by uppercase if any):

• ²
• )°
• $£
• !§
• :/

I do not know the appropriate scancodes or values in glfw for those keys however.

Adding the missing keys is fine, but I do not want to change fyne's behavior in terms of handling digits as some applications may be relying on this, hence a way to access glfw directly might be more suitable in my case, and others in the future may want to hack further than what's possible.

For example a Canvas can be tested & instanciated as a desktop.Canvas, is there anything similar for keyboard?

[andydotxyz]

Most European countries do not use the same keymap and keys which are equal on us keymaps aren't equal anymore.

Indeed, this is why we want to not have developers worry about scancodes and keymaps. Our APIs are focused on the user intent rather than the implementation where possible :).

Adding the missing keys is fine, but I do not want to change fyne's behavior in terms of handling digits

I don't think that adding new key mappings will break behaviour.

[MagicalTux]

I don't think that adding new key mappings will break behaviour.

I fully agree, however this is more complex than it looks like.

For example on my keyboard (French) when I press & I receive a Key1 KeyEvent. That's because & is the lowercase of 1 on the keyboard side. However if I press A I do indeed get KeyA. The behavior as it is today is not consistent and it's difficult to make something that will both work for games (need to handle keys pressed separately from modifiers as these typically can trigger things, such as run, walk, crouch, etc) while being friendly enough (a US developer will set KeyA to strafe left by default, but on a azerty keyboard that key won't be at the right place and it'll mess up the whole thing).

User intent can be different between regular typing (I want to press A) or gaming (I want to press the key left of S). I grew in France and spent years dealing with games where I had to press W to move forward (W and Z being inverted on French keyboards). You get used to it.

Another example: FyneGameboy has hardcoded the X and Z keys for A/B. On my keyboard the key next to X is W, the Z key is at the left of E on the top row. This is extremely awkward to use for anyone with a non-qwerty keyboard and non configurable in the case of FyneGameboy.

Now, providing the scancode can be a pain. It could be XT, AT or USB, or whatever appears in the future. Maybe it could all be converted to a single standard (fyne standard, even?) or anything. Having in the KeyEvent both the logical (character or anything) and physical information of the key would make sense, a lot of frameworks out there do that. The developer can then choose to use whatever makes sense for him.

It also helps avoiding breaking the current behavior.

[andydotxyz]

You're certainly right this is a complex issue :)

Now, providing the scancode can be a pain. It could be XT, AT or USB, or whatever appears in the future. Maybe it could all be converted to a single standard (fyne standard, even?) or anything.

I guess that's what I was considering as well. I know that many toolkits expose the hardware side of this but I don't think many toolkits have succeeding in abstracting correctly the hardware details so apps can be built once and run correctly on all platforms. I don't know the right way to get the best of all worlds.

Another challenge we have is shortcuts - we have recently discovered that keyboards that have certain character combinations will not correctly map to (for example) ctrl-A because their "A" key is not actually an A, but on some OS they map the different letter to certain keys for internationalisation workarounds.

[MagicalTux]

Another option for multiplatform UI is Flutter by Google. Their use for raw keyboard events on all platform rely on USB scancodes (see flutter PhysicalKeyboardKey class documentation).

I don't think it matters much which standard is used however as long as constants are made available.

In the meantime if there is a way to access glfw/gomobile directly via Fyne (some interface{} thing somewhere? or maybe I can instantiate these directly? To be quite honest I haven't looked into it yet since I've been working on other parts but I'm getting closer to needing this) I may be able to get more information, and implement something that works and can be used to then improve fyne - or not if it's too specific, but at least it'll allow me to move forward.

[andydotxyz]

Good illustration, thanks. I think this is what fyne.Key is attempting to do - refer to the source of an event be it physical, software or simulated. The result of key presses is the TypedRune and TypedKey (the first is for visible chars, the second for other keys) so we already have an abstraction built in. If physical keys are missing from the list we should add them. Most are in the main package, but some that appear only on desktop (such as "Super") are in the driver/desktop package.

[andydotxyz]

ScanCode now added, we can discuss next how a KeyPosition field or PositionName() function could add a further "abstraction".