A keyboard input model

[pyfisch]

TLDR: I think that Winit needs more expressive keyboards events and to follow a written specification to keep platform inconsistencies to a minimum. I propose to adapt the JS KeyboardEvent for winit and to follow the UI Events specification for keyboard input.

Winit is used for many applications that need to handle different kinds of keyboard input.

• Games: Physical location of keys like WASD for movement and actions. Text inpput for names and chat.
• GUI applications: Text input and keyboard shortcuts.
• the Servo Browser: Wants to support JS KeyboardEvent well.

Currently there are two events for text input in Winit: KeyboardInput and ReceivedCharacter.

pub struct KeyboardInput {
    pub scancode: ScanCode,
    pub state: ElementState,
    pub virtual_keycode: Option<VirtualKeyCode>,
    pub modifiers: ModifiersState,
}

The KeyboardInput event carries information about keys pressed and released. scancode is a platform-dependent code identifying the physical key. virtual_keycode optionally describes the meaning of the key. It indicates ASCII letters, some punctuation and some function keys. modifiers tells if the Shift, Control, Alt and Logo keys are currently pressed.

The ReceivedCharacter event sends a single Unicode codepoint. The character can be pushed to the end of a string and if this is done for all events the user will see the text they intended to enter.

Shortcomings

This is my personal list in no particular order.

1. List of VirtualKeyCode is seen as incomplete (#71, #59). Without a given list it is hard to decide which keys to include and when the list is complete. Also it is necessary to define each virtual key code so multiple platforms will map keys to the same virtual key codes. While it probably uncontroversial that ASCII keys should be included for non-ASCII single keys found on many keyboards like é, µ, or ü it is more difficult to decide and to create an exhaustive list.
2. While VirtualKeyCode should capture the meaning of the key there are different codes for e.g. "0": Key0 and Numpad0 or LControl and RControl.
3. The ScanCode is platform dependent. Therefore apps wanting to use keys like WASD for navigation will assume an QWERTY layout instead of using the key locations.
4. It is unclear if a key is repeated or not. Some applications only want to act on the first keypress and ignore all following repeated keys. Right now these applications need to do extra tracking and are probably not correct if the keyboard focus changes while a key is held down. (#310)
5. A few useful modfiers like AltGraph and NumLock are missing.
6. There is no relation between ReceivedCharacter and KeyboardInput events. While this is not necessary for every application some (like browsers) need it and have to use ugly (and incorrect) work-arounds. (#34)
7. Dead-key handling is unspecified and IMEs (Input Method Editors) are not supported.

In general there are many issues that are platform-dependant and where it is unclear what the correct behavior is or it is not documented. Both alacritty and Servo just to name two applications have multiple issues where people mention that keyboard input does not work as expeced.

Proposed Solution

Winit is not the first software that needs to deal with keyboard input on a variety of platforms. In particular the web platform has a complete specification how keyboard events should behave which is implemented on all platforms that Winit aims to support.

While the specification talks about JS objects it can be easily ported to Rust. Some information is duplicated in KeyboardEvent for backwards compatibility but this can be omitted in Rust so Winit stays simpler.

See the keyboard-types for how keyboard events can look like in Rust.

• (shortcoming 1) VirtualKeyCode is replaced with a Key. This is an enum with all the values for functional keys and a variant for Unicode values that stores printable characters both from the whole Unicode range. Specification
• (shortcoming 2) is also adressed by this. There is just one value for keys like "Control" but if necessary one can distinguish left/right or keyboard/numpad keys by their location attribute.
• (shortcoming 3) ScanCode is complemented by Code. Codes describe physical key locations in a cross-platform way. Specification
• (shortcoming 4) a repeat attribute is added.
• (shortcoming 5) All known modifier keys are supported. **Specification Note: W3C decided to include some keys that are usually handled in hardware and don't emit keyboard events (like Fn, FnLock)
• (shortcoming 6) received characters and keyboard events are now one (exceptions see below)
• (shortcoming 7) to handle dead keys and IMEs a composition event is introduced. It describes the text that should be added at the current cursor position. Specification Note: The introduction composition events makes it a bit harder to get "just the text" which is currently emitted by ReceivedCharacter. Either ReceivedCharacter is kept around for easier use or a utility function is provided that takes keyboard and composition events and emits the printable text.

Implementation

This is obviously a breaking change so there needs to be a new release of winit and release notes. While the proposed events are very expressive it is possible to convert Winit to the new events first and then improve each backend to emit the additional information about key-codes, locations, repeating keys etc.

Thank you for writing and maintaining Winit! I hope this helps to get a discussion about keyboard input handling started and maybe some ideas or even the whole proposal is implemented in Winit.

[maroider]

The API is mostly settled, and implementation work is ongoing. I wouldn't be surprised if it takes at least another month before everything is coded, tested, and merged, although #1788 is quite usable ATM if you're only targeting Windows.

[Rua]

That's great news! I'm not on Windows so that doesn't help me, but I wonder why the PR is "for Windows". What happens if I try to use it elsewhere?

[maroider]

Your code will simply not build. The PR is "for Windows" because it changes the public API, but only implements the API on Windows.

[ArturKovacs]

Implementation for other platforms have their own respective PRs such as #1888 and #1890. Furthermore these PRs target the new-keyboard branch instead of the master branch.

[garasubo]

@ArturKovacs I'm interested in implementing this for X11 platform. But can we merge https://github.com/rust-windowing/winit/pull/1788 into new-keyboard branch first? Otherwise, it seems we need to implement the same change for the common codes, like examples directory.

[ArturKovacs]

That sounds awesome @garasubo! I know that @maroider did work on the X11 implementation but I'm not sure how far he's gotten with it. Maybe the two of you could cooperate or race :D I would really want to see this completed as soon as possible.

1788 is currently waiting to get approved by @msiglreith. But you don't need to wait for that to get merged into the new-keyboard branch first, instead you can add my repo as a remote and you can checkout my new-keyboard branch and work on top of that. (Although I guess this means that we should merge PRs into winit's new-keyboard branch without squashing... But that shouldn't concern you @garasubo)

[maroider]

As mentioned, I'm working on the Wayland and X11 implementations simultaneously, as I want them to share logic where possible. My efforts on the X11 implementation have admittedly ground to a halt due to issues with initializing libxkbcommon, and I intend to take another stab at getting it working today. I would of course be grateful for any help I can get, be it code or advice. If you'd like, @garasubo, we can chat more about this on Matrix or Discord (where you'll find me as maroider#8969).

[inodentry]

Feel free to ping me if you need any testing on Wayland and XWayland. :slightly_smiling_face: I am a multilingual user with keyboard layouts (incl. custom ones) for different scripts.

[mahkoh]

Afaict the API as discussed here does not support multiple seats which makes it cumbersome impossible to implement it on platforms that support multiple seats. E.g. consider the following sequence of events:

• Seat 1: Press Alt
• Seat 1: Press A
• Seat 2: Press Shift
• Seat 2: Press B
• Seat 1: Press C

I'm not sure which sequence of events this would generate with this API.

[dhardy]

I'm pretty sure a winit app would be restricted to one seat and only report events from that. It's not like the apps on Seat 1 should know what is going on at Seat 2.

[mahkoh]

I'm pretty sure a winit app would be restricted to one seat and only report events from that.

That seems like a significant restriction. Has this already been discussed elsewhere?

[mahkoh]

It's not like the apps on Seat 1 should know what is going on at Seat 2.

I missed this when I read your comment the first time. I think there is a misunderstanding what a seat is. A seat is simply a mouse and/or keyboard set. If you have multiple independent sets, then you have multiple seats. Each seat is represented on the screen by its own cursor.

Iirc on wayland it is standard that each tablet connected to the PC is its own seat. This means that you have to support multiple seats to fully support tablet input on wayland.

Another example of multiple seats are VNC applications that represent the remote mouse/keyboard as a separate seat so that the remote user doesn't have to fight with the local user over control of the cursor.

[veryjos]

This issue is huge and a few years old, I'm having trouble tracking..

Is the key repeat solution proposed in the original issue available yet? I can't find anything in the source.

[ArturKovacs]

I think this will be very helpful to you: #1806

To answer your question, nothing is merged into master yet but the implementations are mostly done actually. See the tracking issue I referenced above.

[kchibisov]

The #2662 was merged addressing most of this. For follow ups I'd suggest to open separate issues.