input-method: unclear how keyboard grabs routing should work

[emersion]

When keyboard grabs are used, the input routing should be set up like so:

• Compositor sends keyboard events to the IME keyboard grab
• IME creates a virtual-keyboard and emulates key events
• Compositor sends emulated key events to regular clients

This makes it necessary to know which virtual keyboard is the IME's. Maybe a VNC session is running, and ideally VNC key events would go through the IME as well.

One solution would be to add a way to tie a virtual keyboard to an input method in the input-method protocol.

See https://github.com/swaywm/wlroots/pull/1864#issuecomment-627820399

cc @dcz-purism @xdavidwu

---

wlroots has migrated to gitlab.freedesktop.org. This issue has been moved to:

https://gitlab.freedesktop.org/wlroots/wlroots/-/issues/2322

[dcz-purism]

Would adding a client-controlled flag like "bypass grabs|respect grabs" on the virtual keyboard do the trick? I think it's a bit more universal if the client was using some other way to grab while providing a virtual keyboard.

In the example of squeekboard, a more "keyboardy" layout like the terminal could always respect grabs as a simple keyboard emulator in conjunction with another input method (or even its own grab). A more "texty" layout like Kana would be too far removed from what input methods expect, so it would work better by always providing characters directly.

[emersion]

In both examples ("keyboardy" and "texty" IMEs), we don't want the IME's virtual-keyboard emulated input events to go through the IME keyboard grab again, because that would be an infinite loop. Am I missing something?

[dcz-purism]

I was thinking about the scenario of having two "IME"s: one providing keyboard, and another turning the keyboard input into something useful.

Note that the input-method protocol is delierately only "texty", and won't be used by keyboards.

A larger input method could do both: provide a virtual keyboard, and offer keyboard input conversion. The routing would look sth like this:

on-screen widget -> virtual_keyboard (grab) -> IME interprets and forwards -> virtual_keyboard (no grab) -> application

Both virtual keyboards in this scenario "belong" to the same IME. This could be adjusted by telling the compositor only about the second one, I guess.

I think my main question here is: why make input-method and virtual-keyboard know about each other when we can just set a flag on the virtual-keyboard?

[emersion]

why make input-method and virtual-keyboard know about each other when we can just set a flag on the virtual-keyboard?

The flag wouldn't just be "bypass grabs|respect grabs": we don't want keyboards to be able to bypass drag-and-drop or xdg-popup grabs for instance. The flag would be about input-method grabs only. So the two protocols would need to know about each other in the end.

I'm not a fan of a flag because it feels backwards: instead of having the compositor figure out the virtual-keyboard is tied to the input-method and do the right thing, the client would make a decision and inform the compositor about it.

Is there a use-case for this flag for virtual-keyboards not tied to an input-method?

[dcz-purism]

I think there's a weak case to be made for using multiple physical keyboards simultaneously (this is kinda the mirror of the two virtual keyboards situation), where they would have different layouts and keymaps, and one would feed the input method, while the other would submit its events directly. However, with the recent conversations about virtual-keyboard, this doesn't seem sensible, and it's better done by translating keyboard input to input method events inside the compositor anyway.

You're right, the flag would have to be restricted to the ability to be grabbed by the input method. I can't think of a case where input method events should bypass xdg_popup (not sure what you mean with drag-and-drop though).

I think such a flag would be very similar to a connection between virtual-keyboard and input-method, except without interdependencies. The same information is getting communicated. Connection example:

virtual_keyboard.should_bypass_grabs_from(input_method)

versus flag example:

virtual_keyboard.set_should_bypass_grabs_from_input_method()

The difference is that the second version continues after the input method is replaced with a new one, and that multiple grabs are all ignored (although that should be impossible in the first place). So the first version maintains a clear mapping. Right now I can't tell if those are properties I care about, but there are still observations I want to make:

• the client is in control in both cases
• the second one is easier when things are running in separate processes
• the second one doesn't pull the input method protocol as a dependency

Did I understand your concern correctly, or did I get it wrong?

(There's also the option that the input method communicates which virtual keyboard it owns, but I would rather not reference a controversial protocol inside a less controversial one.)