Replacing input/output mechanism/framework

[BlueDrink9]

For Wayland and unicode support, and for reducing general jankiness, we will probably have to replace the mechanism we use for outputting keys/text, and the system for input/reading keys.

Aim to refactor interface and key monitors to add an abstraction layer between Autokey and the system.

Several suggestions have come up. One possiblity is to use evseive and create an entirely new device for the inputs (One device named AutoKey Input would be able to send both mouse and key signals) and then only have autokey listen to hotkeys sent via the other keyboards, which evdev/uinput makes pretty easy (at least in theory).

Another is using xdo, via libxdo (see #255)

[gusbemacbe]

For Wayland and unicode support, and for reducing general jankiness, we will probably have to replace the mechanism we use for outputting keys/text, and the system for input/reading keys.

I have been waiting for Autokey's support for Wayland-based Sway.

[josephj11]

I wonder if someone could package AutoKey in a container (appImage/snap/flatpak ...) so it could bring its environment, including X11, with it. Maybe that would work with Wayland. I use a few apps in containers, but don't know anything about building them.

[gusbemacbe]

Maybe @probonopd know.

[BlueDrink9]

Resources I am following to look into this at the moment:

Low-level key binding and sending (probably always requires root access)

• https://www.codedbearder.com/posts/writing-keyboard-remapper-wayland/
• https://git.alemauri.eu/alema/hkd
• https://python-libevdev.readthedocs.io/en/latest/

Key sending

• xdo

Key binding

• keybind (uses XRecord, same as us)

[BlueDrink9]

So far my research suggests that, for sending unicode, xdo will be the best option. However, as Thomas encountered in @255, xdo operates at a level above xrecord, meaning it sees xdo events as input. So I think the challenge will be overcoming that, possibly by well-timed ignoring/passthrough of grab events.

This change will sadly not help users on Wayland as of yet, because the scene for key binding and sending libraries is still too young. (But refactoring to plug in xdo will help in the future.)

I think refactoring the interface more is still necessary. I am working on that at the moment, but I want to separate out the interfaces for sending vs grabbing, as well as for window and mouse interaction. In addition, I am unsure how much of the threading code is directly related to Xlib, so if possible I would like to abstract that up a layer.

[BlueDrink9]

In #255, Thomas identifies 3 issues with simply replacing xrecord's send with libxdo.

I’ve just tested python-xdo by simply replacing the __sendString() function. 1. It causes the active window to look like it looses it’s focus as soon as expansion starts, but it won’t receive the typed text. (It looks like unfocused but behaves like it’s focused.) 2. And it causes feedback loops within AutoKey, as AK starts to see it’s own output as input. 3. Additionally, the KDE keyboard layout manager messes with the output and does an unwanted double-translation of key symbols.

I can reproduce all of them on my machine. However, I have now solved 1. By going through the program flow for running a phrase, and moving an xdo send into each part, I found out exactly where it was that autokey was blocking the program from receiving input in the that cli python xdo wasn't having issues with.

A phraserunner tells the iomediator to 'begin send' at the start of executing a phrase. https://github.com/autokey/autokey/blob/ab367da9fc4d3cbf75b07f8329b0003c30baf5f7/lib/autokey/service.py#L407 That calls grab_keyboard on the interface, https://github.com/autokey/autokey/blob/ab367da9fc4d3cbf75b07f8329b0003c30baf5f7/lib/autokey/interface.py#L585 https://github.com/autokey/autokey/blob/ab367da9fc4d3cbf75b07f8329b0003c30baf5f7/lib/autokey/interface.py#L594 which apparently blocks xdo input. All I needed to change was to wrap xdo's send in a

self.ungrab_keyboard()
...
self.grab_keyboard()

and it would successfully send the string.

Issue 2. was also fairly simple. Simply telling autokey to pause the service before sending anything with xdo prevented xdo's input from recursively triggering anything in autokey.

Issue 3. still happens, however I don't think we need to worry about it. It only seems to affect which recursive functions are triggered, but fixing 2 means we ignore the keys autokey sees. I think what happens is xdo operates at a high level, and sends the keys as the application sees them. However, the OS in my and Thomas's case modifies the keyboard layout using xmodmap, which is why we both see autokey claiming the "raw key" input as something else (the key unmodified by the OS, as if sent by the keyboard). In my case, it matches the label on the physical key. For example, if I press the 't' key, I get a 'g' outputted. If I tell xdo to send 'g', autokey sees a raw keypress of 't'. That only happens for characters that can be send physically. For Chinese characters, it sees \x00 (But thanks to xdo, the correct string is actually send to the application).

Long story short, this is fully feasible, and I could whip up a hack patch in a day or two and merge it to develop. However, I'd also like to refactor it to make it a clean drop-in, to make wayland integration easier in the future. But I will do that separately.

Unicode output support for Autokey is on its way!!!

[BlueDrink9]

See #610

[Elliria]

I'm not sure how useful this page will be, but it lists several alternatives to xdotool: https://www.reddit.com/r/linuxquestions/comments/u5mxzi/xdotool_alternative_for_wayland/

[josephj11]

I added those tools to my personal AutoKey notes, but I don't currently run Wayland anywhere to try them out. I couldn't find wdotool either. It may have been a typo.

[Elliria]

Yeah, or maybe someone just having some fun and not being serious.