Add UI automation portal

[Jmgr]

Would it be possible to add an automation portal? I'm the developer of Actiona, a UI automation tool, and I'm trying to migrate away from using XTest to simulate mouse and keyboard actions.

I know there is RemoteDesktop and ScreenCast, but their use-case is a bit different and they don't provide all the required features.

This portal could include input and windowing events/actions, like:

• Mouse events (motion, button pressed/released)
• Get and set absolute cursor position (warp)
• Keyboard events (key pressed/released)
• Press and release keys (both with and without keyboard layout translation)
• Choose a position on screen (like Screenshot.PickColor, but returns an absolute position)
• Choose a rectangle on screen
• Choose a color on screen (Screenshot.PickColor does that)
• Choose a window on screen
• List windows
• Get and set the foreground window
• Close/minimize/maximize/... windows

[KotoWhiskas]

Agree, also ability to send keyboard/mouse events to particular windows even when they are not focused would be useful

[KotoWhiskas]

This would also help to resolve the keepassxc issue with autotyping on wayland.

[orowith2os]

This should be resolved with the creation of libei and the appropriate portal updates to go with it already.

@KotoWhiskas password managers are a separate problem, and as far as I can see, are a bit stalled in different areas.

[swick]

libei is made for emulating devices, not for automating UI. Different use cases, different requirements.

[orowith2os]

Different use cases, different requirements.

Is there anything in particular blocking libei from doing this? I'd figure not. You can 100% automate stuff with it. How well, depends..

Another path to take is abusing future accessibility stuff for automation, which I think already works.

[swick]

Is there anything in particular blocking libei from doing this?

Yes. It operates on the level of a device. It can't warp a pointer to a specific location because devices don't even know there is a thing with locations. Similarly an emulated keyboard can't give you specific text because it has no concept of a keymap.

[whot]

sorry, that's not quite correct: libei keyboards can have a keymap that's assigned by the EIS implementation, basically the same as wl_keyboard. And absolute devices have "Regions" which are also assigned to by the EIS implementation, that's how the remote desktop stuff works and how we can map the absolute input devices to the respective streams too (esp with the new region mapping id in libei 1.1).

but it's true that libei itself is just a transport layer, most of the decisions would have to be part of the portal API.

[swick]

Oh, totally forgot about absolute pointer devices. Maybe it is useful for automation after all?