[feat] Expose `WindowEvent::KeyboardInput` in Rust

stijnfrishert commented 2 days ago

Describe the problem

I want to be able to react to key events coming from the application event loop.

I am building a shortcut system for my Desktop app, on macOS. I would like to be able to trigger hotkeys, even if the application doesn't have any windows open. This rules out any js key listener solutions, because they are tied to windows. Plus, this feels like something that should be done over in Rust-land, anyway.

The current recommended solution is to use rdev. This works up to a point (see https://github.com/tauri-apps/tauri/issues/11670), but requires my app to ask for full access to all OS-level keyboard inputs (Accessibility), which is a permission I would rather not depend on.

In reality, this is something that should be solvable by propagating events from the event loop. I scoured through Tao a bit and found key event types, so this seems to be within the realm of possibility?

Describe the solution you'd like

I want to be able to register a key event handler on my app. I envision this would look something like this, but that's of course up for debate:

tauri::Builder::default()
    .on_key_event(|app, event|{ }))
    .run()

I can imagine plug-ins might want to do this too, so maybe we should have a system to register (and unregister) multiple handlers.

Alternatives considered

Binding to key event listeners in js, but this can only be done on windows. Besides, imho this is something to be solved on the Rust side, using the application event loop.
Using rdev, but this requires my app to ask for full keyboard monitoring on an OS-level, which is not a permission I want to ask for, really.

In addition, people have to twiddle with device_event_filter, which is neither intuitive nor easy to find.

Additional context

I've searched through the Discord and issue list, and it seems more people have been requesting this.

amrbashir commented 1 day ago

similar to the other issue, your problem may just be solved by our global shortcut plugin, https://tauri.app/plugin/global-shortcut/

FabianLars commented 1 day ago

i disagree, app-local shortcuts and/or an equivalent to js key events in rust are a regularly requested features (i was sure there was one on github already before this but i can't find it...)

amrbashir commented 1 day ago

app-local shortcuts and/or an equivalent to js key events in rust

not sure what you mean by this, please explain more

FabianLars commented 1 day ago

basically key events that only work when the app is in focus.

for example, let's say you want to listen to ctrl-s to save data. in your app most of the data is hosted on the rust side though so listening to the ctrl-s event on the rust side would be the easiest solution.

All current "solutions" have drawbacks:

global-shortcut: has to be registered and unregistered on focus change. also the "stealing" nature of it may not be always desirable.
rdev: also listens to events globally meaning you have to keep track of the focus state. The bigger issue is the macos accessability permission requirement though. And it doesn't work on wayland.
listening to "keydown" (or similar) in js and forwarding that with tauri events to the backend: works alright, but still weird. Also potentially taxes the ipc more than needed

i'm not exactly sure how it behaves on macos but at least on windows tao's WindowEvent::KeyboardInput behaves exactly as desired

i realize i said mostly the same as OP but idk what else to say

FabianLars commented 1 day ago

for example, let's say you want to listen to ctrl-s to save data. in your app most of the data is hosted on the rust side though so listening to the ctrl-s event on the rust side would be the easiest solution.

I also often get asked how to have global-shortcuts not be global because a shortcut like api is easier (for shortcuts) than listening to key events but that would just be an extra imo

amrbashir commented 23 hours ago

You keep calling it app-local, app-level where but I assume you mean window-level keyboard input but in Rust which is just forwarding taos' WindowEvent::KeyboardInput event, right? which is fine, we can expose it

FabianLars commented 22 hours ago

yes

stijnfrishert commented 13 hours ago

You keep calling it app-local, app-level where but I assume you mean window-level keyboard input but in Rust which is just forwarding taos' WindowEvent::KeyboardInput event, right? which is fine, we can expose it

You keep calling it app-local, app-level where but I assume you mean window-level keyboard input but in Rust which is just forwarding taos' WindowEvent::KeyboardInput event, right? which is fine, we can expose it

Yes, and no. In any case, I'd already be very happy if we could receive Tao's keyboard events in Rust Tauri by forwarding them.

There is a difference between app- and window-level key events, but only on macOS. Mac apps can launch without having any window open, and can stay active when your last window closes. This is also why the menu resides at the top of your screen, and isn't tied to any specific window.

A hotkey like Cmd + N to create a new "project" should be triggerable even without having any window open. To Windows and Linux users that seems exotic, but a lot of macOS apps work this way, including all Apple apps that come with the OS itself (Finder, for example). Technically this works, because key events are received by NSResponders, and NSWindow is one, but so is theNSApplication itself.

But to reiterate, at this stage I'm already happy if I can just receive window events forwarded from Tao.

All current "solutions" have drawbacks:

global-shortcut: has to be registered and unregistered on focus change. also the "stealing" nature of it may not be always desirable.

rdev: also listens to events globally meaning you have to keep track of the focus state. The bigger issue is the macos accessability permission requirement though. And it doesn't work on wayland.

listening to "keydown" (or similar) in js and forwarding that with tauri events to the backend: works alright, but still weird. Also potentially taxes the ipc more than needed

Yeah, so receiving window events directly alleviates the accessibility permission, because apps have access to their own keyboard events by default.

For something like the rdev solution to work, I would need app-level focus/gain listeners to figure out when to register and unregister the key listeners. If not, they do a hostile take-over of hotkeys across your OS, and Cmd + N also triggers when, say, VSCode is in focus.

And, for macOS users, JS keydown events also don't fix this entirely, because you can't listen to non-window keydown events. :/

In any case, thank you for taking this request seriously. I know more people have been requesting this, and although it's not a waterproof argument, this is something most other app frameworks also support out-of-the-box.

I'm happy to work with you and send in a PR, especially because the amount of macOS devs is probably low, but I would need some guidance/mentorship on where to look in the code.

tauri-apps / tauri