Feature request: enable all actions other than the `ChangeLayer` action in TCP server, e.g `setMouse`

[wis]

Is your feature request related to a problem? Please describe.

Right now, it appears the TCP server in Kanata only supports one command, ChangeLayer, it should support all the actions possible in Kanata.

Describe the solution you'd like.

Specifically I need the setMouse, moveMouse and the actions for the mouse buttons.

Describe alternatives you've considered.

I tried thinking of some, I can't come up with any.

Additional context

In the example for a TCP client, and also in the code for the TCP server, it appears that only one action is supported, ChangeLayer.

[wis]

This feature request already has an open issue.

Apologies.

[rszyma]

I'd suggest keeping this issue open, because #496 is mainly about receiving info from kanata, while this issue you're asking to add RPC. This is definitely interesting, though I'm wondering what would be the use case? Some kind of remote control?

[jtroo]

Maybe this could expose the fake key operations, since fake keys should be able to perform any action.

[rszyma]

But this would require defining fake keys for every single action that user wants to expose via tcp server. It kinda makes sense to use fake keys to restrict what can be done via RPC from security standpoint, and for sure we shouldn't allow cmd action. But besides that I don't see why restrict any other actions, except maybe for simpler implementation.

[jtroo]

Simpler implementation is the main concern. It doesn't seem worthwhile adding code to handle every type of action in the TCP server code when fake keys already exist and seem to fit the use case very well.

[wis]

@jtroo, I really apologize for responding this late, I don't know why I've always thought that I DID respond the day I saw your reply, the day after you posted it, and I've always been really looking forward to see this feature implemented.

I'd suggest keeping this issue open, because https://github.com/jtroo/kanata/issues/496 is mainly about receiving info from kanata, while this issue you're asking to add RPC. This is definitely interesting, though I'm wondering what would be the use case? Some kind of remote control?

@rszyma, you naming the feature "RPC" and naming the use case "remote control for Kanata" is perfect and I haven't thought of it that way. I've confused the two features in my mind because I need both features for my use case: RPC, and, the subscribing to key press events.

My use case requires RPC because I do not want to duplicate the functionality of simulating global input in my program, as Kanata does, and does perfectly, and my use case also requires the functionality outlined in the other feature request open GitHub issue that I mentioned and mistakenly closed this issue in favor of (the functionality of listening to global key events, noop key press events in my case), because I also do not want to globally intercept and/or block key input in my program, as Kanata does, doing that may interfere with Kanata, and I actually do prefer and like the idea of having a layer-mode in Kanata that is for "remote controlling" my program's window, the window is just a transparent, click-through always on top window that renders text labels and changes the rendering of the text labels on each key press, the text labels are for letter combinations/permutations that you can type, it is for a mouse mode, a mode that is part of my Kanata config for mouseless/keyboard-only computer use, the "remote controlled" window basically* looks like warpd**:

GIF copied from the README.md for warpd :

so why do I need both features for my use case? I need key press events from Kanata while in the mouse-picker layer/mode to update the rendering of the picker on each key press and I need RPC to send mouse events using Kanata (through Kanata), events like move mouse to so and so X and Y, i.e. RPC with parameters, which as far as I can tell, the current implementation, with fake keys, does not support, with the current implementation you can not supply parameters to actions like SetMouse, e.g. {"SetMouse":{"x": 200, "y": 400}}. delegating these functions to Kanata (the functions of global input interception and global input simulation), enables me to not need to reimplement functionality that is in Kanata and so elegantly and beautifully implemented, and also enables me to support all the platforms that Kanata supports, and doing that is inline with the Unix Philosophy:

• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams, because that is a universal interface.

My program's only job then becomes just the rendering, it re-renders when key input events are received from Kanata, and when a complete permutation is typed (i.e. in a functional programming sense, it .take()s 4 letter-key press events, making a 4 letter permutation), it calculates the X and Y of the text label of that letters permutation on the screen, and then sends a sequence of commands to Kanata through RPC, setMouse: X, Y, then mlft or mmid or mrgt

The program can be further modularized by making it receive input through standard input, one letter/character at a time, and making it output X & Y coordinates of selected text labels to standard output, and then use a simple glue script/program to glue them together. I believe I can use the --debug flag when starting Kanata to listen to key press events through standard output which enables me to do without the other feature, for the time being.

*here's how my program that implements "picker" mouse mode looks:

https://github.com/jtroo/kanata/assets/9993663/91b84c02-8801-411b-9ca6-1fe5c570e4f4

in this screen capture video the first to letters out of the 4 letter permutation were typed at my normal speed, which through muscle memory has gotten fast, fast enough to not clearly show in the recording the second frame the window rendered, if you don't pause the video at the right split second, the latter 2 letters were typed at slow speed for demonstration purposes.

_{** (I built my program before knowing about warpd, but I do not claim it is an original idea of mine, I've discovered it has been implemented many times over the years)}

[jtroo]

I see how fake keys indeed doesn't support the use case of arbitrary setmouse values.

One thing to note though with setmouse is that it does not work in Linux because I could not find a uinput level mechanism to do it, and I didn't want to get into integrating with the various DEs.

Otherwise it should be fairly simple to support forwarding the mouse actions though.

[jtroo]

Places to change:

https://github.com/jtroo/kanata/blob/bcad5e0001ef3d4237f367a547471fdb1704522f/tcp_protocol/src/lib.rs#L20

https://github.com/jtroo/kanata/blob/bcad5e0001ef3d4237f367a547471fdb1704522f/src/tcp_server.rs#L120

Reference for sending uinput events:

https://github.com/jtroo/kanata/blob/bcad5e0001ef3d4237f367a547471fdb1704522f/src/oskbd/linux.rs#L324

[rszyma]

One thing to note though with setmouse is that it does not work in Linux because I could not find a uinput level mechanism to do it, and I didn't want to get into integrating with the various DEs.

FWIW, drawing tablets operate by reporting absolute position coords. So maybe kanata could try to emulate one? I've experimented with it in the past, unfortunately no success. I couldn't find any useful info about how to do it properly, and also I don't own the real thing to cross-check with the virtual one.

Anyways, that'a a bit offtopic, it's separate discussion itself.