Capture input from xinput sent from other programs?

openstenoproject / plover

Open source stenotype engine

http://opensteno.org/plover

GNU General Public License v2.0

2.32k stars 281 forks source link

Capture input from xinput sent from other programs? #1191

Open user202729 opened 3 years ago

user202729 commented 3 years ago

Summary

Currently, it's impossible to use Plover with TeamViewer keyboard input.

I think that for most such remote-control programs (tested with TeamViewer) they use xinput to send input (just like Plover)

Capturing from xinput is possible with some simple code change, but sending keys after capturing xinput requires a significant change.

Perhaps after that some IME or uinput can be used to send input instead (which is independent from xtest) (although I think it requires root?)

More idea: adding another type of plugin (output plugin), and people can write their own plugins and let the user select from the possible methods.

I don't think there's any straight forward way (it might be possible to start a new X server (I think this is the correct term?), but then TeamViewer would capture screen from that server)

Reproducing

This part is straight forward.

Plover Version

Newest, with some patches that should not affect the issue.

System

X on some GNU/Linux distribution.

benoit-pierre commented 3 years ago

Regarding the idea of output plugins, see https://github.com/benoit-pierre/plover/commit/2d41c1173e44142df6be0e7e13d33a09203a7e83 (from this branch).

user202729 commented 3 years ago

Note that because Plover can capture from a subset of keys, and there are spare keys, so this can be done without alternative method to output (IME or python-uinput (requires root)) -- capture the keys as usual, but only send keys that no other applications use.

Support for output plugin remains necessary.

More about output plugin: in Benoit's implementation above, it's a breaking change (suppress_keys; params as required input of KeyboardEmulation; StenoEngine constructor); although it can be made non-breaking without too much difficulty.

I cherry-pick it to upstream/master in https://github.com/user202729/plover/tree/output-plugin and it appears to work (but not with the console UI plugin, see above)

It would not be easy to maintain compatibility (other than making KeyboardEmulation load the config to determine the correct plugin; but then that breaks the API of KeyboardEmulation)

user202729 commented 3 years ago

I implemented a test solution with https://github.com/user202729/plover-xtest-input and https://github.com/user202729/plover-unused-xtest-output .

Initially I intended to use https://github.com/user202729/plover-uinput-output but that requires root privilege. (edit Implemented now, at least for simple keys on the keyboard. Still require sudo to work without password. Also implement a hack to make it work on Plover versions without plugins)

user202729 commented 3 years ago

It turns out that the feature is not that useful because TeamViewer automatically change key held into sequence of repeated key presses.

... Perhaps except arrow keys.

In any case, this is definitely not Plover's bug. I have some ideas how to workaround that...

Set up a separate P2P connection to transmit keypresses data
Send it through TeamViewer's key presses (note actually implemented in the plugin above + https://github.com/user202729/dotterel , 7 keys per chord, fast enough, still somewhat buggy)

Both methods require modification of both client and server side.

5/6 keys per chord isn't that bad.

Reconstruct keyboard state on the target machine

Not very reliable.

Reverse engineer TeamViewer (or otherwise somehow intercept the key presses before it's sent to the X server) and feed it to Plover. (perhaps it isn't that hard...?)

user202729 commented 3 years ago

Cross-link: Special case of issue 1187 , but this one, to be usable, would need an alternative output method.