Reduce reliance on external libraries / certain upstream packages are unmaintained

[ehfd]

https://github.com/selkies-project/selkies-gstreamer/commit/11006e393b9b7af81ab2b01d82a0092167da3e87

As specified here, the upstream python-uinput package is not maintained for 6-7 years.

Might need a change in the uinput interface to a project that continues to update, or maintain a selkies-project fork.

Moreover, why can't we completely replace pynput.mouse.Controller() and pynput.keyboard.Controller() with Xlib.xtest?

Also, we need to update all the dependencies for JavaScript.

[ehfd]

Merging with #90.

[ehfd]

We may prefer to replace python-uinput with python-evdev, as we have gotten rid of /dev/uinput dependencies.

As far as I see, our reliance on python-uinput is limited to uinput.BTN_LEFT, uinput.BTN_MIDDLE, uinput.BTN_RIGHT, uinput.REL_X, uinput.REL_Y, uinput.REL_WHEEL, in webrtc_input.py. We aren't creating a new virtual input device with /dev/uinput, and our current usage should be covered with python-evdev.

Moreover, why can't we completely replace pynput.mouse.Controller() and pynput.keyboard.Controller(), and just use Xlib.xtest like we did in: https://github.com/selkies-project/selkies-gstreamer/blob/61f6b6c834337a02a9ad264fbd1be237418ce5f8/src/selkies_gstreamer/webrtc_input.py#L273-L277

[ehfd]

https://github.com/selkies-project/selkies-gstreamer/pull/113

python-xlib is also unmaintained by the maintainer.

[ehfd]

I am generally not too happy with dependencies: python-uinput (dependency is effectively removed but still persists, plus it is unmaintained), pynput (capabilities can be replaced by directly using python-xlib), basicauth (capabilities are trivial and replaceable with default packages), gputil (unmaintained and easily replaceable with another package such as pynvml and gpustat).

python-xlib is also used extensively and there is no viable alternative, but the repository maintainer is not that responsive.

[ehfd]

CC @cruizba

[cruizba]

I think I found a possible way to fix all inconsistencies and errors related to the keyboard, potentially paving the way for the removal of pyinput and, possibly in the future, python-xlib. You can find my work in this branch of my fork: GitHub - cruizba/selkies-gstreamer.

Let me share the backstory of how I arrived to this.

Upon further investigation, I realized that certain keysyms which reach this section of webrtc_input.py, particularly those representing Spanish characters on my keyboard (like ñ), lacked corresponding keycodes. I've noticed that X server loads some generic keycodes, but it doesn’t cover all of them. This might relate to the default keyboard setting, which is typically a Generic 105-key PC, but I am not sure.

In a remote desktop environment, the server is unaware of the client's keyboard. Therefore, modifying the server's keyboard configuration seemed impractical as a solution to me...

To explore how other projects handle keyboard input remotely, I looked into Neko. Neko, along with Guacamole keyboard lib in the browser, manages keyboard events from the browser. When the keysym reaches the neko server, they process input data through a set of C functions in xorg.c, which they integrate with Go.

The key function of xorg.c is XKey(). It appears that XKey dynamically assigns KeySyms to Keycodes when there is no existing association. If a keysym is sent without a corresponding keycode, it registers this new mapping in the XOrg keyboard configuration (As far as I can understand from the code). This approach is also utilized in TigerVNC's source code for keyboard keys mapping!

Consequently, I've created xorg.py to integrate the build of xorg.c and changed the keyboard part of webrtc_input.py to utilize the new xorg class. This solution works exceptionally well! It detects all keyboard keys correctly without altering any remote keyboard settings and while using my own keyboard configuration.

The challenge was creating a new addon and compiling the xorg.so file to implement this functionality. The project now requires additional headers and libraries. However, the good news is that this addon could potentially replace python-xlib for mouse interactions and other X interactions.

If you're interested in testing it, my branch is fully functional. Simply run vscode with the devcontainer as usual and execute the task [run] re-build, re-install and run selkies-gstreamer. It's configured to compile and install the library.

[ehfd]

This is insanely good. @cruizba I am open to compiling libraries. I had an issue where the '>' key was pushed as '<', and it was indeed an issue with the keyboard layout.

I would like to merge this ASAP. Could you help with the PR?

[ehfd]

@cruizba One other possible option is to start using cython. Then, xorg.c and xorg.h can be seamlessly integrated into the Python package.

[cruizba]

I would like to merge this ASAP. Could you help with the PR?

Sure, I'll try to continue this weekend, I am doing this in my spare time.

I had an issue where the '>' key was pushed as '<', and it was indeed an issue with the keyboard layout.

Is this specific issue fixed? Did you test it?

cython

I will take a look. I am not a python expert, but I've written some things in the past. But I suppose that this let you write C code with python syntax, am I right?

Also, I want to optimize the xorg.c implemenation a little bit.

I see that the table of keycodes has a max of 255 keycodes: https://github.com/TigerVNC/tigervnc/issues/93

I think if you change keyboard layouts frequently, the table fills entirely. So I was thinking about having an auxiliary map to save associations and then removing Key Codes just when a key up event arrives, so the table never fills.

[ehfd]

@totaam

I would like to consult your opinion.

https://github.com/cruizba/selkies-gstreamer/tree/fix-inputs/addons/xorg-iface https://github.com/cruizba/selkies-gstreamer/blob/fix-inputs/src/selkies_gstreamer/xorg.py

Would it be plausible to integrate this code into a wheel file? Asking because Xpra actively uses Cython.

[cruizba]

I will try to rewrite the essential parts of the keyboard in Cython. I'm pretty sure it is possible to do.

[totaam]

@ehfd Seeing how few method calls you actually need, porting to Cython would be trivial. However, wheels are portable, Cython modules are not. Using python-xlib or a ctypes wrapper would be a portable solution.

[ehfd]

Does XKey() exist in python-xlib?

Using an external .so file looks like the current approach. I would accept that with the Joystick Interposer because it needs to set LD_PRELOAD systemwide, but this is slightly outside my comfort zone for this purpose.

[ehfd]

I had an issue where the '>' key was pushed as '<', and it was indeed an issue with the keyboard layout.

@cruizba

https://github.com/selkies-project/selkies-gstreamer/blob/9d39f730f67890cbd58ec5c0f8f77ededafd1fbc/src/selkies_gstreamer/webrtc_input.py#L320-L324

[totaam]

Does XKey() exist in python-xlib?

XKey is not an xlib function, so no.

.. this fix may present issues in some keyboard layouts

Hmm. Been there, got scars to prove it. These kinds of hacks don't last long.

[cruizba]

@ehfd

Does XKey() exist in python-xlib? A ctypes wrapper is also totally fine.

No, files present at https://github.com/cruizba/selkies-gstreamer/tree/fix-inputs/addons/xorg-iface are just a simple module using x11 libs, it is not part of the X library.

But I can try to rewrite the xorg.c using Ctypes, instead of calling the xorg.so

[ehfd]

Understood. Thank you all.

I will give you the collaborator role in Discord if you tag me. @cruizba

[ehfd]

python-uinput was removed from the dependencies.

[ehfd]

I could really receive help from someone who could update the breaking web interface. Old Vue and Vuetify are breaking a lot of capabilities. Basically a 2019-era antique.

Follow web dependencies up in #108.

This issue will be focused on backend dependencies.