extended input device support

[totaam]

think: tablet pcs can be used with the program running on the higher end desktop machine.

Pressure sensitivity, fine-grained scrolling, etc.

[totaam]

See investigation in #1131#comment:5

Looks like we want uinput.

[totaam]

Best links from #1131#comment:5 and more:

• python-uinput examples
• linux kernel event codes
• Getting started with uinput: the user level input subsystem
• Using smooth scrolling over uinput
• /dev/uinput with X11 and blacklisting a device
• libsuinput
• X Input Driver HOWTO
• Arch mouse buttons
• GfxTablet: GfxTablet shall make it possible to use your Android device (especially tablets) like a graphics tablet.
• virtual mouse and eventx
• Disable scrolling on <input type=number> for html5 client

[totaam]

More information:

• Create a Wacom-like Linux uinput device for work with touchscreen and pen
• X11 XCB xpyb xev -> pyxev, for XI2 touch events and gestures
• Input device routing
• [ftp://www.x.org/pub/X11R7.5/doc/man/man4/evdev.4.html evdev man page]
• libinput docs: tablet support and helper tools: libinput-list-devices and libinput-debug-events
• Understanding evdev

[totaam]

2016-06-03 11:51:49: antoine commented

---

And some information from a trackpad which works for smooth scrolling. From libinput-list-devices:

Device:           DLL0704:01 06CB:76AE Touchpad
Kernel:           /dev/input/event13
Group:            6
Seat:             seat0, default
Size:             101.33x56.67mm
Capabilities:     pointer 
Tap-to-click:     disabled
Tap-and-drag:     enabled
Tap drag lock:    disabled
Left-handed:      disabled
Nat.scrolling:    disabled
Middle emulation: n/a
Calibration:      n/a
Scroll methods:   *two-finger edge 
Click methods:    *button-areas clickfinger 
Disable-w-typing: enabled
Accel profiles:   none

$ xinput list | grep -i touchpad
⎜   ↳ DLL0704:01 06CB:76AE Touchpad               id=11   [slave  pointer  (2)]

$ xinput list-props 11
Device 'DLL0704:01 06CB:76AE Touchpad':
  Device Enabled (137):   1
  Coordinate Transformation Matrix (139): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
  libinput Tapping Enabled (275): 1
  libinput Tapping Enabled Default (276): 0
  libinput Tapping Drag Lock Enabled (277):   0
  libinput Tapping Drag Lock Enabled Default (278):   0
  libinput Accel Speed (279): 0.000000
  libinput Accel Speed Default (280): 0.000000
  libinput Natural Scrolling Enabled (281):   0
  libinput Natural Scrolling Enabled Default (282):   0
  libinput Send Events Modes Available (259): 1, 1
  libinput Send Events Mode Enabled (260):    0, 0
  libinput Send Events Mode Enabled Default (261):    0, 0
  libinput Left Handed Enabled (283): 0
  libinput Left Handed Enabled Default (284): 0
  libinput Scroll Methods Available (285):    1, 1, 0
  libinput Scroll Method Enabled (286):   1, 0, 0
  libinput Scroll Method Enabled Default (287):   1, 0, 0
  libinput Click Methods Available (288): 1, 1
  libinput Click Method Enabled (289):    1, 0
  libinput Click Method Enabled Default (290):    1, 0
  libinput Disable While Typing Enabled (291):    1
  libinput Disable While Typing Enabled Default (292):    1
  Device Node (262):  "/dev/input/event13"
  Device Product ID (263):    1739, 30382
  libinput Drag Lock Buttons (293):   <no items>
  libinput Horizonal Scroll Enabled (264):    1

$ xinput query-state 11
2 classes :
ButtonClass
  button[1]=up
  button[2]=up
  button[3]=up
  button[4]=up
  button[5]=up
  button[6]=up
  button[7]=up
ValuatorClass Mode=Relative Proximity=In
  valuator[0]=467
  valuator[1]=1056
  valuator[2]=-1401
  valuator[3]=2254

$ xinput get-button-map 11
1 2 3 4 5 6 7 

$ xinput get-feedbacks 11
1 feedback class
PtrFeedbackClass id=0
  accelNum is 2
  accelDenom is 1
  threshold is 4

Two finger scrolling as seen with libinput-debug-events:

event13   POINTER_AXIS      +2.73s    vert 6.07* horiz 0.49*
event13   POINTER_AXIS      +2.74s    vert 6.56* horiz 0.49*
event13   POINTER_AXIS      +2.75s    vert 6.89* horiz 0.82*

xev only sees pairs of button events:

ButtonPress event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291002, (106,47), root:(144,149),
    state 0x10, button 5, same_screen YES

ButtonRelease event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291002, (106,47), root:(144,149),
    state 0x1010, button 5, same_screen YES

ButtonPress event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291044, (106,47), root:(144,149),
    state 0x10, button 5, same_screen YES

ButtonRelease event, serial 33, synthetic NO, window 0x2800001,
    root 0xd6, subw 0x0, time 14291044, (106,47), root:(144,149),
    state 0x1010, button 5, same_screen YES

[totaam]

FWIW: wmi exposes input device information:

import wmi
w = wmi.WMI()
[(x.Name, x.Description, x.HardwareType, x.PointingType) for x in list(w.InstancesOf("Win32_PointingDevice"))]

For more information on device fields, see WMI Tasks: Computer Hardware

[totaam]

xorg ML response from input maintainer Peter Hutterer: emit "smooth scrolling/pixel perfect" events: one option is to emulate a trackstick-like device and send a middle button mouse press before sending scroll events (and relative events otherwise). In libinput this will scroll automatically, in evdev it's a configuration script away (EmulateWheel and EmulateWheelButton). And that scroll type is 'smooth' (at least in libinput, for evdev see [https://bugs.freedesktop.org/show_bug.cgi?id=51533])

The other option is to emulate a touchpad and forward the touches as-is. Then you let userspace take care of converting 2 finger to scroll, etc. That works for android devices with screen, but obviously not for others.''

[totaam]

2017-02-24 04:17:20: antoine uploaded file main_v2.c (4.7 KiB)

uinput example

[totaam]

New thread about the same issue: Virtual Touch Events for XTEST. Includes this code sample: [/attachment/ticket/173/main_v2.c].

[totaam]

This will also help handling javascript wheel events better, see #1424#comment:3

[totaam]

See events client side: monitoring events (keyboard, mouse) in X using:

xinput test-xi2 --root

[totaam]

2017-05-03 07:27:36: antoine uploaded file xi-events.patch (33.7 KiB)

handle XI2 events

[totaam]

2017-05-03 17:25:18: antoine uploaded file xi-events-v2.patch (18.7 KiB)

updated patch for r15792

[totaam]

More links:

• libinput now requires axis resolutions for graphics tablets: libinput 1.6, evemu-describe must return a non-zero resolution for the device to work
• libinput scroll sources
• libevdev - creating uinput devices

Maybe we want to ensure that the keyboard layout is set per device and does not change: New udev property: XKB_FIXED_LAYOUT for keyboards that must not change layouts (but how do we set this without udev?)

Preparatory work in r15790 + r15792. The patch attached allows us to receive XINPUT2 events: raw touch, keyboard and pointer events. Still TODO:

• make sure the XI events have all the attributes we need (we'll need to inject the gtk.Window wrappers somehow)
• augment the packets to include the optional device-id we now have
• disable regular events when we have XI events enabled
• enumerate the devices and send them as part of the hello packet, send updates when the device topology changes
• and the big one: create the matching uinput devices on the server side
• other plaforms: also enumerate the devices and include the device id

[totaam]

More useful pointers:

• xi2reader.py: get_valuator_names
• cursor tracking
• XI2proto
• XI2 recipes
• events_x.cc: GetScrollOffsets
• XIDeviceEvent to KeySym

[totaam]

2017-05-04 11:58:01: antoine uploaded file xi-events-v3.patch (35.5 KiB)

updated patch with device discovery

[totaam]

This bug may prevent us from using synthetic events: XSendEvent() fails on XInput2 events

Other problems:

• device information and events have float values - those won't work with the bencoder
• raw events are better for us: they don't have acceleration, but the raw events also don't have the window ID in the event structure
• cooked events have the window ID, but it's the window ID of the window manager, not our window... To solve both, we could record all XI2 events until we get a "regular" event and then use the extra data. The problems with this approach if that the raw pointer / scroll events are fine grained and may not fire a "regular" event until much later... and some may be discarded altogether.

[totaam]

2017-05-04 18:01:03: antoine uploaded file xi-events-v4.patch (29.4 KiB)

updated patch for r15794

[totaam]

2017-05-05 17:30:04: antoine uploaded file xi-events-v5.patch (49.2 KiB)

updated patch

[totaam]

New issues:

• some other packets send the pointer position update (ie: window configure), those should now also provide the device id... except we don't have it since the event may not be triggered by a device event - PITA but we can probably just use the fallback code for this case without causing problems
• wayland doesn't send us XI events unless the pointer is in one of our windows (by design) - not a problem, saves us parsing events for nothing and then filtering them
• wayland sends all mouse events through a single "xwayland-pointer" slave device, making it useless for testing distinct device behaviour
• the way XI2_Window overrides events needs to be generalized - fugly code
• the XI2 event parser should provide the modifiers state (we copy it from the regular event for now)
• we should watch for structure events and only update the list of windows we query when we get an event (and we may also need to stop before the root window with some window managers?)
• trying to trigger the smooth scrolling using a laptop trackpad over an xterm window, all I see are lots of XI_Motion events with the occasional XI_ButtonPress + XI_ButtonRelease pair when we reach the threshold for a mousewheel event - this could be caused by wayland? (I was expecting touch events... but maybe we need raw events here?)

[totaam]

2017-05-06 17:59:30: antoine uploaded file xi-events-v7.patch (56.4 KiB)

better approach: register event handlers directly with the XI2 helper class

[totaam]

With the updated patch, we get the correct events for pointer motion and clicks (keyboard events remain to be done).

Now, the hard part: setting up the server side devices.

If we use uinput, we should ship a file in /etc/X11/xorg.conf.d/ to skip our virtual devices by default, to avoid creating loops when the client is running locally - we can match the device in an [ftp://www.x.org/pub/X11R7.7-RC1/doc/man/man5/xorg.conf.5.xhtml#heading9 inputclass] section using MatchVendor and then set Option "Ignore" "true"

The problem with uinput is that as raw as the xinput2 events are, they're not from the same level as uinput.. so some interpretation / un-translation is still going to be needed to re-inject them in such a way that libinput will generate the exact events we want.

But maybe we don't want to use uinput + libinput at all? We could also write our own input driver and feed it the events directly? This way we don't need elevated permissions to talk to /dev/uinput and we can just create our own private input socket. How would that work with dynamic devices though?

[totaam]

2017-05-07 11:42:52: antoine uploaded file xi-events-v8.patch (63.9 KiB)

updated patch: export all device properties

[totaam]

2017-05-07 17:33:47: antoine uploaded file xi-events-v9.patch (68.7 KiB)

updated patch: server know knows details of the device used for some events

[totaam]

With the latest patch, the server has all the information it needs about the input devices (device changes need to be propagated - easy) and the input events include the device id (most of them - easy to add).

As for simulating the devices on the server, these examples may help:

• evdevshift, in particular the setfacl workaround so we don't require root
• evmapd

The problem is that those examples work on "evdev" devices.. and we get the device information from the X11 server using the "xinput2" API... which is mapped from the kernel's evdev devices via the "libinput" or the "evdev" Xorg driver layer. It should be possible to retrieve the corresponding evdev device that libinput / evdev uses, but the events we'll receive from the X11 server will be "xinput2" events...

[totaam]

2017-05-08 08:04:00: antoine uploaded file 00-ignore-xpra-devices.conf (0.2 KiB)

tell Xorg to always ignore the virtual input devices we create

[totaam]

2017-05-08 08:04:38: antoine uploaded file 00-only-xpra-devices.conf (0.3 KiB)

tell Xdummy to add our virtual input devices

[totaam]

2017-05-08 08:05:35: antoine uploaded file 61-evdev-xpra.rules (0.0 KiB)

change the udev permissions on input devices

[totaam]

2017-05-08 11:03:40: antoine uploaded file xi-events-v10.patch (79.9 KiB)

use uinput to inject events

[totaam]

Installing:

• [/attachment/ticket/173/00-ignore-xpra-devices.conf] into /etc/X11/xorg.conf.d/
• [/attachment/ticket/173/00-only-xpra-devices.conf] into ~/.xpra/xorg.conf.d (maybe we should move to /etc/xpra/xorg.conf.d/ instead?)
• [/attachment/ticket/173/61-evdev-xpra.rules] to /etc/udev/rules.d/ and then running udevadm control --reload-rules
• this change to the xorg.conf we ship:

  Option "AutoEnableDevices" "true"
  Option "AutoAddDevices" "true"

---

udev will create input devices that a non-root Xorg can access. (we should probably use group access instead of 666 permisions, and restrict the changes to our virtual input devices, not all input devices... see udev rules: controlling permissions and ownership) The default Xorg server will ignore our virtual input devices:

/usr/libexec/gdm-x-session[9751]: (II) config/udev: Adding input device Fake xpra pointer (/dev/input/event17)
/usr/libexec/gdm-x-session[9751]: (**) Fake xpra pointer: Ignoring device from InputClass "Xpra devices should not be used automatically"

but Xdummy will add and enable them:

[ 96341.568] (II) config/udev: Adding input device Fake xpra pointer (/dev/input/event17)
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "evdev pointer catchall"
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "libinput pointer catchall"
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "All devices"
[ 96341.568] (**) Fake xpra pointer: Applying InputClass "Xpra devices should not be used automatically"
[ 96341.568] (II) Using input driver 'libinput' for 'Fake xpra pointer'
[ 96341.568] (**) Fake xpra pointer: always reports core events
[ 96341.568] (**) Option "Device" "/dev/input/event17"
[ 96341.568] (**) Option "_source" "server/udev"
[ 96341.569] (II) event17 - (II) Fake xpra pointer: (II) is tagged by udev as: Mouse
[ 96341.569] (II) event17 - (II) Fake xpra pointer: (II) device is a pointer
[ 96341.569] (II) event17 - (II) Fake xpra pointer: (II) device removed
[ 96341.584] (**) Option "config_info" "udev:/sys/devices/virtual/input/input68/event17"
[ 96341.584] (II) XINPUT: Adding extended input device "Fake xpra pointer" (type: MOUSE, id 6)
[ 96341.584] (**) Option "AccelerationScheme" "none"
[ 96341.584] (**) Fake xpra pointer: (accel) selected scheme none/0
[ 96341.584] (**) Fake xpra pointer: (accel) acceleration factor: 2.000
[ 96341.584] (**) Fake xpra pointer: (accel) acceleration threshold: 4

With this in place, I can simulate devices via uinput and only Xdummy sees the events. This can be verified easily with:

xpra info  | grep -i cursor.position

For injecting events, maybe python-evdev will be more useful than python-uinput? (at least the capabilities are better documented)

With the latest patch, I can inject relative motion... but not absolute. (mouse pointers don't do absolute) Looks like this needs more options to be able to configure the virtual devices properly (see min==max "kernel bug"):

[107288.819] (II) config/udev: Adding input device Logitech M705 - xpra pointer (/dev/input/event17)
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "evdev pointer catchall"
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "libinput pointer catchall"
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "All devices"
[107288.819] (**) Logitech M705 - xpra pointer: Applying InputClass "Xpra devices should not be used automatically"
[107288.819] (II) Using input driver 'libinput' for 'Logitech M705 - xpra pointer'
[107288.819] (**) Logitech M705 - xpra pointer: always reports core events
[107288.819] (**) Option "Device" "/dev/input/event17"
[107288.819] (**) Option "_source" "server/udev"
[107288.820] (II) event17 - (II) Logitech M705 - xpra pointer: (II) is tagged by udev as: Mouse
[107288.820] (EE) event17 - (EE) Logitech M705 - xpra pointer: (EE) kernel bug: device has min == max on ABS_X
[107288.820] (II) event17 - (II) Logitech M705 - xpra pointer: (II) was rejected
[107288.844] (II) event17 - not using input device '/dev/input/event17'.
[107288.844] (EE) libinput: Logitech M705 - xpra pointer: Failed to create a device for /dev/input/event17
[107288.844] (EE) PreInit returned 2 for "Logitech M705 - xpra pointer"
[107288.844] (II) UnloadModule: "libinput"
[107288.845] (II) config/udev: Adding input device DLL0704:01 06CB:76AE Touchpad - xpra pointer (/dev/input/event18)
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "evdev pointer catchall"
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "libinput pointer catchall"
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "All devices"
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: Applying InputClass "Xpra devices should not be used automatically"
[107288.845] (II) Using input driver 'libinput' for 'DLL0704:01 06CB:76AE Touchpad - xpra pointer'
[107288.845] (**) DLL0704:01 06CB:76AE Touchpad - xpra pointer: always reports core events
[107288.845] (**) Option "Device" "/dev/input/event18"
[107288.845] (**) Option "_source" "server/udev"
[107288.847] (II) event18 - (II) DLL0704:01 06CB:76AE Touchpad - xpra pointer: (II) is tagged by udev as: Mouse
[107288.847] (EE) event18 - (EE) DLL0704:01 06CB:76AE Touchpad - xpra pointer: (EE) kernel bug: device has min == max on ABS_X
[107288.847] (II) event18 - (II) DLL0704:01 06CB:76AE Touchpad - xpra pointer: (II) was rejected
[107288.872] (II) event18 - not using input device '/dev/input/event18'.
[107288.872] (EE) libinput: DLL0704:01 06CB:76AE Touchpad - xpra pointer: Failed to create a device for /dev/input/event18
[107288.872] (EE) PreInit returned 2 for "DLL0704:01 06CB:76AE Touchpad - xpra pointer"
[107288.872] (II) UnloadModule: "libinput"

[totaam]

2017-05-08 11:45:51: antoine uploaded file xi-events-v11.patch (79.0 KiB)

switch to using python-evdev

[totaam]

Mostly merged in r15802 (big), this doesn't do anything useful yet and is not enabled unless the client specifies --input-devices=xi.

With "xi" enabled, the server gets a list of devices, which is not used for anything yet, and the pointer packets include the device responsible for the event (as well as more details on the event), ie:

process_mouse_common from device=Logitech G400s Optical Gaming Mouse

This will have to do for this release, at least the xinput code is merged and won't bitrot too much. Enabling uinput devices for touch events only should work, but the better approach is to write a new xorg input driver so we can inject the events exactly as we receive them.

[totaam]

This may help with MS Windows to get the device ID and layout for each keyboard device:

$ python -c "from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_Keyboard')
print([(x.Properties_['DeviceID'].Value, x.Properties_['Layout'].Value) for x in res])
[(u'ACPI\\PNP0303\\4&1D401FB5&0', u'00000809')]

[totaam]

2017-07-30 19:05:24: antoine uploaded file uinput-server.patch (18.5 KiB)

implement pointer as a uinput device

[totaam]

Moved the uinput server-side support to #1611.

We can keep this ticket as a tracker.

[totaam]

r16721 uses xinput2 devices on posix clients by default.

[totaam]

See #1615 for touch device support