search

gvalkov / python-evdev

Python bindings for the Linux input subsystem
https://python-evdev.rtfd.org/
BSD 3-Clause "New" or "Revised" License
334 stars 112 forks source link

Wayland behavior different from X11? #186

Closed RedBearAK closed 1 year ago

RedBearAK commented 1 year ago

I'm trying to get a keymapper app to work with Wayland the same way it works with X11, to access the window attributes of the focused window and provide app-specific shortcut remappings.

I should note that I didn't have anything to do with creating the keymapper (keyszer) or the app it was forked from (xkeysnail), and I barely understand how they work. I'm just trying to do some minor adaptation of keyszer to get it to work with Wayland. Specifically, with Wayland/GNOME, via communicating with a GNOME shell extension over DBus.

The good news is that I was able to obtain the window attributes and the per-application matching is working.

The bad news is that I'm getting behavior from the virtual keyboard (which I understand is created via evdev) that I've never seen in X11. The logic of the keymapper doesn't seem to be at fault. The keystroke events that show up in the log are the same as when the app is running in X11.

What's happening is odd. This is just an example: Keyszer has the ability to process strings into keystroke output, to "type" macros out with the virtual keyboard. So you can have it type a string like this:

You have double-tapped the Grave key!

It will get translated into things like Shift-Y, O, U, Space, etc.

But when I attempt this kind of output, triggered by an input shortcut, in Wayland, I get this: 

you have double-tapped the grave key1

The keystrokes for pressing Shift, typing and releasing keys like "y" and "g" and "1", and releasing Shift, are all there as they are supposed to be, in the log. Adding debugging output shows the key events getting triggered just as they would normally be in X11. But things just aren't coming out right in Wayland. Toggling CapsLock has no effect on the output, unlike X11 where it will reverse the shifted state of normal keys in the string.

This issue seems to be affecting the modifier keys, while leaving the regular part of the keyboard alone. Thus the result being all lowercase, instead of mixed-case. Other shortcuts also don't work, as the app seems to see different modifiers than it should.

The physical device and virtual device seem to be showing up just fine in Wayland, in libinput list-devices: 

Device:           AT Translated Set 2 keyboard
Kernel:           /dev/input/event1
Group:            3
Seat:             seat0, default
Capabilities:     keyboard 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      n/a
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Accel profiles:   n/a
Rotation:         n/a
Device:           Keyszer (virtual) Keyboard
Kernel:           /dev/input/event6
Group:            5
Seat:             seat0, default
Capabilities:     keyboard pointer 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      disabled
Nat.scrolling:    disabled
Middle emulation: disabled
Calibration:      n/a
Scroll methods:   button
Click methods:    none
Disable-w-typing: n/a
Accel profiles:   flat *adaptive
Rotation:         n/a

I'm just wondering if anyone here might have insight into what's happening here. As far as we know, the logic of the keymapper when catching input combos and transforming them into output combos isn't doing anything wrong. The keyboard device is being grabbed like it normally is, and the virtual keyboard is being created and used. But the results are different than expected.

Keymapper app: https://github.com/joshgoebel/keyszer

sezanzeb commented 1 year ago

As far as I remember keyboard handling in Wayland is individual to each desktop environment. Have you tried it in both gnome and kde on wayland?

RedBearAK commented 1 year ago

Have you tried it in both gnome and kde on wayland?

@sezanzeb

No, and I don't know that there is a method yet to obtain the window information in KDE/Wayland, so that might be slightly pointless. Although maybe I could test it with a completely generic keymap that doesn't care about the window context. For now I've been concentrating on getting this DBus to GNOME shell extension thing working. My config is from Kinto.sh, which does a lot of app-specific shortcut remapping to make everything work like a Mac.

https://github.com/rbreaves/kinto/pull/750/files

As far as I remember keyboard handling in Wayland is individual to each desktop environment.

So Wayland may be somehow interfering with the output from the virtual keyboard? Is that what you're saying? What can I do to troubleshoot this further?

sezanzeb commented 1 year ago

As far as I remember keyboard handling in Wayland is individual to each desktop environment.

So Wayland may be somehow interfering with the output from the virtual keyboard? Is that what you're saying? What can I do to troubleshoot this further?

My understanding is very limited. But from what I know wayland is just a protocol, and gnome and kde have to implement all of the stuff themselves.

Is it possible to add some delay between the injected keystrokes? I once had problems when keycodes were injected too fast.

sezanzeb commented 1 year ago

The output form sudo evtest might be interesting when reproducing the problem in wayland

RedBearAK commented 1 year ago

Is it possible to add some delay between the injected keystrokes? I once had problems when keycodes were injected too fast.

Yes, there are a couple of places to do that. I've actually had quite a few problems with "macros" (sets of multiple keystrokes all emitted at once) for quite a long time. Do you have any references for what causes that problem? Is it a defect in libinput or some kind of device buffer? We've had a really hard time figuring out why that happens.

The output form sudo evtest might be interesting when reproducing the problem in wayland

I did try that, but the output in the same environment seems... normal? Although I'm not exactly sure what to look for. 

No device specified, trying to scan all of /dev/input/event*
Available devices:
/dev/input/event0:  Power Button
/dev/input/event1:  AT Translated Set 2 keyboard
/dev/input/event2:  VirtualPS/2 VMware VMMouse
/dev/input/event3:  VirtualPS/2 VMware VMMouse
/dev/input/event4:  QEMU QEMU USB Tablet
/dev/input/event5:  spice vdagent tablet
Select the device event number [0-5]: 1
Input driver version is 1.0.1
Input device ID: bus 0x11 vendor 0x1 product 0x1 version 0xab41
Input device name: "AT Translated Set 2 keyboard"
Supported events:
  Event type 0 (EV_SYN)
  Event type 1 (EV_KEY)
    Event code 1 (KEY_ESC)
    Event code 2 (KEY_1)
    Event code 3 (KEY_2)
    Event code 4 (KEY_3)
    Event code 5 (KEY_4)
    Event code 6 (KEY_5)
    Event code 7 (KEY_6)
    Event code 8 (KEY_7)
    Event code 9 (KEY_8)
    Event code 10 (KEY_9)
    Event code 11 (KEY_0)
    Event code 12 (KEY_MINUS)
    Event code 13 (KEY_EQUAL)
    Event code 14 (KEY_BACKSPACE)
    Event code 15 (KEY_TAB)
    Event code 16 (KEY_Q)
    Event code 17 (KEY_W)
    Event code 18 (KEY_E)
    Event code 19 (KEY_R)
    Event code 20 (KEY_T)
    Event code 21 (KEY_Y)
    Event code 22 (KEY_U)
    Event code 23 (KEY_I)
    Event code 24 (KEY_O)
    Event code 25 (KEY_P)
    Event code 26 (KEY_LEFTBRACE)
    Event code 27 (KEY_RIGHTBRACE)
    Event code 28 (KEY_ENTER)
    Event code 29 (KEY_LEFTCTRL)
    Event code 30 (KEY_A)
    Event code 31 (KEY_S)
    Event code 32 (KEY_D)
    Event code 33 (KEY_F)
    Event code 34 (KEY_G)
    Event code 35 (KEY_H)
    Event code 36 (KEY_J)
    Event code 37 (KEY_K)
    Event code 38 (KEY_L)
    Event code 39 (KEY_SEMICOLON)
    Event code 40 (KEY_APOSTROPHE)
    Event code 41 (KEY_GRAVE)
    Event code 42 (KEY_LEFTSHIFT)
    Event code 43 (KEY_BACKSLASH)
    Event code 44 (KEY_Z)
    Event code 45 (KEY_X)
    Event code 46 (KEY_C)
    Event code 47 (KEY_V)
    Event code 48 (KEY_B)
    Event code 49 (KEY_N)
    Event code 50 (KEY_M)
    Event code 51 (KEY_COMMA)
    Event code 52 (KEY_DOT)
    Event code 53 (KEY_SLASH)
    Event code 54 (KEY_RIGHTSHIFT)
    Event code 55 (KEY_KPASTERISK)
    Event code 56 (KEY_LEFTALT)
    Event code 57 (KEY_SPACE)
    Event code 58 (KEY_CAPSLOCK)
    Event code 59 (KEY_F1)
    Event code 60 (KEY_F2)
    Event code 61 (KEY_F3)
    Event code 62 (KEY_F4)
    Event code 63 (KEY_F5)
    Event code 64 (KEY_F6)
    Event code 65 (KEY_F7)
    Event code 66 (KEY_F8)
    Event code 67 (KEY_F9)
    Event code 68 (KEY_F10)
    Event code 69 (KEY_NUMLOCK)
    Event code 70 (KEY_SCROLLLOCK)
    Event code 71 (KEY_KP7)
    Event code 72 (KEY_KP8)
    Event code 73 (KEY_KP9)
    Event code 74 (KEY_KPMINUS)
    Event code 75 (KEY_KP4)
    Event code 76 (KEY_KP5)
    Event code 77 (KEY_KP6)
    Event code 78 (KEY_KPPLUS)
    Event code 79 (KEY_KP1)
    Event code 80 (KEY_KP2)
    Event code 81 (KEY_KP3)
    Event code 82 (KEY_KP0)
    Event code 83 (KEY_KPDOT)
    Event code 85 (KEY_ZENKAKUHANKAKU)
    Event code 86 (KEY_102ND)
    Event code 87 (KEY_F11)
    Event code 88 (KEY_F12)
    Event code 89 (KEY_RO)
    Event code 90 (KEY_KATAKANA)
    Event code 91 (KEY_HIRAGANA)
    Event code 92 (KEY_HENKAN)
    Event code 93 (KEY_KATAKANAHIRAGANA)
    Event code 94 (KEY_MUHENKAN)
    Event code 95 (KEY_KPJPCOMMA)
    Event code 96 (KEY_KPENTER)
    Event code 97 (KEY_RIGHTCTRL)
    Event code 98 (KEY_KPSLASH)
    Event code 99 (KEY_SYSRQ)
    Event code 100 (KEY_RIGHTALT)
    Event code 102 (KEY_HOME)
    Event code 103 (KEY_UP)
    Event code 104 (KEY_PAGEUP)
    Event code 105 (KEY_LEFT)
    Event code 106 (KEY_RIGHT)
    Event code 107 (KEY_END)
    Event code 108 (KEY_DOWN)
    Event code 109 (KEY_PAGEDOWN)
    Event code 110 (KEY_INSERT)
    Event code 111 (KEY_DELETE)
    Event code 112 (KEY_MACRO)
    Event code 113 (KEY_MUTE)
    Event code 114 (KEY_VOLUMEDOWN)
    Event code 115 (KEY_VOLUMEUP)
    Event code 116 (KEY_POWER)
    Event code 117 (KEY_KPEQUAL)
    Event code 118 (KEY_KPPLUSMINUS)
    Event code 119 (KEY_PAUSE)
    Event code 121 (KEY_KPCOMMA)
    Event code 122 (KEY_HANGUEL)
    Event code 123 (KEY_HANJA)
    Event code 124 (KEY_YEN)
    Event code 125 (KEY_LEFTMETA)
    Event code 126 (KEY_RIGHTMETA)
    Event code 127 (KEY_COMPOSE)
    Event code 128 (KEY_STOP)
    Event code 140 (KEY_CALC)
    Event code 142 (KEY_SLEEP)
    Event code 143 (KEY_WAKEUP)
    Event code 155 (KEY_MAIL)
    Event code 156 (KEY_BOOKMARKS)
    Event code 157 (KEY_COMPUTER)
    Event code 158 (KEY_BACK)
    Event code 159 (KEY_FORWARD)
    Event code 163 (KEY_NEXTSONG)
    Event code 164 (KEY_PLAYPAUSE)
    Event code 165 (KEY_PREVIOUSSONG)
    Event code 166 (KEY_STOPCD)
    Event code 172 (KEY_HOMEPAGE)
    Event code 173 (KEY_REFRESH)
    Event code 183 (KEY_F13)
    Event code 184 (KEY_F14)
    Event code 185 (KEY_F15)
    Event code 217 (KEY_SEARCH)
    Event code 226 (KEY_MEDIA)
  Event type 4 (EV_MSC)
    Event code 4 (MSC_SCAN)
  Event type 17 (EV_LED)
    Event code 0 (LED_NUML) state 0
    Event code 1 (LED_CAPSL) state 0
    Event code 2 (LED_SCROLLL) state 0
Key repeat handling:
  Repeat type 20 (EV_REP)
    Repeat code 0 (REP_DELAY)
      Value    250
    Repeat code 1 (REP_PERIOD)
      Value     33
Properties:
Testing ... (interrupt to exit)
Event: time 1676243377.373547, type 4 (EV_MSC), code 4 (MSC_SCAN), value 1c
Event: time 1676243377.373547, type 1 (EV_KEY), code 28 (KEY_ENTER), value 0
Event: time 1676243377.373547, -------------- SYN_REPORT ------------
Event: time 1676243381.192914, type 4 (EV_MSC), code 4 (MSC_SCAN), value 3a
Event: time 1676243381.192914, type 1 (EV_KEY), code 58 (KEY_CAPSLOCK), value 1
Event: time 1676243381.192914, -------------- SYN_REPORT ------------
Event: time 1676243381.193177, type 17 (EV_LED), code 1 (LED_CAPSL), value 1
Event: time 1676243381.193177, -------------- SYN_REPORT ------------
Event: time 1676243381.298976, type 4 (EV_MSC), code 4 (MSC_SCAN), value 3a
Event: time 1676243381.298976, type 1 (EV_KEY), code 58 (KEY_CAPSLOCK), value 0
Event: time 1676243381.298976, -------------- SYN_REPORT ------------
Event: time 1676243382.196118, type 4 (EV_MSC), code 4 (MSC_SCAN), value 2a
Event: time 1676243382.196118, type 1 (EV_KEY), code 42 (KEY_LEFTSHIFT), value 1
Event: time 1676243382.196118, -------------- SYN_REPORT ------------
Event: time 1676243382.315390, type 4 (EV_MSC), code 4 (MSC_SCAN), value 2a
Event: time 1676243382.315390, type 1 (EV_KEY), code 42 (KEY_LEFTSHIFT), value 0
Event: time 1676243382.315390, -------------- SYN_REPORT ------------
Event: time 1676243384.964422, type 4 (EV_MSC), code 4 (MSC_SCAN), value 38
Event: time 1676243384.964422, type 1 (EV_KEY), code 56 (KEY_LEFTALT), value 1
Event: time 1676243384.964422, -------------- SYN_REPORT ------------
Event: time 1676243385.061679, type 4 (EV_MSC), code 4 (MSC_SCAN), value 38
Event: time 1676243385.061679, type 1 (EV_KEY), code 56 (KEY_LEFTALT), value 0
Event: time 1676243385.061679, -------------- SYN_REPORT ------------
Event: time 1676243386.718295, type 4 (EV_MSC), code 4 (MSC_SCAN), value db
Event: time 1676243386.718295, type 1 (EV_KEY), code 125 (KEY_LEFTMETA), value 1
Event: time 1676243386.718295, -------------- SYN_REPORT ------------
Event: time 1676243386.824009, type 4 (EV_MSC), code 4 (MSC_SCAN), value db
Event: time 1676243386.824009, type 1 (EV_KEY), code 125 (KEY_LEFTMETA), value 0
Event: time 1676243386.824009, -------------- SYN_REPORT ------------
Event: time 1676243389.384827, type 4 (EV_MSC), code 4 (MSC_SCAN), value 01
Event: time 1676243389.384827, type 1 (EV_KEY), code 1 (KEY_ESC), value 1
Event: time 1676243389.384827, -------------- SYN_REPORT ------------
Event: time 1676243389.485497, type 4 (EV_MSC), code 4 (MSC_SCAN), value 01
Event: time 1676243389.485497, type 1 (EV_KEY), code 1 (KEY_ESC), value 0
Event: time 1676243389.485497, -------------- SYN_REPORT ------------
Event: time 1676243391.247205, type 4 (EV_MSC), code 4 (MSC_SCAN), value 1d
Event: time 1676243391.247205, type 1 (EV_KEY), code 29 (KEY_LEFTCTRL), value 1
Event: time 1676243391.247205, -------------- SYN_REPORT ------------
Event: time 1676243391.371776, type 4 (EV_MSC), code 4 (MSC_SCAN), value 1d
Event: time 1676243391.371776, type 1 (EV_KEY), code 29 (KEY_LEFTCTRL), value 0
Event: time 1676243391.371776, -------------- SYN_REPORT ------------
Event: time 1676243398.660274, type 4 (EV_MSC), code 4 (MSC_SCAN), value 3a
Event: time 1676243398.660274, type 1 (EV_KEY), code 58 (KEY_CAPSLOCK), value 1
Event: time 1676243398.660274, -------------- SYN_REPORT ------------
Event: time 1676243398.754692, type 4 (EV_MSC), code 4 (MSC_SCAN), value 3a
Event: time 1676243398.754692, type 1 (EV_KEY), code 58 (KEY_CAPSLOCK), value 0
Event: time 1676243398.754692, -------------- SYN_REPORT ------------
Event: time 1676243398.754961, type 17 (EV_LED), code 1 (LED_CAPSL), value 0
Event: time 1676243398.754961, -------------- SYN_REPORT ------------
RedBearAK commented 1 year ago

Wait, adding a 0.05s delay to a part of the output module does have a significant effect. As in, it makes the intended transformed set of keystrokes "work" as they should. 

    def __send_sync(self):
        time.sleep(0.05)
        _uinput.syn()

But there is still the very strange problem where the input combo, that the app window is never supposed to "see", is still being "seen" in addition to the transformed keystrokes. I've been working with this keymapper for a couple of years in X11 environments and have never seen this phenomenon. Which is why I came over here to see if there is a chance the input is not being successfully isolated from Wayland when the keymapper "grabs" the device, which has always worked fine in X11.

RedBearAK commented 1 year ago

Alright, I slowed things WAAAY down with a 0.5s delay, and got this: 

WWhhaatt  tthhee  hheecckk

The macro string is supposed to be: 

What the heck

The CapsLock key that triggers the macro is "seen" by the app immediately (I see an on-screen notification for CapsLock and NumLock keys) and then this macro string is slowly typed out, with each character typed twice. Maybe because the long delay is before sending a sync, and something is interpreting as the key being tapped again? But there are two capital "W", which would require Shift-W in both cases.

I'm not completely familiar with how the event code sequences work, and why this long delay causes this doubling up of the characters.

RedBearAK commented 1 year ago

Issue with CapsLock was a red herring, produced by the Boxes application not doing a true "grab" of my real keyboard to isolate CapsLock key presses from the host OS.

So what I'm left with is that in two cases (Kubuntu/X11/KDE in a VM, and now Ubuntu/Wayland/GNOME in a VM) I have extreme unreliability of transformed keyboard shortcuts without adding a significant time delay to the key events coming from the virtual keyboard.

I think I'll close this because the premise was a mistake. The problem is different.

But that doubling of the characters, that' s a clue about something...

sezanzeb commented 1 year ago

The output form sudo evtest might be interesting when reproducing the problem in wayland

I did try that, but the output in the same environment seems... normal? Although I'm not exactly sure what to look for.

the output of the macro would have been interesting. The sequence of inputs that causes "you have double-tapped the grave key1" to be written.