xkeysnail/keyszer, Ulauncher, Alt+F1, and keyboard focus issues

[RedBearAK]

@rbreaves @joshgoebel

This is an odd thing I would like to document. Maybe it will lead to an improvement at some point.

When I'm running Kinto with either keymapper (xkeysnail or keyszer) and try to activate Ulauncher's dialog window (a Spotlight clone like Albert), I have often experienced problems with the Ulauncher dialog failing to acquire the keyboard focus.

I've seen this in both Ubuntu and Fedora, with either xkeysnail or keyszer running. I can't say this never, ever happens without a keymapper running, but it's an extremely rare event. When a keymapper is running I can often trigger the configured Ulauncher shortcut many times and observe the dialog failing to get the keyboard focus numerous times in a row.

Obviously this makes it fairly useless as a quick keyboard-based launcher, if I have to click on the dialog to start typing more often than not.

Ulauncher has a file named ulauncher-toggle that's just a script that uses a dbus command, and then has a very short sleep delay and tries to use wmctrl to give the focus to the Ulauncher dialog that pops up. Normally this works fine. In the case where the keymapper is running, nothing I added to that script would get it to work reliably. Changing the delay time, even looping the wmctrl command that should give the focus to the Ulauncher dialog. I tried many things.

Strangely, I was able to make a script that runs independently, watches the window list in a loop (via wmctrl -l), and gives the focus to the Ulauncher dialog within 0.1 seconds whenever it shows up in the list. Works perfectly to control the problem. Uses the exact same commmand that is in the ulauncher-toggle script. Why doesn't something similar work from within the ulauncher-toggle script? No idea.

Fortunately there seems to be a simpler way to bypass the issue, without using up tons of PIDs with a looping shell command. I set up a custom keyboard shortcut in the GNOME keyboard settings that runs the ulauncher-toggle command directly. I left the keyboard shortcut the same, so Kinto's config is still remapping Cmd+Space onto Alt+F1, which now triggers the GNOME custom shortcut. Works perfectly. No more issue with keyboard focus.

Same thing happens if I set Ulauncher to bind to a shortcut that doesn't get remapped by the keymapper. Even if the keymapper was running, as long as it wasn't actually the keymapper/evdev that was sending the combo as output, it would work perfectly. If I just disable this line in the Kinto config and have Ulauncher bind to Ctrl+Space instead of Alt+F1, there's no problem.

    # C("RC-Space"):              C("Alt-F1"),                    # Default SL - Launch Application Menu (gnome/kde)

So there is some sort of interaction between A) the keymapper sending a transformed combo, and B) whatever method the Ulauncher process uses to bind to that shortcut, and C) that somehow interferes with Ulauncher's ability to pop up it's dialog window and grab the keyboard focus.

Or... Maybe it's as simple as Ulauncher not liking the Alt+F1 shortcut (but only when sent from the keymapper, I've tested using Alt+F1 without the keymapper running and that was also never a problem). If I change the remapped shortcut to something other than the default Alt+F1 it seems to be yet another way to bypass the issue:

    # C("RC-Space"):              C("Alt-F1"),                    # Default SL - Launch Application Menu (gnome/kde)
    C("RC-Space"):              C("Shift-C-F3"),                    # Default SL - Launch Application Menu (gnome/kde)

I've often had issues getting certain applications to accept Alt+F1 and a couple of other shortcuts as new shortcuts while Kinto was running, even if I press the remapped Alt key and the literal F1 key, rather than Cmd+Space. They just don't get detected properly. That must be somehow at the heart of this.

So the solutions I've found are:

• Change the remapped output shortcut from Alt+F1 to something else, like Shift+Ctrl+F3, and have Ulauncher bind to that.
• Have the remapped Alt+F1 trigger a custom system shortcut that just runs ulauncher-toggle directly. Change the shortcut Ulauncher binds to something unrelated.
• Disable the Ctrl+Space to Alt+F1 remap entirely, have Ulauncher bind to Ctrl+Space.

I think the last one may be problematic depending on the DE.

[joshgoebel]

Weird. I have no idea what would be special about Alt-F1 as a combo.

[RedBearAK]

@rbreaves @joshgoebel

I think this actually might be a more general problem with new windows failing to receive the keyboard focus while the keymapper is running and processing combos. I'm frequently experiencing things like the Properties dialog in Nautilus popping up without being the "front" window. Some keys (arrows) still make things happen in the unfocused Properties dialog, but I can see in the terminal that the keymapper is not picking up the new window as being active (because it isn't, it's gray like any other "background" window).

I think this really needs to be looked into, but at this point I have no idea what exactly to look at.

When I disable the keymapper and use the normal shortcut to invoke the same kinds of windows, they ALWAYS have the keyboard focus. Or, with the keymapper running, if I use the shortcut that isn't getting remapped, once again there is no window focus issue. The window focus issue only appears when I use one of the remapped shortcuts.

    C("RC-i"):                  C("Alt-Enter"),                 # File properties dialog (Get Info)

So in this instance, if I use Cmd+I to invoke Properties, the window that appears frequently doesn't have the focus. But if I only use Alt+Enter, the window will have the focus. I can do it over and over again and it always has the focus.

This is a problem since it interferes with the flow of having keyboard shortcuts properly control windows without needing to fix which window has focus by clicking or using Alt+Tab/Alt+Grave to get the new window in focus.

[RedBearAK]

Update:

A partial bypass of this issue is to install a GNOME extension like "Grand Theft Focus" which completely disables the "focus stealing prevention" feature of GNOME. This allows the Ulauncher window to always have focus, even when it's being activated by a transformed keyboard shortcut processed through the keymapper.

A side effect of disabling GNOME's focus stealing prevention in this way is that if any background window has an error dialog or something similar suddenly appear, that whole window with its new dialog will pop to the front of the window stack automatically, and cover the window you're working with. It's subjective whether this is a positive or negative side effect.

This "fix" works on Ulauncher but doesn't work on the dialogs (like Nautilus file properties) that don't get focus when activated with a transformed shortcut.

[joshgoebel]

Someone really should make a "record and playback" utill.. record real keystrokes... play them back with a virtual keyboard (on some sort of timer or something)... and observe the behavior... rule out whether this is some problem with virtual devices or just some issue with the key timing itself.

[RedBearAK]

@joshgoebel

Well, it's not quite the same thing, but I tried manipulating my output.py tweak from 1/1000s to longer and longer delays. The focus issue starts to go away at around 1/20s, and is almost completely gone with a delay of 1/10s. Of course that's a relative eternity and starts to play havoc with just regular usage of the keyboard.

This seems remarkably similar to the problem I was having getting Unicode keystrokes to successfully complete in KDE without a 1/10s sleep delay.

Seems like something, somewhere really doesn't like receiving keystrokes from the virtual keyboard too quickly. I can try to fast-type the regular direct shortcuts that don't get transformed and can't get the focus issue to occur, no matter how fast the physical shortcut keys are typed together. I'm definitely able to hit Alt and Enter virtually simultaneously (so there can't be more than a couple of milliseconds between them) and the shortcut still works, without the dialog having a focus problem.

Maybe there's some sort of kernel cycle time element that doesn't affect physical keyboards but is messing with the virtual keyboard. Very wild guess.

[joshgoebel]

Maybe there's some sort of kernel cycle time element that doesn't affect physical keyboards but is messing with the virtual keyboard.

Then we might as well throw in the towel and give up.

I'm definitely able to hit Alt and Enter virtually simultaneously (so there can't be more than a couple of milliseconds between them)

It's probably longer than you think.... hence my suggesting to record the exactly sequence, then play it back... and see what behavior you get... if the exact sequence played back was broken then that would be pretty conclusive I think. It's also possible the sequencing (or holds/releases) matters and is subtly different.

---

Ok, maybe we can do this just a slightly harder way... lets start by becoming just a mindless proxy of the real keyboard:

def on_event(event, device):
  _output.send_event(event)
  return

If you made that change and ran it - what behavior would you get? Would your virtual "proxy" of the physical always work - or will it be broken in weird ways? We would HOPE to see zero behavioral differences.

[RedBearAK]

@joshgoebel

I think I did what you were suggesting, in transform.py, but the result didn't work well. Not sure what result you were expecting from that, other than nothing being transformed anymore.

Is it evdev that's responsible for creating the virtual keyboard, or does it use some other module in turn? Maybe I can go ask whoever is closest to the virtual keyboard code if they have any clue about this discrepancy in behavior between the physical and virtual keyboard devices.

[joshgoebel]

other than nothing being transformed anymore.

Right... exactly... you have to test the original non-transformed keystrokes (to get the simplest possible test)... but send thru the virtual keyboard... so just hit whatever the actual keys and combos are.

Alt-F1 sounds like a perfectly easy combo to test over and over.

• Does it always work?
• Or does it break sporadically?

In this way you can rule out the virtual keyboard being the problem (if it always worked), or prove it (or some configuration detail) is.

[RedBearAK]

have to test the original non-transformed keystrokes

Right, I kind of figured out what you meant to test with that. I tried with the problematic Properties dialog, using the standard Alt+Enter while the patch was in place. There's no problem with focus. So... I guess that means maybe the issue is in how the keymapper is issuing key events after the transform? Not necessarily a problem with the virtual keyboard device itself?

Interesting.

OK, also tested Alt+F1 to directly activate Ulauncher, both before and after disabling the GNOME extension to re-enable the focus stealing prevention (have to also restart the GNOME shell to get it fully working again). Can't make the focus problem happen.

If this is really all going through the virtual keyboard just from making that one small change to the code, then this means... something. So this is really down to something that happens (or doesn't happen) during or after transforming the key events.

[joshgoebel]

Next I'd suggest adding some logging to output#send_event so you can see the exact key events timestamps... and we'd have to carefully review that log vs the key mapper log...

There are several details/differences still lurking here... send_key_action is sending a sync event after every key... and the transform code is simply stripping all the [I forget the name events - the ones with the scan codes?], but run evtest and you'll see them... so still could be lots of different nuances. Maybe scan codes matter sometimes?

Ultimately though we need an "exact" replay system so you can edit the typing "program" - both the original one (that works) and the one we generate... and find out what the difference that matters is. You can probably already do this with custom macros just by getting the raw _uinput device then coding against it using write.

# program here
# delay a bit just to breath
_uinput.write( #...
_uinput.write( #... 
_uinput.write( # ...
_uinput.write( # ...
_uinput.write( # ...
_uinput.write( # ...

So you could have a key mapped that just ran your custom macro that did the low-level typing... I'd be very careful to start with a combo that allowed a "clean state" for the macro... maybe like mapping cmd-F1 -> alt-f1... such that after keyszer releases CMD... the keyboard state should be "clean" for your alt-f1 code to run... (or running with the smart hibernate on should be even better)... Of course you really aren't mapping to Alt-F1, you'd be mapping to your custom program that was writing the key codes directly to the output.

[RedBearAK]

Adding some logging I can do, and I understand the gist of what you're getting at. But I'd need more context to know what to do with the rest of that.

Is it correct to say that when the keymapper is active, all keystrokes are actually going through the virtual keyboard? Modmapped keys are transformed to other keys, and combos/keys in keymaps are transformed into other combos/keys, but isn't everything else (all the un-transformed keystrokes) still coming from the virtual keyboard device?

If that's the case, logging and looking carefully at what comes out with non-transformed combos versus transformed combos should offer a clue already, right?

[RedBearAK]

Here is an interesting difference I'm already seeing. The keymapper always issues a left shift key press/release when it starts. (This has never actually worked to set the keymap in place, I still have to type any modifier key in order to get everything to start working correctly. The same thing happens after quitting the keymapper.)

InputEvent(0, 0, 1, <Key.LEFT_SHIFT: 42>, <Action.PRESS: 1>)
InputEvent(0, 0, 1, <Key.LEFT_SHIFT: 42>, <Action.RELEASE: 0>)

But when I actually type a left shift on the keyboard, the logging output doesn't look the same.

InputEvent(1674022772, 478402, 4, 4, 42)
InputEvent(1674022924, 977692, 0, 0, 0)

At the moment I have no idea what this difference means.

[joshgoebel]

https://python-evdev.readthedocs.io/en/latest/apidoc.html#evdev.events.InputEvent

That's looks like a PRESS/42 SYNC/0... but I'm not sure what the 4 is? What does evtest output say for comparison?

[joshgoebel]

4 4 (EV_MSC MSC_SCAN) the scancode I mentioned. Inputs usually have:

• keydown
• scancode
• keyup

With a sync or two thrown in for good measure... you have to take the raw #s and decode them https://github.com/torvalds/linux/blob/master/include/uapi/linux/input-event-codes.h

[RedBearAK]

One press/release of the left shift key while evtest is running.

Event: time 1674034628.049499, type 4 (EV_MSC), code 4 (MSC_SCAN), value 2a
Event: time 1674034628.049499, type 1 (EV_KEY), code 42 (KEY_LEFTSHIFT), value 1
Event: time 1674034628.049499, -------------- SYN_REPORT ------------
Event: time 1674034628.157082, type 4 (EV_MSC), code 4 (MSC_SCAN), value 2a
Event: time 1674034628.157082, type 1 (EV_KEY), code 42 (KEY_LEFTSHIFT), value 0
Event: time 1674034628.157082, -------------- SYN_REPORT ------------

[RedBearAK]

This is when the "dummy" patch is in place, still left shift key press/release.

##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f108eaad810>, InputEvent(1674034827, 917872, 4, 4, 42))
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f108eaad810>, InputEvent(1674034827, 917872, 1, 42, 1))
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f108eaad810>, InputEvent(1674034827, 917872, 0, 0, 0))
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f108eaad810>, InputEvent(1674034828, 26067, 4, 4, 42))
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f108eaad810>, InputEvent(1674034828, 26067, 1, 42, 0))
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f108eaad810>, InputEvent(1674034828, 26067, 0, 0, 0))

Seems like there are more events happening for things that aren't being transformed. Assuming the "0, 0, 0" event corresponds to the "SYN_REPORT" lines in the evtest output, there are a total of six events for a single key press. But transformed keys without the patch in place don't seem to have that many events. Will have to keep looking at things with and without the patch in place.

[RedBearAK]

This is for left shift after removing the patch. Only four events.

##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f0d8a433070>, InputEvent(1674035452, 110437, 4, 4, 42))

(II) in LEFT_SHIFT (press)
(DD) on_key LEFT_SHIFT press
(DD) suspending keys [LShift<Key.LEFT_SHIFT>]
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f0d8a433070>, InputEvent(1674035452, 110437, 0, 0, 0))
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f0d8a433070>, InputEvent(1674035452, 212091, 4, 4, 42))

(II) in LEFT_SHIFT (release)
(DD) on_key LEFT_SHIFT release
(DD) resume because of mod release
(DD) resuming keys: [<Key.LEFT_SHIFT: 42>]
(OO) press LEFT_SHIFT 1674035452.215213
(OO) release LEFT_SHIFT 1674035452.2168956
##  output#send_event: self, event = (<keyszer.output.Output object at 0x7f0d8a433070>, InputEvent(1674035452, 212091, 0, 0, 0))

So what isn't there from the un-transformed output is...

InputEvent(1674034827, 917872, 1, 42, 1)
...
InputEvent(1674034828, 26067, 1, 42, 0)

Whatever those do, exactly. Seems to correspond to the 2nd and 5th evtest events.

Event: time 1674034628.049499, type 1 (EV_KEY), code 42 (KEY_LEFTSHIFT), value 1
...
Event: time 1674034628.157082, type 1 (EV_KEY), code 42 (KEY_LEFTSHIFT), value 0

Edit: I guess the "missing" events emanate from send_key_action instead of send_event.

[joshgoebel]

We're most certainly handling KEY events... what we usually "pass thru" directly (via send_event) are the MSC and SYN events.

But yes, _uninput has two output sinks, write and write_event which are subtly different... long-term we should probably standardize on one of them.

The default behavior is going to muck with the order of events for sure since the SYN and MSC events are going to now come BEFORE the KEY event...

--

Perhaps we should be dropping MSC and SYN entirely (and always generating them on our own)...

[RedBearAK]

I just really noticed that the first two fields (sec, usec) in the input events seem to contain the exact same values for the event like "4, 4, 42" and the following "0, 0, 0" event. That seems a little odd. No matter how fast the code is, it doesn't seem likely that the two events are literally always happening during the same millisecond (edit: microsecond!) time slice, especially given how far apart the logging lines are. So I have to wonder how/why that duplication of the time slice happens.

Edit: Probably a meaningless observation. The three non-transformed events all have the same microsecond time slice for each press event, and then a separate time slice for each group of three release events. Must just be how it works.

Perhaps we should be dropping MSC and SYN entirely (and always generating them on our own)...

I'm open to trying some different things, but you'd have to suggest where to put what.

If I'm matching things up correctly, it does seem as if the EV_KEY event should come after the MSC event (if we actually want to imitate the "real" keyboard activity as closely as possible) and before the "0, 0, 0" SYN events. But in the transformed events in the log it's MSC, SYN, KEY, while the evtest result is in the order MSC, KEY, SYN. Heaven knows if that actually makes any difference to anything.

Edit: The order actually isn't consistent. Probably the only way to get it consistent is to generate all events internally, as suggested.

[RedBearAK]

@joshgoebel

This is interesting, but I don't know what it means. If I hold the first key of the combo (the modifier) for a couple of seconds before hitting the next key, it becomes very difficult to trigger the focus problem. Cranking up the suspend time from my usual 0.1s to the default of 1s has a similar effect. But, strangely, it doesn't seem to matter whether I hit the combo within the 1s suspend time, or hold the modifier down until after the suspend time has definitely passed.

Aha, if I crank the suspend window down further to 0.01s, or even set it to zero seconds (effectively disabling suspend) it also becomes apparently impossible to trigger the unfocused dialog problem. Huh. 🤔

So yet another theory develops from this. What if something is "stumbling" during the transition window between when the modifier key is suspended and then resumed, and I just happen to be running into that transition window a lot due to the 0.1s suspend time I've been using, because of the Modifier+click touchpad issue? With a too-short interval or a very long interval, what is being avoided statistically in either case is the moment of transition between suspending and resuming the key. Which has probably been happening a lot right in the middle of my combos.

Hmm.

Ever since I've been able to cure Firefox/Thunderbird and the VSCode variants of the bad menu activation bahavior when they see the Alt key, I haven't really had much reason to need any suspend at all. Seems like I can generally get away with setting it to zero in my case, and it has to at least be shorter than the time between holding Cmd or Shift and clicking (less than 0.2s to make it reliable), so it might as well just be zero. But, this seems like good evidence that something about the suspend-to-resumed transition is problematic, and should probably be looked at.

[RedBearAK]

@joshgoebel

It may be my sleep delay tweak in the sync function that is at the root of this. 😞

    def __send_sync(self):
        _uinput.syn()
        sleep(1/1000)

I noticed that the focus issue really goes away only when I set the suspend window either very long or short (as described above), but I also have to disable the line in send_key_action that calls this function. It was already disabled in send_event, that wasn't my doing.

Turns out that if I just remove the sleep delay, the focus issue also goes away. It doesn't seem to matter whether the sleep delay comes before or after the "_uinput.syn()" line. It causes the focus issue either way.

So I have to choose between dialog windows reliably getting the keyboard focus, or macros being reliable.

I wonder if there is somewhere else to try adding a sleep delay to keep macros from imploding, but that won't end up causing this window focus issue. 🤔

[RedBearAK]

@joshgoebel

Well. I just did something I probably should have done earlier, but I assumed that macros would still be unreliable when removing the sync line, due to the fact that it removes the sleep delay that "fixed" macros to be 99.9% reliable for me.

By a strange twist, macros appear to be super reliable after removing the sync line entirely. In addition to fixing the dialog focus issue. So...

Just how important is the sync line? Is it necessary for anything? Or can we just straight up leave it out?

I was using a complex multi-line macro to test with, and it was failing in all other circumstances every 10-20 times I tried it. But I just filled a text file with over a thousand lines without a single failure. I wanted to make sure it wasn't some strange lucky fluke. It doesn't seem to be.

More good news: Remember that problem I had with Unicode sequences failing to complete in KDE, that led me to submit a PR with a suggested 1/10s delay in the Unicode keystrokes function? Seems to be resolved with this same change of removing the sync line. I removed the 1/10s delay and just disabled the sync line in output, and now the Unicode stuff seems to work without a hitch in Kubuntu.

[joshgoebel]

Are you still sending the original (straight from keyboard) syncs thru the pipeline if you add logging to send_event?

[RedBearAK]

Are you still sending the original (straight from keyboard) syncs thru the pipeline if you add logging to send_event?

Yep. Answered and conversation continued in https://github.com/joshgoebel/keyszer/pull/127