Idea: Interleave press/release of input/actual instead of suspending

[joshgoebel]

    Would this https://github.com/mooz/xkeysnail/pull/137 work better instead of suspending modifiers? That's what I used to fix the Firefox-Alt issue and the likes.

Originally posted by @laech in https://github.com/joshgoebel/keyszer/issues/20#issuecomment-1336972455

[joshgoebel]

That would be very interesting. You could easily apply that patch here with a very short suspend timeout. I wouldn't want to switch to this behavior without a lot of testing first to confirm there are no negatives. @RedBearAK Any thoughts on this one?

[RedBearAK]

@joshgoebel

Any thoughts on this one?

I like the idea and I'm very interested to see if it will work. Will try to test it on my own, but I'm not that familiar with this particular section of the code.

I guess I'll need to reactivate the menu bar in Firefox and VSCode to give it a solid test. We'll see.

[RedBearAK]

@joshgoebel

The send_combo function in output.py is significantly different in keyszer. It's not obvious to me how to translate the xkeysnail patch to work with the newer keyszer code.

If someone makes a branch I'll be able to test it.

[RedBearAK]

The concept of this seems sound, at least when you turn off the keymapper and hold Alt, then Ctrl, then release Alt, Firefox won't recognize the Alt key and won't focus the menu bar.

But I tried some variations of moving the press and release modifier sections around and haven't found a variation that stopped Firefox from focusing the menu bar if I did Alt+key. The keyszer send_combo just seems a bit too different.

Had the suspend timer set to zero or 0.01s. But it may still have some sort of effect that keeps the interleaving from solving the issue with Firefox.

There might be something that could come out of injecting a different modifier key press before releasing the original modifier, to fake the effect that happens when you do this manually. Like if it's Alt-press then inject a Ctrl-press, then release Alt, then release the injected Ctrl key press. If it's Ctrl, interleave a Meta, and so on. But that's probably not something I can figure out how to do.

[joshgoebel]

@RedBearAK Isn't this really just switching mod_keys_we_need_to_lift with mods_we_need_to_press? Right now we release first, this is all about pressing first...

[RedBearAK]

Those are the things I was moving around, on one side or the other, or both. But I think it also has something to do with the reversing on the other side of the normal key.

I don't have the brain power to try and do a flowchart of what's happening, all I know is that it didn't work with any of the variations I tried.

Like I said above, I think some logic would need to be worked out where you make sure that the right modifier is released after the press of another. And quite possibly some delay tuning between the different modifiers to make sure it "takes" in the same way as when you manually do something like:

Alt (press)
Ctrl (press)
Alt (release)
normal keystroke
Ctrl (release)

I didn't have much luck adding delays among the modifier presses/releases, but that may be because I just didn't choose exactly the right place to put the delay.

The manual method of doing this definitely gets Firefox to completely ignore the Alt key press-release and never activate the menu bar, even after the useful shortcut combo is completed. So I'm sure that somehow this could ultimately be made to work.

That being said, I noticed that when I have suspend set to zero it seems to kind of interfere with fast Cmd+Tab switching in GNOME. I'll get that notification that the window I tried to switch to is waiting in the background, even though I have an extension to block that and just raise the window.

Setting the suspend timer back to 0.1s seems to cure that glitch with quick task switching. I don't know exactly what causes that problem, and I'd recommend doing a lot of testing before thinking about removing suspend entirely, even if we can get the interleaving working and theoretically make it redundant.

[joshgoebel]

But I think it also has something to do with the reversing on the other side of the normal key.

I'm not sure how, at least for a simple combos... like take our alt-ctrl example... there is only one key to be released... or perhaps you're talking about our re-assertion of alt... that we should re-assert it BEFORE we lift ctrl... making it less likely to trigger as a "stand-alone" key... but if so you'd just reverse the two code blocks again... reassigning before lifting instead of now where we lift then reassert.

Of course suspend gets involved here as naturally we try to suspend keys that we aren't sure are "real" after a combo... such as if someone was still holding alt... now we're certain they didn't want it for Alt but for a combo so we should be even safer suspending it than before... (vs retriggering it)

And yes all of these ideas would require making sure you got the code exactly right and there were testing the exact thing you thought you were. :-)

[RedBearAK]

or perhaps you're talking about our re-assertion of alt... that we should re-assert it BEFORE we lift ctrl... making it less likely to trigger as a "stand-alone" key...

Yes, I think that's what I'm thinking about. That you have to actually do a real press and release of Alt before releasing Ctrl. With timing that allows the app to correctly interpret the sequence as the Alt key stuff happening while Ctrl is held down. .

I'm pretty sure suspend is messing with the timing of this, even somehow when it's set to zero.

But I'd have to put in a lot more debugging output and probably even take apart the lists to have a clear idea of just what exactly is happening, and possibly suss out why it didn't work.

Not sure I want to mess with it too much before finishing up the Wayland support branch in a way that might be acceptable for merging.

[RedBearAK]

Hey, I think I actually got this working, inside the VM. I'll have to play with the delays to see how far I can trim it down. It may be that I wasn't giving it big enough delays previously.

Testing using my Option-key characters, which if I just tap the physical Meta key that's been modmapped to Alt will activate the Firefox menu. But when I'm doing Option+key or Shift+Option+key, the menu is not activating even after the (very slow) sequence is over.

This is what it took:

        # for key in mod_keys_we_need_to_lift:
        #     self.send_key_action(key, RELEASE)
        #     released_mod_keys.append(key)

        for key in [mod.get_key() for mod in mods_we_need_to_press]:
            self.send_key_action(key, PRESS)
            pressed_mod_keys.append(key)
            sleep_ms(_THROTTLES['key_post_delay_ms'])
        sleep_ms(_THROTTLES['key_post_delay_ms'])

        for key in mod_keys_we_need_to_lift:
            self.send_key_action(key, RELEASE)
            released_mod_keys.append(key)
            sleep_ms(_THROTTLES['key_post_delay_ms'])
        sleep_ms(_THROTTLES['key_post_delay_ms'])

We will see what I can trim out and still keep it working.

[RedBearAK]

Preliminary reductionist testing suggests the success in the VM is piggy-backing on the existing delays around the normal key that fix a lot of things already. Will have to try something similar in the host OS and report back.

[RedBearAK]

This appears to work well with Option+Shift (although I'm not sure that was ever a problem), but if it's just Option+key combos it is very dependent on how quickly I let go of the Option(Meta modmapped to Alt) key. If I let go of it fast enough, no problem.

This is on the host, which normally only requires a few milliseconds delay to fix macros, not 40+70. So I think what was happening before is that I was always still holding the Alt key by the time the combo finished doing its thing. So Firefox would always react to the Alt release.

I'm giving it 30+60 on the normal key, plus 50ms each on the mod press and release before the normal key. And then I still have to let go of the physical Meta (logical Alt) key pretty quickly, or it activates the menu.

So, still can't call the interleaving a complete success yet. But promising.

[RedBearAK]

I don't get it. In the log, I can see that the release of Alt happens before the Unicode address, yet if I don't let go of the Alt key quickly enough, after the Unicode sequence ends the Firefox menu bar appears. This really happens only after the Unicode sequence is complete. I can make the delays really long and watch it type out the address and create the character, and then menu bar.

By the time the menu bar appears, I may have had the Alt key released for multiple seconds, and there is no record in the log of the Alt key being released later than the beginning of the Unicode address. In fact it's released before the "u" of the Ctrl+Shift+U shortcut. Which makes sense, otherwise the shortcut wouldn't work at all. Yet still, somehow Firefox thinks I'm pressing and releasing, or just releasing, the Alt key after all that happens.

Or it received it as input earlier, but doesn't react until the Unicode sequence is complete. For some reason.

The way it happens really makes very little sense to me.

[RedBearAK]

(OO) press LEFT_ALT 1677386118.8861177

(II) in W (release)
(DD) on_key W release

(II) in LEFT_META (release)
(DD) on_key LEFT_ALT release
(DD) resume because of mod release
(OO) release LEFT_ALT 1677386118.8994493

This is in the log after the Unicode sequence, in the case when the menu bar appears. But the Alt key was physically released before the Unicode sequence. Soooooo....

I have no idea how to deal with this.

Whatever happened to that post_combo timeout idea? Seems like that may be what is needed here. Because the interleaving seems to be technically doing what is necessary to get Firefox to ignore the Alt key, until it sees the new Alt key event after the combo.

And here is the after event from the case where the menu bar does not appear.

(II) in W (release)
(DD) on_key W release

(II) in LEFT_META (release)
(DD) on_key LEFT_ALT release
(DD) silent lift of spent mod LEFT_ALT

[joshgoebel]

This is in the log after the Unicode sequence, in the case when the menu bar appears. But the Alt key was physically released before the Unicode sequence. Soooooo....

Feel like I need more context/log, not sure what you're trying to show me.

[RedBearAK]

This is just the difference in the log immediately after the Unicode sequence completes. In the first case, the Firefox menu bar is activated. In the second case, the menu bar doesn't appear.

Part of the problem may be the repeat delay being quite short.

(II) in LEFT_META (press)
(DD) MODMAP: LEFT_META => LEFT_ALT in ['Cond modmap - GUI - Windows kbd']
(DD) on_key LEFT_ALT press
(DD) suspending keys [LAlt<Key.LEFT_ALT>]
(DD) resuming keys: [<Key.LEFT_ALT: 56>]
(OO) press LEFT_ALT 1677389044.2516947

(II) in LEFT_META (repeat)
(DD) on_key LEFT_ALT repeat
(OO) repeat LEFT_ALT 1677389044.4002235

(II) in LEFT_META (repeat)
(DD) on_key LEFT_ALT repeat
(OO) repeat LEFT_ALT 1677389044.4324012

(II) in W (press)
(DD) on_key W press

(DD) WM_CLS: 'firefox' | WM_NME: 'Idea: Interleave press/release of input/actual instead of suspending · Issue #124 · joshgoebel/keyszer — Mozilla Firefox'
(DD) DVN: 'AT Translated Set 2 keyboard' | CLK: 'False' | NLK: 'False'
(DD) KMAPS: ['OptSpecialChars toggles', 'OptSpecialChars - US',
             'User hardware keys', 'Firefox Browsers Overrides',
             'General Web Browsers', 'Wordwise - not vscode',
             'Cmd+Dot not in terminals', 'General GUI overrides - not Chromebook',
             'General GUI overrides - Fedora', 'General GUI overrides - GNOME',
             'General GUI']
(DD) COMBO: LAlt-W => [Shift-Ctrl-U, <Key.KEY_2: 3>, <Key.KEY_2: 3>, <Key.KEY_1: 2>, <Key.KEY_1: 2>, <Key.ENTER: 28>] in KMAP: ['OptSpecialChars - US']
(DD) spent modifiers []
(OO) press LEFT_SHIFT 1677389044.4387658
(OO) press LEFT_CTRL 1677389044.4889956
(OO) release LEFT_ALT 1677389044.5393097
(OO) press U 1677389044.6497092
(OO) release U 1677389044.6498058
#### from send_combo pressed_mod_keys: pressed_mod_keys = [<Key.LEFT_SHIFT: 42>, <Key.LEFT_CTRL: 29>]
#### from send_combo modifier in reversed(pressed_mod_keys): modifier = <Key.LEFT_CTRL: 29>
(OO) release LEFT_CTRL 1677389044.7401068
#### from send_combo modifier in reversed(pressed_mod_keys): modifier = <Key.LEFT_SHIFT: 42>
(OO) release LEFT_SHIFT 1677389044.7401912
(OO) press KEY_2 1677389044.8004167
(OO) release KEY_2 1677389044.800543
#### from send_combo pressed_mod_keys: pressed_mod_keys = []
(OO) press KEY_2 1677389044.9509714
(OO) release KEY_2 1677389044.9510822
#### from send_combo pressed_mod_keys: pressed_mod_keys = []
(OO) press KEY_1 1677389045.101509
(OO) release KEY_1 1677389045.1016195
#### from send_combo pressed_mod_keys: pressed_mod_keys = []
(OO) press KEY_1 1677389045.2520523
(OO) release KEY_1 1677389045.2522278
#### from send_combo pressed_mod_keys: pressed_mod_keys = []
(OO) press ENTER 1677389045.4027667
(OO) release ENTER 1677389045.4029796
#### from send_combo pressed_mod_keys: pressed_mod_keys = []
(OO) press LEFT_ALT 1677389045.4933133

(II) in W (release)
(DD) on_key W release

(II) in LEFT_META (release)
(DD) on_key LEFT_ALT release
(DD) resume because of mod release
(OO) release LEFT_ALT 1677389045.5062768

[RedBearAK]

No, the repeat delay doesn't seem to have anything to do with it. I set a long keyboard repeat delay of 1500ms and it didn't help avoid the activation of the menu bar if I don't quickly release the Meta(Alt) key.

It doesn't have to be super fast, just quick. Seems to be very low probability that I'm releasing within the 0.1s of the suspend timer when it works. More like within 20ms/0.2s.