Discussion: Conditional modmaps & sticky

[joshgoebel]

So the context is "sticky keys" (my naming)... the desire to be able to "hold down" a modifier all the way thru from the input to the output. This isn't possible with xkeysnail 0.4 because it lifts the modifiers after every single combo. This is undesirable when we want the modifier's continuous hold to preserve a state, such as "i'm still using the tab switcher UI".

My idea is to go back to basics and try and reproduce what is happening as plainly as possible... by associating the keys on both sides (the sticky part)... so if we have a combo like:

Cmd-Tab => Ctrl-Tab

Hitting Cmd-Tab would first bind Cmd and Ctrl together... then run the combo, then lift tab (whole continuing to hold Ctrl on the output as long as you hold cmd on the input). IE, as if we had created a physical binding between the physical Cmd key and the virtial Ctrl-key.

It would do this until you used another incompatible macro. Any other macros with similar stickiness could preserve the hold. There are edge cases here I think, but first I'm more worried about the larger environment changing out from underfoot.

---

So what happens if someone hits:

Cmd-Tab => output Ctrl-Tab
releases tab, but not Cmd

They are still holding Cmd, but they have tabbed into a different window.

• What if there is now a different modmap - such that Cmd no longer points to Ctrl at all?

• What happens if the modmap hasn't changed but they have now tabbed into a different keymap that no longer has a Cmd-Tab combo?

A side issue here is that at this point we don't KNOW the window has changed - since we're querying it only when potential combos happen - so to handle this properly we'd need to actively be notified of window changes I think. But first I'd like to try and establish the "correct" behavior here so I can add it to the test suite... (even if I have to skip those tests)

[RedBearAK]

@joshgoebel

They are still holding Cmd, but they have tabbed into a different window.

Actually it's up to the window manager exactly what happens with each press of Tab while holding Alt. But normally the user is not actually switched into a different window until the release of Alt (or Cmd on macOS). Most switchers also allow the user to press Escape and avoid switching at all.

Most of this post is beyond me, but I don't think watching for window changes is necessarily going to be helpful. The window shouldn't change until after the held key is released, and only if the user has a different application icon/window selected when they release, and hasn't pressed Escape.

And then there's KDE/Kwin, where if the user leaves the "Show selected window" option enabled, the window manager will pop each window into the foreground while you Tab-Tab-Tab, so you would probably see those as window events, but you still don't actually get switched into the window until you release Alt. AFAIK.

[joshgoebel]

Actually it's up to the window manager exactly what happens with each press of Tab while holding Alt

Well, we get to decide first. The WM doesn't even see the input if we choose not to send it.

But normally the user is not actually switched

I dunno about "normally", but perhaps you are right for many DEs. I use Awesome WM and when I hit Cmd-Tab the switch of apps is immediate.

The window shouldn't change until after the held key is released

How could you know this? It could change at any time, my setup is one example of this... my hands are still on the keyboard while an entirely different window has already taken focus. Not a hypothetical. We have to remember than Linux is many different WMs, many different DEs - the behavior is often far wider than you might think if you're just using one distro/DE.

where if the user leaves the "Show selected window" option enabled,

Ah, I do wonder how that works... if it works how you think good... if they are actually being focused for input though then it has the same issue as I have on Awesome WM.

---

One thing I'm really trying to do is rethink this whole idea of how the mappings are done to be more "organic"... instead of looking for combos and then sending them thru I'd rather translate "live" and send all keystrokes thru live... but that's hard when keys can change meaning at any time. :-)

So the hard part is decide what happens what that meaning changes. Perhaps a sticky key could even "hold open" a modmap or keymap... IE, if you were alt-tabbing then no matter WHAT rules you had (or what window as focused) that alt-tab would be marked as special and allow you to keep hitting it.

But then we get into things like pairs... since you might want to stop alt-tabbing and alt-shift-tab (go in reverse) or some such - and this isn't the same combo at all...

Of course we could just say "put global shortcuts" in the top-level keymap... and that would be good I think - but modmaps are potentially still an issue since conditional modmaps wrestle control away from the default modmap...

I'm just trying to think thru these edge cases and define the desired behavior (or at least understand the current undesired behavior)... right now there are far too many holes in how it works - things that are left unspecified. It's how both kinto and xkeysnail (0.4) seem to work so well for so many people while BOTH at the same time having rather glaring issues in behavior.

[RedBearAK]

@joshgoebel

Probably should have said "typically" rather than "normally", but the point remains. Despite the relative popularity of tiling window managers in Linux, even most Linux users are going to spend most of their time using DEs that present a GUI task switcher dialog, just like you'd see in Windows or macOS. In which case, the actual switching and giving the keyboard focus to the other window doesn't happen until you release the applicable modifier key. Although, macOS does switch immediately with the Alt+Grave shortcut, while most Linux DEs don't. Pretty sure even the "Show selected window" in KDE is just a preview of the window you're about to switch to, rather than already giving it keyboard focus. But I could be wrong.

So the situation is pretty mixed as far as when the actual focus changes. That's all I was getting at.

[joshgoebel]

Although, macOS does switch immediately with the Alt+Grave shortcut,

Which (on linux) would be the same app (WM_CLASS) I assume but could be a different WM_NAME (window title)... so it'd be possible with conditionals to have modmaps/keymaps disappearing out from under you as your tabbing with grave. (after conditional WM_NAME is supported)

---

I wonder if we're talking past each other a bit? My only/larger point was that I have no desire to break/ignore tiling window managers (doubly so as a user of one) - so the fact that they exist at all, even if it's a minority, means that "focus never changes" during input cannot be blindly assumed.

There are also other cases (other than grave when you already mentioned) such as closing an app... if one did:

• Hold CMD
• W (close)
• some other keys part of other combos

You'd now have the same issue that CMD has potentially become ambiguous as your window focus has changed (immediately) as a result of closing a window. Yes, holding CMD there seems a bit contrived, I'd agree, but possible. :-)

It may be some of these cases can be ignored, but I usually like knowing what all the cases are rather than just leaving large swaths of behavior "unspecified". :-)

[joshgoebel]

Right now I'm thinking that initially this (switching maps out from under you) might just be a "known issue" and we acknowledge it and see if it actually matters in daily usage.

If we had active (vs passive) monitoring of the current window then we could more actively manage the situation - such as detecting when maps change live and basing our hold/release based on the "compatibility" of the two maps... ie, if two modmaps were compatible - when we might not care that they changed at all but if you are holding down CMD when the meaning of CMD changes entirely - then we might decide to take some sort of action.

[RedBearAK]

@joshgoebel

There are also other cases (other than grave when you already mentioned) such as closing an app... if one did:

    Hold CMD
    W (close)
    some other keys part of other combos

You'd now have the same issue that CMD has potentially become ambiguous as your window focus has changed (immediately) as a result of closing a window. Yes, holding CMD there seems a bit contrived, I'd agree, but possible. :-)

I know that I frequently do things like Cmd+T, T, T, T and Cmd+W, W, W, W and Cmd+V, V, V, V all without lifting the Cmd key. And that hasn't been a problem with Kinto. But... I guess between apps I'm usually doing Cmd+Tab to the other app, or Cmd+Grave to another window of the same app, and I have to release Cmd (Alt on PC) to actually go to the other window to continue doing other shortcuts.

But closing a window that results in a different app grabbing focus is definitely a frequent occurrence. I'm not certain I've ever really experienced a problem around that though.

Not really sure why Cmd by itself would be "ambiguous" since it shouldn't be triggering anything by itself. Does holding Cmd as the app changes stop the map from changing, leading to a shortcut remap from the previous app possibly being triggered?

I wonder if we're talking past each other a bit?

Probably, but I think we're in fundamental agreement about not assuming too much. I see that you're trying to deal with the problem of the app changing in the middle of using shortcuts without a final release of what was started in the previous app.

If I'm understanding correctly.

Right now I'm thinking that initially this (switching maps out from under you) might just be a "known issue" and we acknowledge it and see if it actually matters in daily usage.

Yeah, I've been living with Kinto in various Linux installs for at least a couple of years now and this may have caused an issue on occasion, but I can't think of anything specific that would provide proof of that.

[joshgoebel]

Cmd+V, V, V, V all without lifting the Cmd key. And that hasn't been a problem with Kinto.

It wouldn't. Kinto excels at holding keys down. :-) But in this case 0.4 would work equally well since as far as I know those would all work equally well as discrete combos (if you weren't holding Cmd).

I'm not certain I've ever really experienced a problem around that though.

You likely wouldn't if you weren't in the middle of a key combo - and even then to notice you'd have to specifically have incompatible maps... I wager most people simply don't try to do held combos across boundaries like this, so it may be a real case, but one that doesn't matter much in practice. You'd be surprised though at how fuzzy xkeysnail already is on what you type vs what comes out the other end - and (in many cases) no one notices. But when you start writing tests you have to pin all this behavior down and codify it.

I see that you're trying to deal with the problem of the app changing in the middle of using shortcuts without a final release of what was started in the previous app.

Yeah, and particular so if the meaning of the key being held has changed - if the meaning hasn't changed there really isn't an issue...

Not really sure why Cmd by itself would be "ambiguous" since it shouldn't be triggering anything by itself.

It's ambiguous because it may be that it should trigger itself [the actual Cmd]. That is the whole problem... every time you hit Cmd you can mean one of two things:

• I want to press the actual REAL life Cmd key
• I'm using Cmd as part of a mapping that has nothing to do with Cmd on the output

With the exception of modmap (which always remaps) the keymappers goal should be is to pass input thru verbatim (with NO changes) - UNLESS it's a combo - in which case history should rewind and only the combo [output] keys should be pressed, not the REAL [input] keys.

Every time you hit any modifier (used in a combo that doesn't include that same modifier) it's always ambiguous. That's what suspension is for - suspend the keys until the ambiguity is removed.

Does holding Cmd as the app changes stop the map from changing, leading to a shortcut remap from the previous app possibly being triggered?

That's what we need to protect against IMHO.

[RedBearAK]

@joshgoebel

It's ambiguous because it may be that it should trigger itself [the actual Cmd]. That is the whole problem... every time you hit Cmd you can mean one of two things:

Every time you hit any modifier (used in a combo that doesn't include that same modifier) it's always ambiguous. That's what suspension is for - suspend the keys until the ambiguity is removed.

Now it's making sense, I think. Finally. I keep forgetting that the same modifier key can be mapped onto different modifiers in the output according to the total matched input shortcut. Derp.

[joshgoebel]

Yeah, you can't really solve this without the ability to look into the future - or just suspend all output until you have better information (which works almost as well).

[joshgoebel]

This is no longer theoretical, because I juse ran into it badly. Because of the dual modmaps that Kinto has:

define_conditional_modmap(lambda wm_class: wm_class.casefold() not in terminals,{
    # - Mac Only
    Key.LEFT_META: Key.RIGHT_CTRL,  # Mac
    Key.LEFT_CTRL: Key.LEFT_META,   # Mac
    Key.RIGHT_META: Key.RIGHT_CTRL, # Mac - Multi-language (Remove)
})

# [Conditional modmap] Change modifier keys in certain applications
define_conditional_modmap(re.compile(termStr, re.IGNORECASE), {
    Key.LEFT_META: Key.RIGHT_CTRL,  # Mac
    # Left Ctrl Stays Left Ctrl
    Key.RIGHT_META: Key.RIGHT_CTRL, # Mac - Multi-language (Remove)

Take notice that the meaning of control changes. So I also have a modmap for Hyper, but then I also want the Ctrl-left/right to still switch screens (like it always had) - so I map that back to Hyper, but I have to do it twice, because the modmap for terminals and non-terminals is different.

define_keymap(lambda wm_class: wm_class.casefold() not in terminals,{
    K("LSuper-LEFT"): K("Hyper-LEFT"),
    K("LSuper-RIGHT"): K("Hyper-RIGHT"),
},"screen paging (not term)")

define_keymap(re.compile(termStr, re.IGNORECASE), {
    K("C-LEFT"): K("Hyper-LEFT"),
    K("C-RIGHT"): K("Hyper-RIGHT"),
},"screen paging (term)")

Now the problem... I press Ctrll and start screen switching (from an open terminal)... and it's (After modmap) seeing Hyper... so it binds them both as sticky... and then I wind up on a NON-terminal and that's when i release ctrl... but NOW (thanks to the other modmap) Ctrl is no longer Ctrl, it's super... so it registers as Super being lifted (while control remains down, and hence hyper is also still held down)... so now Hyper is stuck... and you can't just hit Ctrl to unstick it because Ctrl is Meta at the moment... you can only unstick it by switching to a terminal and hitting ctrl...

Ugh.

[joshgoebel]

It seems clear to me now that (if we are going to track state at all) that ever key needs lots of meta information:

PressedKey:
  suspended: boolean,
  pressed_at: time,
  actual_key: Key
  mapped_key: Key

So in this case when the key goes sticky perhaps we'd have something like:

PressedKey:
  suspended: false,
  pressed_at: 12:15pm,
  actual: Ctrl
  mapped: Ctrl
  sticky: [Hyper]

So now when I release Ctrl (which could mean any number of keys at the moment of release) it can now be certain that Ctrl was sticky mapped to Hyper and it can release hyper. Even though at the moment of release Ctrl is modmapped to Meta/Super.

[joshgoebel]

(or perhaps this is evidence for why we should prefer the press/release system that 0.4.0 uses and sticky should be the rate exception to the rule...)

[joshgoebel]

How about this as an actual proposal for discussion. I'm getting close on most of the other things but sticky/binding is still kind of hanging open. So since this seems rather Kinto specific (but still useful) lets make it a discrete vs automatic... So I'd suggest:

define_keymap(# ...
    K("RC-Tab"): [BIND("RC", "Alt"), K("Alt-Tab")],   
    K("RC-Shift-Tab"): [BIND("RC", "Alt"), K("Alt-Shift-Tab")],
    K("RC-Grave"): [BIND("RC", "Alt"), K("Alt-Grave")],          
    K("RC-Shift-Grave"): [BIND("RC", "Alt"), K("Alt-Shift-Grave")], 

IE, allow a single key bind... I think that's what matters in most (all?) cases yes? Binding CMD/Ctrl, or the equivalent? Are there DE that really want multiple keys bound and held?

@RedBearAK How is your cmd-tabbing working with the current "half-done" support? Is it sticky enough if you start with cmd-tab? If so (and we could confirm that across all envs) that means binding a single key is good enough.

The BIND command ties RC to Alt such that RC held on input means Alt remains held on output - and won't be subject to how xkeysnail likes to be "jumpy" with it's keys - it would remain pressed on output until it was released on the input.

Or (with my new configs) we could allow wrappers for grouping so this is a lot nicer:

keymap("not remotes",
    BIND(("RC", "Alt"), {
        K("RC-Tab"): K("Alt-Tab"),                
        K("RC-Shift-Tab"): K("Alt-Shift-Tab"),    
        K("RC-Grave"): K("Alt-Grave"),            
        K("RC-Shift-Grave"): K("Alt-Shift-Grave"),
    }),
    {
        # more config here
    }

Such that any mappings inside BIND would insert the initial BIND command at the beginning of the sequence... ie this is exactly the same config as above just syntactic sugar to avoid repeating yourself.

[RedBearAK]

@joshgoebel

How is your cmd-tabbing working with the current

Ah, it seemed to be working OK as long as I start with Cmd+Tab and don't try to use Cmd+Shift+Tab or start with Cmd+Shift+Tab. Transitioning to Cmd+Grave also seems to work OK, to pick an individual window from an app with multiple windows open, but Cmd+Shift+Grave gives the same kind of "stuck task manager" problem as Cmd+Shift+Tab. Cmd+Shift+Grave by itself will also shift instantly to a window in same app without showing the task switcher, like Cmd+Shift+Tab instantly switches apps.

And just now I started having a pretty major problem with Shift and Alt not wanting to release, every mouse click in VSCode was trying to select text or set a secondary cursor, and cycling the modifier keys didn't get them to release. Had to switch back to the Kinto branch.

BIND(("RC", "Alt"), {

Any way of avoiding the double open parentheses and the difficulty of keeping track of the closing outside parentheses? I think I would actually be fine with the un-simplified version just for avoiding excessive repeating punctuation and nesting.

And you never know when you might want to individually disable/enable or edit some of those lines via script. That's not nearly as easy with that sort of nesting. It's kind of fundamental to how Kinto adapts the config file to different DEs.

As far as I understand, Kinto uses pass_through_key to just let Alt+Tab still be Alt+Tab and bypass the modmap. Or maybe Ctrl+Tab becomes Alt+Tab, which is then passed through? I've never completely understood how that works.

    K("M-Tab"): pass_through_key,                 # Default - Cmd Tab - App Switching Default
    K("RC-Tab"): K("M-Tab"),                      # Default - Cmd Tab - App Switching Default
    K("RC-Shift-Tab"): K("M-Shift-Tab"),          # Default - Cmd Tab - App Switching Default
    K("RC-Grave"): K("M-Grave"),                  # Default not-xfce4 - Cmd ` - Same App Switching
    K("RC-Shift-Grave"): K("M-Shift-Grave"),      # Default not-xfce4 - Cmd ` - Same App Switching

Weird that there's no pass_through_key for Cmd+Shift+Tab or the Grave shortcuts, but I have no problem using any of them.

[joshgoebel]

And just now I started having a pretty major problem with Shift and Alt not wanting to release, every mouse click in VSCode was trying to select text or set a secondary cursor

The only stuck keys issues I've seen so far have been modmaps getting ripped out from under you... the latest (git) release should be resilient against this though.

Any way of avoiding the double open parentheses

The programmer in me screams this is correct because you aren't passing two things, you're passing one object - a key binding pair, hence the tuple... but I think for users editing the config directly it's ok to have to learn a little bit of syntax... this isn't really something Kinto users have to deal with (for the most part)

and the difficulty of keeping track of the closing outside parentheses?

That's just the syntax. Function calls have to have a paired set of ().

I think I would actually be fine with the un-simplified version just for avoiding excessive repeating punctuation and nesting.

Well the grouping would just a niceity... either way you need something PER combo to tell it how to bind the keys... if someone didn't want to use the nicer syntax they wouldn't have ti.

I've never completely understood how that works.

I don't completely yet understand what these are supposed to do either so I can't confirm their behavior, but I'm pretty sure some of them are broken.

Weird that there's no pass_through_key for Cmd+Shift+Tab or the Grave shortcuts, but I have no problem using any of them.

In non-kinto versions combos that aren't mapped should work "As-is" (well other than the modmapping, which always happens)... you sholdn't need "pass thru" everywhere... I'd love to know what the intention of these are and what they are supposed to do.

I have no tests for any of these yet - and I add a 3rd "ignore" (which I'm pretty sure works)...

[joshgoebel]

OK as long as I start with Cmd+Tab and don't try to use Cmd+Shift+Tab

Are you sure? I'd think you could use Cmd+shift+tab if you were still holding Cmd from the prior Cmd-tab... you just can't start with that combo...

[joshgoebel]

Or perhaps go a little more generic and reuse BIND, but simpler:

keymap("not remotes",
   append_to_each(BIND("RC", "Alt"), {
        K("RC-Tab"): K("Alt-Tab"),                
        K("RC-Shift-Tab"): K("Alt-Shift-Tab"),    
        K("RC-Grave"): K("Alt-Grave"),            
        K("RC-Shift-Grave"): K("Alt-Shift-Grave"),
    }),

I can stomach that better since now BIND itself becomes the object pair. That's a bit more programmery though...

I do'nt suppose you'd like a dict better (now with {}!):

BIND({"RC": "ALT"})

[joshgoebel]

The nice thing about it just being python is anyone who knows a bit of python can build their own abstractions, invent new keywoard etc... as long as at the end of the day the API is passed the data it expects. Like my example of you using for loops to build your complex maps...

[RedBearAK]

@joshgoebel

Are you sure? I'd think you could use Cmd+shift+tab if you were still holding Cmd from the prior Cmd-tab... you just can't start with that combo...

Yes, the problem was arising from starting with Cmd+Tab and then using Cmd+Shift+Tab or Grave while the task switcher was open, holding Cmd (physical Alt, logical Ctrl) in between.

And doing another run, which automatically does a pull each time, I'm back to having exceptions for every key. Including Ctrl+C. Had to close the terminal tab to kill the process.

The nice thing about it just being python is anyone who knows a bit of python can build their own abstractions, invent new keywords etc... as long as at the end of the day the API is passed the data it expects. Like my example of you using for loops to build your complex maps.

Yes, I've had brief glimpses of the unusually malleable nature of python. How few actual functions are baked in, and so forth. It's a very interesting language, with all the libraries giving it power. There's obviously a dozen or more different ways to do what you're suggesting.

BIND({"RC": "ALT"})

Well, at least the punctuation layers are different. But I think I'd stick with the per-line formatting. It's much easier to just go to a specific line and understand what that line is doing, or what is being done to it. The nature of the config file is a lot of individual shortcuts that may need to be tweaked. Any kind of nesting logic makes that a bit less straightforward. Nesting and looping are great for making a "program" more compact, but for this context I don't think it's that helpful to other end users of the config. I think of the Kinto config file more like a... phone book. Trying to compress and nest the information in a phone book to use up fewer pages would usually not be considered helpful to the people trying to use it as a clear reference.

But that's just me.

[joshgoebel]

I'm back to having exceptions for every key.

You'd have to show me the exceptions - I've seen no issues on my end...

[joshgoebel]

Yes, the problem was arising from starting with Cmd+Tab and then using Cmd+Shift+Tab or Grave while the task switcher was open, holding Cmd (physical Alt, logical Ctrl) in between.

Which problem? I feel like maybe we're talking past each other... I actually have a test now that makes sure the latest version works (tabbing forwards & backwards), just so long as you start with the simpler combo.

https://github.com/joshgoebel/keyszer/blob/main/tests/test_sticky_bind.py#L48

Or maybe it didn't quite work before but does now?

[joshgoebel]

Well, I'm assuming that holding meta is what matters (which seems like it would have to be true - you can let up on shift and tab during the process)... the test only asserts that META is held the entire time (without lifting) with tabbing, and shift tabbing inbetween.

[joshgoebel]

The nature of the config file is a lot of individual shortcuts that may need to be tweaked. Any kind of nesting logic makes that a bit less straightforward.

The entire file is full of nesting. :-) combos are nested inside keymaps and modmaps. :)

[RedBearAK]

@joshgoebel

Which problem? I feel like maybe we're talking past each other.

The problem of the task switcher dialog getting "stuck" open on the screen instead of switching to the selected app when lifting the Cmd key. If I used either of the "Shift" shortcuts it task switcher wouldn't go away, and the system would act like the Alt key was still being held. And maybe the Shift key as well.

You'd have to show me the exceptions - I've seen no issues on my end...

Exceptions and tracebacks for every key press. I had to kill the terminal tab so I couldn't save them.

Is there something more robust than git pull? I don't know why I keep having issues that you're not seeing on your end, unless somehow what I pull down on my system is different from what you're using locally.

The entire file is full of nesting. :-) combos are nested inside keymaps and modmaps. :)

That is technically true, but it was still pretty easy for me even back when I had no understanding of python to just ignore the K() stuff and see what the shortcuts were doing, line by line. Outside of the K() there's usually just the one other layer defining where the shortcuts would take effect. That too was not too difficult to grasp even early on. But if there were further layers of nesting changing what the individual lines were doing, I would have found that quite a bit more confusing.

[joshgoebel]

The problem of the task switcher dialog getting "stuck" open on the screen instead of switching to the selected app when lifting the Cmd key.

That's weird, I wonder if you see the same behavior on the latest (when you get it running)... you might read the test - see if that doesn't look right - it looks right to me. I think perhaps you were running into modmap bugs before tabbing across modmap boundaries, which I've since fixed...

I had to kill the terminal tab so I couldn't save them.

Take a photo with your phone? Change the magic abort key and use that to terminate it without closing the terminal.... you could make the abort key "F" or anything on your keyboard really.

https://github.com/joshgoebel/keyszer/blob/main/src/keyszer/input.py#L187

I couldn't live without my F16 now. :-)

Is there something more robust than git pull?

Git pull is robust... it wouldn't surprise me if it was your config file again or something else unique to your system...

But if there were further layers of nesting changing what the individual lines were doing, I would have found that quite a bit more confusing.

I think you're speculating, spitballing, but you might be right... anyways... like I said if there is a nice wrapper people could still write their config the longer more repetitive way if they really wanted... that'd be a decision for the config author.

[joshgoebel]

You could also run pytest first and see if all the tests are passing each time you update.