[BUG] <s-tab> does not work with wezterm

dojoteef commented 1 year ago

Version of Kakoune

v2021.08.28-Kakoune 2022.10.31

Reproducer

Install WezTerm
Install a version of kakoune after removal of ncurses
Launch kakoune and press : followed by <tab> and <s-tab>

Outcome

Notice that <s-tab> does not cycle backward, but rather seems to reset the autocomplete state as if you have just pressed <tab>.

Expectations

<s-tab> should work correctly as expected.

Additional information

I'm running on macOS. I've verified that the issue does not occur with the default Terminal.app. It also does not occur with older versions of kakoune that depend on ncurses (regardless of which terminal is used). I'm not intimately familiar with the way escape sequences are handled, but I verified that if I press <s-tab> in wezterm, I see the escape sequence <ESC>[Z. I did this using showkey. This is the exact same sequence I see when using Terminal.app, which is weird since the latest versions of kakoune work fine on that terminal.

I know vim has timeout and ttimeout for handling escape sequence timing. Is there a similar setting for kakoune that I need to investigate? Could there be a different issue going on?

Screwtapello commented 1 year ago

I can reproduce this, using the "AppImage" build of WezTerm. Running cat in both gnome-terminal and WezTerm, <tab> always produces a tab, <s-tab> always produces CSI Z.

Running kak -n -e 'buffer *debug*; set buffer debug keys' in gnome-terminal, Tab produces <tab> and Shift-Tab produces <s-tab> as one would expect.

Running that same command in WezTerm, Tab produces <tab> and Shift-Tab produces <a-[> (CSI, intepreted as a key-press) and the Z seems to have gone completely missing.

mawww commented 1 year ago

For some reason, WezTerm decided to use CSI ~ instead of CSI u as most terminal emulator do, Kakoune parser does not currently support that. Adding it should be easy, but I think it would be great to first understand why WezTerm decided to deviate from common practice, and if it introduces other differences.

EDIT: looking at wezterm's doc, it seems adding enable_csi_u_key_encoding = true in its config should do the trick.

Screwtapello commented 1 year ago

I assume what you call the CSI ~ encoding is xterm's default, which represents otherwise-impossible key-combinations as modifiers on unused keycode 27:

When modifyOtherKeys is set, ordinary keys may be sent as escape sequences:

When modifyOtherKeys is set to 1, only the alt- and meta-modifiers apply. For example, alt-Tab sends CSI 2 7 ; 3 ; 9 ~ (the second parameter is "3" for alt, and the third parameter is the ASCII value of tab, "9").

When modifyOtherKeys is set to 2, all of the modifiers apply. For example, shift-Tab sends CSI 2 7 ; 2 ; 9 ~ rather than CSI Z (the second parameter is "2" for shift).

The formatOtherKeysresource tells n to change the format of the escape sequences sent when modifyOtherKeys applies. When modifyOtherKeys is set to 1, for example alt-Tab sends CSI 9 ; 3 u (changing the order of parameters). One drawback to this format is that applications may confuse it with CSI u (restore-cursor).

— https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h3-Alt-and-Meta-Keys

I think the other benefit of the CSI 27 ~ encoding was supposed to be that since there's already a bunch of other keys represented as CSI (number) ~ then existing applications would already route such sequences into the keyboard-handling code.

I can confirm that with formatOtherKeys set to its default, starting Kakoune inside xterm or WezTerm and pressing <c-1> opens the <a-[> menu just as described in the first comment. Changing formatOtherKeys to use CSI u encoding, or running Kakoune inside tmux inside xterm works fine (since tmux understands both CSI 27 ~ and CSI u, but always sends CSI u)

If I run Kakoune inside tmux inside WezTerm, pressing <c-1> does not work, but it behaves as though extended key encoding is not enabled. I'm not sure why.

dojoteef commented 1 year ago

@Screwtapello @mawww Thank you both for taking a look!

EDIT: looking at wezterm's doc, it seems adding enable_csi_u_key_encoding = true in its config should do the trick.

Thanks for the suggestion. I can verify that using enable_csi_u_key_encoding = true fixes the issue, but I can't help but worry about this ominous warning for the feature toggle:

It is not recommended to enable this option as it does change the behavior of some keys in backwards incompatible ways and there isn't a way for applications to detect or request this behavior.

Am I opening myself up to a host of subtle bugs in other terminal programs? I hope not.

mawww commented 1 year ago

Well, ideally WezTerm would react to on of those escape sequence by setting CSI u mode:

        "\033[>4;1m"  // request CSI u style key reporting
        "\033[>5u"    // kitty progressive enhancement - report shifted key codes

My memory is a bit fuzzy on how those are specified and if Kakoune is compliant, but looking at WezTerm's code there does not seem to be any way to set enable_csi_u_key_encoding from the application directly, only the Xterm mode. It would be great for WezTerm's maintainer (and other terminal emulator writers) to come up with an agreed upon method of enable CSI u reporting.

dojoteef commented 1 year ago

@mawww are you suggesting that I create an issue on Wezterm's repo and link it here?

mawww commented 1 year ago

Yes, if you do not want to have to manually configure WezTerm with enable_csi_u_key_encoding Kakoune will need a way to tell WezTerm to do that, we current emit the two mentioned escape sequences that some terminal support. We can add one more for WezTerm if necessary, but it would be nice for terminal emulators to agree on a single sequence.

QiBaobin commented 1 year ago

For some reason, WezTerm decided to use CSI ~ instead of CSI u as most terminal emulator do, Kakoune parser does not currently support that. Adding it should be easy, but I think it would be great to first understand why WezTerm decided to deviate from common practice, and if it introduces other differences.

EDIT: looking at wezterm's doc, it seems adding enable_csi_u_key_encoding = true in its config should do the trick.

Another finding, with wezterm s-tab is same with [ in normal mode somehow.

Adding enable_csi_u_key_encoding=true works, but it makes the c-i not be same as tab, c-[ not be same as esc too.

key_event RawKeyEvent { key: Char('\t'), modifiers: SHIFT, phys_code: Some(Tab), raw_code: 48, repeat_count: 1, key_is_down: true, handled: Handled(false) }

krobelus commented 1 year ago

Adding enable_csi_u_key_encoding=true works, but it makes the c-i not be same as tab, c-[ not be same as esc too.

yes, that's expected. I added a mapping for c-[

theotherjimmy commented 1 year ago

Right, so this does not work for me.

I have enabled the enable_csi_u_key_encoding wezterm variable and type in programmer's dvorak. as such, all of my number keys are pressed in combination with the shift key. Further, map refuses to map them back to their respective numbers claiming:

Error: 1:1: 'map': Shift modifier only works on
special keys and lowercase ASCII, not '1'

for reference, this is what debug keys reports in kakoune:

 18│Client 'client0' got key '<s-5>'¬
 17│Client 'client0' got key '<s-4>'¬
 16│Client 'client0' got key '<s-3>'¬
 15│Client 'client0' got key '<s-3>'¬
 14│Client 'client0' got key '<s-2>'¬
 13│Client 'client0' got key '<s-0>'¬
 12│Client 'client0' got key '<s-7>'¬
 11│Client 'client0' got key '<s-5>'¬
 10│Client 'client0' got key '<s-4>'¬
  9│Client 'client0' got key '<s-3>'¬
  8│Client 'client0' got key '<s-2>'¬
  7│Client 'client0' got key '<s-0>'¬

and wezterm's debug log give the keycodes:

15:29:07.556  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[55;2u", Char('7') Modifiers(NONE | SHIFT)
15:29:07.677  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[53;2u", Char('5') Modifiers(NONE | SHIFT)
15:29:07.773  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[52;2u", Char('4') Modifiers(NONE | SHIFT)
15:29:07.811  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[51;2u", Char('3') Modifiers(NONE | SHIFT)
15:29:07.865  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[51;2u", Char('3') Modifiers(NONE | SHIFT)
15:29:07.881  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[50;2u", Char('2') Modifiers(NONE | SHIFT)
15:29:07.930  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[48;2u", Char('0') Modifiers(NONE | SHIFT)
15:29:08.110  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[55;2u", Char('7') Modifiers(NONE | SHIFT)
15:29:08.152  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[53;2u", Char('5') Modifiers(NONE | SHIFT)
15:29:08.217  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[52;2u", Char('4') Modifiers(NONE | SHIFT)
15:29:08.247  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[51;2u", Char('3') Modifiers(NONE | SHIFT)
15:29:08.293  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[50;2u", Char('2') Modifiers(NONE | SHIFT)
15:29:08.321  TRACE  wezterm_term::terminalstate::keyboard > key_down: sending "\u{1b}[48;2u", Char('0') Modifiers(NONE | SHIFT)

caksoylar commented 1 year ago

I don't think there is anything kakoune can do about that, wezterm seems to be receiving shifted keystrokes, isn't it? However you are implementing programmer's dvorak, wezterm still seems to be receiving non-translated HID keycodes from it.

theotherjimmy commented 1 year ago

Kakoune could let me rebind them, or it could treat <s-1> and 1 as the same.

wezterm seems to be receiving shifted keystrokes, isn't it?

This is true, but that's expected as programmer's dvorak swaps the symbol and numbers on the same key.

caksoylar commented 1 year ago

From what I know (but for programmable keyboards, not software remappers) the layout you use should be transparent to the applications: If you press shift and 1 keys to emit an actual 1 HID keycode then the application shouldn't receive the shift key together with 1. That's why I was wondering how you implemented programmer's dvorak which doesn't seem to work that way.

theotherjimmy commented 1 year ago

It's a software remapping: setxkbmap us -variant dvp

Screwtapello commented 1 year ago

I think this is a limitation of the CSI u/modifyOtherKeys encoding scheme, at least as it's implemented in WezTerm.

There's two main kinds of input that apps want from keyboards:

information about what keys are pressed (like a 104-button game controller) usually used for keyboard shortcuts
information about what text was typed, usually used for... typing

The traditional (VT220) terminal encoding is an unfortunate mixture of the two, where some key (such as F1 or Insert are represented as "base key + modifiers", while other key strokes (such as Shift+1) are represented as the text produced (such as "!").

The mapping between "base key + modifiers" and "text" is done by the system keyboard map, which terminal apps have no access to. If they care about modifiers (for example, Kakoune wants Shift to mean "extend") they have to guess what modifiers produced a particular symbol (for example, Kakoune assumes "?" means "extend /" even if you don't have to press Shift+/ to produce it).

As a result, Kakoune (and apps like it) do not want either of the above kinds of input - they want "text where possible, key presses otherwise". If every key press that produces a printable or ASCII character is represented by that character regardless of modifiers, then "key + modifier" representations can be used to represent non-printable characters, disambiguate methods of typing a printable character (Esc versus Ctrl+[, etc.), and so forth. This maximises compatibility with applications that do not understand the new encoding system, and with applications like Kakoune that want to work with terminals that do and don't support the new encoding system, without requiring reconfiguration.

It looks like WezTerm is doing the opposite of that happy path, however. Rather than only using CSI u/modifyOtherKeys where necessary, it seems to be representing "1" not as ASCII "1" or keypress Shift+! but as Shift+1.

theotherjimmy commented 1 year ago

Thanks @Screwtapello, as that gave me an idea: mapping shift + 1 to 1.

I tried the following wezterm config snippet:

  keys = {
      {
  key = '0',
  mods = 'SHIFT',
  action = wezterm.action.SendString "0",
},{
  key = '1',
  mods = 'SHIFT',
  action = wezterm.action.SendString "1",
},{
  key = '2',
  mods = 'SHIFT',
  action = wezterm.action.SendString "2",
},{
  key = '3',
  mods = 'SHIFT',
  action = wezterm.action.SendString "3",
},{
  key = '4',
  mods = 'SHIFT',
  action = wezterm.action.SendString "4",
},{
  key = '5',
  mods = 'SHIFT',
  action = wezterm.action.SendString "5",
},{
  key = '6',
  mods = 'SHIFT',
  action = wezterm.action.SendString "6",
},{
  key = '7',
  mods = 'SHIFT',
  action = wezterm.action.SendString "7",
},{
  key = '8',
  mods = 'SHIFT',
  action = wezterm.action.SendString "8",
},{
  key = '9',
  mods = 'SHIFT',
  action = wezterm.action.SendString "9",
},

Which worked!!!

FreeFull commented 1 month ago

Seems like this (shift+tab not working) can be worked around by setting enable_kitty_keyboard = true in wezterm's config

mawww / kakoune