macos: send_composed_key_when_alt_is_pressed = false is not easily discoverable

krishnakumarg1984 commented 4 years ago

Describe the bug

Initially, I was unsure whether this is a bug or a feature request. I filed this as a bug because this is absolutely expected in a terminal emulator (atleast for my use as a daily driver).

Readline keybindings (also sometimes called emacs-style keybindings are an integral part of many core terminal workflows. They are shipped by default in bash. A whole host of applications uses them directly for line-editing. Important examples include R and ipython.

The issue here is the use of the meta key on macOS (since the macbook keyboard just doesn't have it, and the default terminal.app doesn't support it). However, it is usually mapped to the alt key on Windows and Linux, which facilitates movements such as moving back by a word or character etc.

However, the macbook has two option keys. Cleverly, iterm2 allows the mapping of the left option and right option keys separately. For instance, I have mapped these to escape and normal respectively. This allows me to use the left option key as the meta key (for readline motions on the command-line) and the right option key for things like accessing the # character (for passwords containing the octothorpe, for use inside inside vim for commenting lines of code in python, R etc.). See the screenshot below from iterm2.

Environment

OS: macOS
Version: wezterm 20200608-110940-3fb3a61

To Reproduce

At the bash prompt type an arbitary sentence eg. Mary had a little lamb. With the cursor at the very end, try left_option + b. The cursor is now expected to move back to the start of the word lamb, and so on. However, currently with wezterm, it ends up deleting the previous word, and a few other unexpected movements happen.

Configuration

No specific configuration for wezterm was used. This issue is present even with the default install.

Expected behaviour

With left_option, obey readline meta (eg. left_option + b, left_option + h etc) With right_option + #, insert the literal character #. (On a UK keyboard, the characters on this key are 3, £ and #) Make this configurable (for macOS) within the wezterm's .lua config file.

Screenshot

From iterm2:

wez commented 4 years ago

I just stepped away from my mac so can't immediately run through the steps in your report, but can share that there are a couple of options that you may wish to experiment with; these can affect how alt is interpreted on macOS.

I hadn't thought about making the left and right modifiers have different effects on the input; I'll give it some thought!

return {
   -- Rather than emitting fancy composed characters when alt is pressed, treat the
   -- input more like old school ascii with ALT held down
   send_composed_key_when_alt_is_pressed = false,

   -- similarly, don't ask the macOS IME/text services to compose input
   use_ime = false,
}

wez commented 4 years ago

I think there are really two issues blended together in this one:

The default Option key behaviour on macOS being tricky for English vs. other keyboard layouts
Treating the left and right modifiers independently in the key binding layer

Would you mind opening the latter as a separate feature request? Then we can focus this one on understanding the option key behaviour that you want to achieve.

wez commented 4 years ago

Please also check that your LANG environment is configured appropriately: https://wezfurlong.org/wezterm/faq.html#lang-and-locale

wez commented 4 years ago

I'm classifying this issue as a documentation problem:

It's not clear that macos defaults to sending composed keys (which we do for a better out of the box experience for international users), or what the option is to influence this
It's not clear that the LANG environment needs to be set to a UTF-8 locale in order for the composed keys to come through intact (I'll look at setting a reasonable LANG by default on macOS)

krishnakumarg1984 commented 4 years ago

I just stepped away from my mac so can't immediately run through the steps in your report, but can share that there are a couple of options that you may wish to experiment with; these can affect how alt is interpreted on macOS.

I hadn't thought about making the left and right modifiers have different effects on the input; I'll give it some thought!
return {
   -- Rather than emitting fancy composed characters when alt is pressed, treat the
   -- input more like old school ascii with ALT held down
   send_composed_key_when_alt_is_pressed = false,

   -- similarly, don't ask the macOS IME/text services to compose input
   use_ime = false,
}

@wez This completely messes things up for me. I am on a UK English keyboard where in the 3,#,£ are all located on the same key. With this change you proposed, I am unable to enter # at the terminal.

Instead when I press option + 3, my prompt suddenly becomes (arg: 3) which I have to exit by pressing Control-C. When I press shift + 3, I get # but immediately followed by a <CR> that executes whatever typed thus far as a shell command.

i think the real solution to access all three characters is to separate the functionality into the left and right option keys. This is how people use iterm2 for a UK English keyboard!

wez commented 4 years ago

As I mentioned above, there are two separate issues being conflated here.

First, I'd like to tackle the first one of those which is making sure that you have LANG set appropriately and are able to control whether ALT sends composed keys (which sounds equivalent to the "normal" behavior in iterm2) or whether it sends ESC+ the uncomposed key.

The ability to change the behavior of left and right alt keys independently is a separate issue.

krishnakumarg1984 commented 4 years ago

My $LANG is en_GB.UTF-8.

With the current default setting of wezterm, a few other keys of my UK Keyboard macbook does not work. To be fair, among all terminal emulators I have evertried, only iterm2 has got it right.

wez commented 4 years ago

In about an hour's time, the latest nightly build will include the changes referenced above which:

pre-define LANG to something based on your system settings, which should avoid issues with your shell and/or tmux with it being unset.
Default Left-Alt to generate Esc+Key and Right-Alt to generate composed output

There are new config settings for the Alt behavior; they are described in the changelog.

Note that only macOS supports separate left vs right alt behavior at this time. I didn't see a way to make X11/Wayland do this and Windows is notoriously bad at this (but does already have some knowledge of real AltGr support). So I'm not sure if this will ever apply beyond macOS right now.

krishnakumarg1984 commented 4 years ago

My £ key (above 3) is no longer reachable after these fixes.

wez commented 4 years ago

When I run wezterm -n (which runs with the default configuration), and set macOS to British English keyboard layout I see the following behaviour:

3 unmodified produces 3
SHIFT + 3 produces £
Right Alt + 3 produces #
Left Alt + 3 produces ESC 3

Can you provide more details on what you mean by "no longer reachable"? Running od -c and typing into it can be helpful to see what is being sent to the terminal.

Please update this issue with your current configuration.

krishnakumarg1984 commented 4 years ago

I ran wezterm -n and with macOS British English keyboard layout, I see the following behaviour

3 unmodified produces 3 (Expected behaviour. So good.)
SHIFT + 3 echoes # followed by Return/Enter (NOT okay. Should produce £)
Right Option + 3 produces # (Expected behaviour. So good.)
Left Option + 3 produces ESC 3 (Expected behaviour. So good.)

The current configuration is:


local wezterm = require 'wezterm';
return {
    color_scheme = "Gruvbox Dark",
    -- font = wezterm.font("JetBrains Mono"),
    font = wezterm.font_with_fallback({
        "CaskaydiaCove Nerd Font Mono",
        -- This is searched for glyphs that aren't found in the first one
        "JetBrains Mono",
        -- This is searched for glyphs that aren't found in either of
        -- the first two
        -- "My third best font",
    }),
    font_rules = {
        -- Define a rule that matches when italic text is shown
        {
            -- If specified, this rule matches when a cell's italic value exactly
            -- matches this.  If unspecified, the attribute value is irrelevant
            -- with respect to matching.
            italic = true,

            -- Match based on intensity: "Bold", "Normal" and "Half" are supported
            -- intensity = "Normal",

            -- Match based on underline: "None", "Single", and "Double" are supported
            -- underline = "None",

            -- Match based on the blink attribute: "None", "Slow", "Rapid"
            -- blink = "None",

            -- Match based on reverse video
            -- reverse = false,

            -- Match based on strikethrough
            -- strikethrough = false,

            -- Match based on the invisible attribute
            -- invisible = false,

            -- When the above attributes match, apply this font styling
            font = wezterm.font("CaskaydiaCove Nerd Font Mono", {italic=true}),
            font = wezterm.font("JetBrains Mono", {italic=true}),
        }
    },
    -- The font size, measured in points
    font_size = 15.0,
    -- How many lines of scrollback you want to retain per tab
    scrollback_lines = 3500,

    -- Enable the scrollbar.  This is currently disabled by default.
    -- It will occupy the right window padding space.
    -- If right padding is set to 0 then it will be increased
    -- to a single cell width
    enable_scroll_bar = true,

    -- What to set the TERM variable to
    term = "xterm-256color",

    -- Constrains the rate at which output from a child command is
    -- processed and applied to the terminal model.
    -- This acts as a brake in the case of a command spewing a
    -- ton of output and allows for the UI to remain responsive
    -- so that you can hit CTRL-C to interrupt it if desired.
    -- The default value is 400,000 bytes/s.
    ratelimit_output_bytes_per_second = 400000,

    -- Constrains the rate at which the multiplexer server will
    -- unilaterally push data to the client.
    -- This helps to avoid saturating the link between the client
    -- and server.
    -- Each time the screen is updated as a result of the child
    -- command outputting data (rather than in response to input
    -- from the client), the server considers whether to push
    -- the result to the client.
    -- That decision is throttled by this configuration value
    -- which has a default value of 10/s
    ratelimit_mux_output_pushes_per_second = 10,

    -- Constrain how often the mux server scans the terminal
    -- model to compute a diff to send to the mux client.
    -- The default value is 100/s
    ratelimit_mux_output_scans_per_second = 100,

    -- If false, do not try to use a Wayland protocol connection
    -- when starting the gui frontend, and instead use X11.
    -- This option is only considered on X11/Wayland systems and
    -- has no effect on macOS or Windows.
    -- The default is true.
    enable_wayland = true,

    -- Specifies how often a blinking cursor transitions between visible
    -- and invisible, expressed in milliseconds.
    -- Setting this to 0 disables blinking.
    -- Note that this value is approximate due to the way that the system
    -- event loop schedulers manage timers; non-zero values will be at
    -- least the interval specified with some degree of slop.
    -- It is recommended to avoid blinking cursors when on battery power,
    -- as it is relatively costly to keep re-rendering for the blink!
    cursor_blink_rate = 800,

    -- Specifies the default cursor style.  various escape sequences
    -- can override the default style in different situations (eg:
    -- an editor can change it depending on the mode), but this value
    -- controls how the cursor appears when it is reset to default.
    -- The default is `SteadyBlock`.
    -- Acceptable values are `SteadyBlock`, `BlinkingBlock`,
    -- `SteadyUnderline`, `BlinkingUnderline`, `SteadyBar`,
    -- and `BlinkingBar`.
    default_cursor_style = "SteadyBlock",

    -- similarly, don't ask the macOS IME/text services to compose input
    use_ime = false,
}

wez commented 4 years ago

Can you run xxd and then press Shift 3 and then Enter and share the output here? I'd like to see exactly what string that is sending in the hexdump. I don't have an explanation for why that behaves differently for you; in theory my test setup should behave the same as yours.

krishnakumarg1984 commented 4 years ago

in xxd, shift+3 and return produces £. I don't see any hexdump string at all.

krishnakumarg1984 commented 4 years ago

The first few keystrokes show the effect of pressing SHIFT + 3 at the bash prompt. No ~/.bashrc, no ~/.bash_profile, no ~/.profile. This simply echoes # followed by RETURN.

The last line shows the effect of entering SHIFT + 3 within xxd.

wez commented 4 years ago

Interesting; that sounds like it may be related to the shell/line editor. Do you have an .inputrc that might influence readline? Can you share the output from env?

Running wezterm -n start -- bash --norc to start bash with no profiles loaded and the British keymap activated gives me this when I press Shift 3:

Screen Shot 2020-06-22 at 17 41 23

krishnakumarg1984 commented 4 years ago

wezterm -n start -- bash --norc still gave me the same behaviour (# followed by Carriage Return) that I described just a few moments ago.

Yes, ~/.inputrc was the culprit, specifically the below line

# Allow Meta (Alt) key bindings [line added to the top of my `.inputrc'].
set convert-meta on

As per the GNU readline manual,

convert-meta

If set to ‘on’, Readline will convert characters with the eighth bit set to an ASCII key sequence by stripping the eighth bit and prefixing an ESC character, converting them to a meta-prefixed key sequence. The default value is ‘on’, but will be set to ‘off’ if the locale is one that contains eight-bit characters.

Do you know why iterm works despite this setting?

wez commented 4 years ago

Do you have iTerm configured to emit UTF-8 or iso-8859-1? You can probably test for this by running xxd then pressing Shift 3 then Enter then CTRL D; it should print the hex dump. Try that in both wezterm and iTerm.

If those are the same then I suspect that there is something different between the environments (eg: LANG or LC_XXX related environment) that impacts how the data is interpreted.

krishnakumarg1984 commented 4 years ago

You can probably test for this by running xxd then pressing Shift 3 then Enter then CTRL D; it should print the hex dump. Try that in both wezterm and iTerm.

In both iterm and wezterm, I get the following hexdump string:

00000000: c2a3 0a

As I said earlier, I have localised the issue to the ~/.inputrc configuration. set convert-meta on. We should perhaps focus on that (I don't know how though)?

wez commented 4 years ago

Please share the output from env in both terminals.

krishnakumarg1984 commented 4 years ago

Here is the output of diff -w -u iterm_env.txt wezterm_env.txt . Please let me know if you'd like to see the two (rather large) individual env outputs?

Note: wezterm is executed as wezterm -n, and weirdly I see that the $PATH is different? Why? Is wezterm missing the sourcing of any startup file for the bash shell?

Secondly, the __CF_USER_TEXT_ENCODING (last entry below) seems to be different in wezterm. Why?

--- iterm_env.txt   2020-06-24 16:49:30.727480732 +0100
+++ wezterm_env.txt 2020-06-24 16:49:52.094179401 +0100
@@ -33,20 +33,20 @@
 LOGNAME=krishna
 LSCOLORS=exfxcxdxbxegedabagacxx
 LS_COLORS=di=34:ln=35:so=32:pi=33:ex=31:bd=34:cd=34:su=0;41:sg=0;46:tw=0;42:ow=33
-OLDPWD=/Users/krishna/deal.ii-candi/configuration
+OLDPWD=/Users/krishna
 OMPI_CC=gcc-9
 OMPI_CXX=g++-9
 OMPI_FC=gfortran-9
-PATH=/Users/krishna/miniconda3/bin:/Users/krishna/miniconda3/condabin:/Users/krishna/node-v12.16.1-darwin-x64/bin:/Users/krishna/bin:/Users/krishna/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/krishna/.fzf/bin
+PATH=/Users/krishna/node-v12.16.1-darwin-x64/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/krishna/miniconda3/bin:/Users/krishna/miniconda3/condabin:/Users/krishna/node-v12.16.1-darwin-x64/bin:/Users/krishna/bin:/Users/krishna/.cargo/bin:/Users/krishna/.fzf/bin
 PS_FORMAT=pid,ppid,user,pri,ni,vsz,rss,pcpu,pmem,tty,stat,args
 PWD=/Users/krishna/Desktop
 SHELL=/usr/local/bin/bash
-SHLVL=1
+SHLVL=2
 SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.4myFoO3TLr/Listeners
 SUDO_EDITOR=/usr/local/bin/nvim
-TERM=screen-256color
-TERM_PROGRAM=iTerm.app
-TERM_PROGRAM_VERSION=3.3.11
+TERM=xterm-256color
+TERM_PROGRAM=WezTerm
+TERM_PROGRAM_VERSION=20200620-160318-e00b076c
 TERM_SESSION_ID=w0t0p0:FBDE88EC-5C4C-48ED-9881-84890B2FF5F2
 TMPDIR=/var/folders/lq/3qgr399x5bv_fzzvvf2vytg40000gn/T/
 TMUX=/private/tmp/tmux-501/default,24740,0
@@ -54,10 +54,12 @@
 TMUX_PLUGIN_MANAGER_PATH=/Users/krishna/.tmux/plugins/
 USER=krishna
 VISUAL=/usr/local/bin/nvim
+WEZTERM_EXECUTABLE=/usr/local/Cellar/wezterm/20200620-160318/WezTerm.app/wezterm
+WEZTERM_EXECUTABLE_DIR=/usr/local/Cellar/wezterm/20200620-160318/WezTerm.app
 WINDOWID=0
 XPC_FLAGS=0x0
 XPC_SERVICE_NAME=0
 _=/usr/bin/env
 _CE_CONDA=
 _CE_M=
-__CF_USER_TEXT_ENCODING=0x0:0:0
+__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0

wez commented 4 years ago

For me, iTerm behaves identically to wezterm when it comes to inputting £; convert-meta does replace it with a3 0a as you described, but it does so for both iterm and wezterm.

So there is presumably something in your iterm configuration that is changing the behavior on your system.

I have no motivation to troubleshoot a software stack I don't maintain!

In terms of next steps, I'll add some information to the FAQ to mention this .inputrc configuration as something that influences encoding behavior, and then I plan to close this issue.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

wez / wezterm