Unicode utf-8 unexpected behavior

[elpupi]

What Operating System(s) are you seeing this problem on?

Linux X11

WezTerm version

20211017-131416-24875004

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

Unicode utf-8 characters behaves strangely. Indeed, if I copy/paste "፨ style 🠒" in the wezterm, the output is is well displayed.

But weirdly, if I write few lines, using the stdin of the cat command for example, as so:

Colors 16

፨ bg 🠒 [30;100mblack[0m ፨ style 🠒 bold    ፨ bg 🠒 [31mred[0m ፨ style 🠒 bold          
፨ bg 🠒 [32mgreen[0m ፨ style 🠒 bold        ፨ bg 🠒 [33morange[0m ፨ style 🠒 bold

I get various weird behaviors :cry: :disappointed:

Indeed, here, you can see that the lines are well displayed:

But if I reload the config (Ctrl-Shift-R or what you have in your config key binding), at one point, all the unicode chars get unrecognized. In this case, I had to paste 3 times the same text block.

I tried to create a minimal case but the behavior is apparently a bit unpredictable and I could not find a reproductible test case unfortunately.

Something even weirder is that from time to time, when I open a fresh wezterm, and I write the exact same command as previously that was working, the bug arises right away like with this echo "፨ style 🠒"

Something I found out, is that if I type any unicode, but only one char, like "፨" for instance, the terminal outputs well the unicode chars again (that explains why you see this ፨ unicode char at the beginning of the first screenshots)

To Reproduce

As I mentioned it in the description, the behavior is a bit chaotic and not really reproducible. Each time, the bug arises but at a different moment. Here I will describe one test case that triggered the bug for me.

1. Open wizterm
2. Copy/paste "፨ style 🠒" in the wezterm, the output is is well displayed.
3. After typing the command cat, paste the following text

   
   Colors 16

፨ bg 🠒 [30;100mblack[0m ፨ style 🠒 bold ፨ bg 🠒 [31mred[0m ፨ style 🠒 bold
፨ bg 🠒 [32mgreen[0m ፨ style 🠒 bold ፨ bg 🠒 [33morange[0m ፨ style 🠒 bold

4. Reload the config via the keyboard shortcut
5. If the bug is not triggered, repeat few time the operation pasting the text block and reload the config

After the bug arises, you can try:

1. Close the terminal and open a new instance
2. Copy/paste ```echo "፨ style 🠒"```
3. Check if the bug arises
4. If yes, close wezterm and open a new instance, then copy/paste only "፨" and unicode chars should get displayed well again

### Configuration

```lua
-- @@@file =>>>> cfg_fonts.lua

local wezterm = require "wezterm"

local cfg = {}

-- Disable annoying default behaviors
cfg.adjust_window_size_when_changing_font_size = false
-- !! this one opens a separate win on first unknown glyph, stealing windows focus !!
cfg.warn_about_missing_glyphs = false

cfg.font_size = 10.0

-- Makes FontAwesome's double-width glyphs display properly!
cfg.allow_square_glyphs_to_overflow_width = "WhenFollowedBySpace"

-- This tells wezterm to look first for fonts in the directory named
-- `fonts` that is found alongside your `wezterm.lua` file.
-- As this option is an array, you may list multiple locations if
-- you wish.
cfg.font_dirs = {"fonts"} -- relative to this config file

local function font_with_fallback(font_family)
    -- family names, not file names
    return wezterm.font_with_fallback({
        font_family,
        "Font Awesome 5 Free Solid" -- nice double-spaced symbols!
    })
end

local function font_and_rules_for_iosevka()
    -- Iosevka Font:
    -- + Has 2 variants for terminals: Term & Fixed. Fixed is same as Term but without ligatures.
    --   in the long run, I'd like to have a keybinding to enable/disable ligatures on demand,
    --   by switching font for example.
    --   --> for now, use Term (with ligatures)
    --
    -- + Has 2 additional variants for horizontal size: Normal & Extended. The Normal is the one
    --   which does not mention 'Extended'. Extended is wider than Normal.
    --   --> use Extended variant, the normal one is way too thin!!!
    local font = font_with_fallback("Iosevka Term Extended")
    local font_rules = nil
    -- It finds automatically all the required fonts, no need to specify all the variants!
    return font, font_rules
end

local function font_and_rules_for_jetbrains()
    -- Use a _very slightly_ lighter variant, so that regular bold really stand out
    local font = font_with_fallback("JetBrains Mono Light")
    local font_rules = {
        {italic = true, font = font_with_fallback("JetBrains Mono Light Italic")},
        {italic = true, intensity = "Bold", font = font_with_fallback("JetBrains Mono Bold Italic")},
        {intensity = "Bold", font = font_with_fallback("JetBrains Mono Bold")}
    }
    return font, font_rules
end

local function font_and_rules_for_cascadia()
    local font = font_with_fallback("Cascadia Code")
    -- local font = font_with_fallback("Cascadia Code Light")
    local font_rules = {
        -- NOTE: There is no Italic in font Cascadia...
        {italic = true, font = font_with_fallback("JetBrains Mono Italic")},
        {italic = true, intensity = "Bold", font = font_with_fallback("JetBrains Mono Bold Italic")},
        {
            intensity = "Bold",
            -- font = font_with_fallback("JetBrains Mono Bold"),
            font = font_with_fallback("Cascadia Code Bold")
        }
    }
    return font, font_rules
end

local function font_and_rules_for_nerd_hasklug()
    local font = font_with_fallback("Hasklug Nerd Font")
    -- local font_rules = {
    --   {
    --     italic = true,
    --     font = font_with_fallback("Hasklug Nerd Font Italic", {italic=true}),
    --   },
    --   {
    --     italic = true,
    --     intensity = "Bold",
    --     font = font_with_fallback("Hasklug Nerd Font Bold Italic", {italic=true,weight="bold"}),
    --   },
    --   {
    --     intensity = "Bold",
    --     font = font_with_fallback("Hasklug Nerd Font Bold", {weight="bold"}),
    --   },
    -- }
    local font_rules = nil
    return font, font_rules
end

-- FIXME (<- this is an example of bolded text)
-- 0 1 2 3 4 5 6 7 8 9
-- Some ligatures: != <-> <-  -> ----> => ==> ===> -- --- /../;;/ #{}
--  <> <!-- --> ->> --> <= >= ++ == === := a::b::c a&&b a||b

cfg.font, cfg.font_rules = font_and_rules_for_nerd_hasklug()
-- cfg.font, cfg.font_rules = font_and_rules_for_cascadia()
-- cfg.font, cfg.font_rules = font_and_rules_for_iosevka()

-- Enable various OpenType features
-- See https://docs.microsoft.com/en-us/typography/opentype/spec/featurelist
cfg.harfbuzz_features = {
    "zero", -- Use a slashed zero '0' (instead of dotted)
    "kern", -- (default) kerning (todo check what is really is)
    "liga", -- (default) ligatures
    "clig", -- (default) contextual ligatures
    "calt" -- (default) contextual Alternates
}

return cfg

-- @@@file =>>>> cfg_keys.lua

local wezterm = require "wezterm"
local act = wezterm.action

-- NOTE: always use wezterm.action_callback when it's in a release version!!
-- PR: https://github.com/wez/wezterm/pull/1151
local function act_callback(event_id, callback)
    if wezterm.action_callback then
        wezterm.log_info(">> cfg_key.lua: wezterm.action_callback is available for this version, use it!")
        return wezterm.action_callback(callback)
    else
        wezterm.log_info(
            ">> cfg_key.lua: wezterm.action_callback is NOT available for this version, fallback to manual setup..")
        wezterm.on(event_id, callback)
        return wezterm.action {EmitEvent = event_id}
    end
end

-- IDEA: A better action syntax, see: https://github.com/wez/wezterm/issues/1150

-- IDEA: helper for keybind definition
local function keybind(mods, key, action) return {mods = mods, key = key, action = action} end

local ctrl_shift = "CTRL|SHIFT"
local ctrl_alt = "CTRL|ALT"

local cfg = {}

cfg.disable_default_key_bindings = true

-- NOTE: About SHIFT and the keybind definition:
-- * For bindings with SHIFT and a letter, the `key` field (the letter)
--   can be lowercase and the mods should NOT contain 'SHIFT'.
-- * For bindings with SHIFT and something else, mod should contain SHIFT,
--   and key should be the shifted key that is going to reach the terminal.
--   (based on the keyboard-layout)
cfg.keys = {
    keybind("SHIFT", "PageUp", act {ScrollByPage = -1}),
    keybind("SHIFT", "PageDown", act {ScrollByPage = 1}), -- Wezterm features
    keybind(ctrl_shift, "r", "ReloadConfiguration"),
    keybind(ctrl_shift, "l", act {ClearScrollback = "ScrollbackAndViewport"}),
    keybind(ctrl_shift, "f", act {Search = {CaseInSensitiveString = ""}}),
    keybind(ctrl_shift, " ", "QuickSelect"),
    keybind(ctrl_alt, " ", "QuickSelect"), -- note: eats a valid terminal keybind
    keybind(ctrl_shift, "d", "ShowDebugOverlay"), -- note: it's not a full Lua interpreter
    -- Copy/Paste to/from Clipboard
    keybind(ctrl_shift, "c", act {CopyTo = "Clipboard"}),
    keybind("CTRL", "v", act {PasteFrom = "Clipboard"}),
    -- Paste from PrimarySelection (Copy is done by selection)
    keybind("SHIFT", "Insert", act {PasteFrom = "PrimarySelection"}),
    keybind(ctrl_shift, "v", act {PasteFrom = "PrimarySelection"}),
    -- keybind(ctrl_alt, "v",      act{PasteFrom = "PrimarySelection"}),
    -- NOTE: the last one eats a valid terminal keybind

    -- Tabs
    keybind("CTRL", "t", act {SpawnTab = "DefaultDomain"}),
    keybind("CTRL", "Tab", act {ActivateTabRelative = 1}),
    keybind(ctrl_shift, "Tab", act {ActivateTabRelative = -1}),
    keybind(ctrl_shift, "w", act {CloseCurrentTab = {confirm = false}}),
    keybind(ctrl_shift, "x", "ShowLauncher"), -- Font size
    keybind("CTRL", "0", "ResetFontSize"), -- Ctrl-Shift-0
    keybind("CTRL", "+", "IncreaseFontSize"), -- Ctrl-Shift-+
    keybind("CTRL", "-", "DecreaseFontSize"), -- Ctrl-Shift-- (key with -)

    ---- custom events
    keybind(ctrl_shift, "g", act_callback("toggle-ligatures", function(win, _)
        local overrides = win:get_config_overrides() or {}

        if not overrides.harfbuzz_features then
            -- If we haven't overriden it yet, then override with ligatures disabled
            overrides.harfbuzz_features = {"calt=0", "clig=0", "liga=0"}
        else
            -- else we did already, and we should disable the override now
            overrides.harfbuzz_features = nil
        end

        win:set_config_overrides(overrides)
    end)),

    keybind(ctrl_shift, "o", act_callback("toggle-opacity", function(win, _)
        local overrides = win:get_config_overrides() or {}
        if not overrides.window_background_opacity then
            overrides.window_background_opacity = win:effective_config().window_background_opacity;
        else
            overrides.window_background_opacity = nil
        end
        win:set_config_overrides(overrides)
    end))
}

return cfg

-- @@@file =>>>> cfg_mouse.lua

local wezterm = require "wezterm"
local mytable = require "lib/mystdlib".mytable

local cfg = {}

cfg.disable_default_mouse_bindings = true

local mouse_bindings = {}

-- Left click always starts a new selection.
-- The number of clicks determines the selection mode: 1:Cell 2:Word: 3:Line
function binds_for_mouse_select(button, streak, selection_mode)
  return {
    -- Select on Down event
    {
      mods="NONE",
      event={Down={streak=streak, button=button}},
      action=wezterm.action{SelectTextAtMouseCursor=selection_mode},
    },

    -- Extend on Drag event
    {
      mods="NONE",
      event={Drag={streak=streak, button=button}},
      action=wezterm.action{ExtendSelectionToMouseCursor=selection_mode},
    },

    -- Complete on Up event
    {
      mods="NONE",
      event={Up={streak=streak, button=button}},
      action=wezterm.action{CompleteSelection="PrimarySelection"}
    },
  }
end

table.insert(mouse_bindings, {
  binds_for_mouse_select("Left", 1, "Cell"),
  binds_for_mouse_select("Left", 2, "Word"),
  binds_for_mouse_select("Left", 3, "Line"),
})

-- Right click always extends the selection.
-- The number of clicks determines the selection mode: 1:Cell 2:Word: 3:Line
function binds_extend_mouse_select(button, streak, selection_mode)
  return {
    -- Extend the selection on Down & Drag events
    {
      mods="NONE",
      event={Down={streak=streak, button=button}},
      action=wezterm.action{ExtendSelectionToMouseCursor=selection_mode},
    },
    {
      mods="NONE",
      event={Drag={streak=streak, button=button}},
      action=wezterm.action{ExtendSelectionToMouseCursor=selection_mode},
    },

    -- Complete on Up event
    {
      mods="NONE",
      event={Up={streak=streak, button=button}},
      action=wezterm.action{CompleteSelection="PrimarySelection"}
    },
  }
end

table.insert(mouse_bindings, {
  binds_extend_mouse_select("Right", 1, "Cell"),
  binds_extend_mouse_select("Right", 2, "Word"),
  binds_extend_mouse_select("Right", 3, "Line"),
})

-- Ctrl-Left click (on Up) opens the link under the mouse pointer if any.
-- (on Down, the click is disabled. This is to avoid bugging the running
-- program which would receive _only_ the down event and not the up event)
table.insert(mouse_bindings, {
  mods="CTRL",
  event={Down={streak=1, button="Left"}},
  action="Nop",
})

table.insert(mouse_bindings, {
  mods="CTRL",
  event={Up={streak=1, button="Left"}},
  action="OpenLinkAtMouseCursor",
})

-- FIXME: I want this to work EVEN IF the current program enabled mouse-reporting.
--        Currently I have to press Ctrl-Shift-click to make this binding work.
-- Another binding override I'd love is the MiddleClick (to work even in tmux!)

-- Clipboard / PrimarySelection paste
table.insert(mouse_bindings, {
  -- Middle click pastes from the primary selection (for any other mods).
  wezterm.permute_any_or_no_mods({
    event={Down={streak=1, button="Middle"}},
    action=wezterm.action{PasteFrom = "PrimarySelection"},
  }),
  -- Alt-Middle click pastes from the clipboard selection
  -- NOTE: Must be last to overwrite the existing Alt-Middle binding done by permute_any_or_no_mods.
  {
    mods="ALT",
    event={Down={streak=1, button="Middle"}},
    action=wezterm.action{PasteFrom = "Clipboard"},
  },
})

  -- To simplify config composability, `mouse_bindings` is a
  -- nested list of (bind or list of (bind or ...)), so we must
  -- flatten the list first.
cfg.mouse_bindings = mytable.flatten_list(mouse_bindings)

return cfg

-- @@@file =>>>> wezterm.lua

-- WezTerm configuration
---------------------------------------------------------------
local mytable = require"lib/mystdlib".mytable

-- Misc
------------------------------------------

local cfg_misc = {
    window_close_confirmation = "NeverPrompt",
    check_for_updates = false,

    -- Avoid unexpected config breakage and unusable terminal
    automatically_reload_config = false,

    -- Make sure word selection stops on most punctuations.
    -- Note that dot (.) & slash (/) are allowed though for
    -- easy selection of paths.
    selection_word_boundary = " \t\n{}[]()\"'`,;:@",

    hide_tab_bar_if_only_one_tab = true,

    -- Do not hold on exit by default.
    -- Because the default 'CloseOnCleanExit' can be annoying when exiting with
    -- Ctrl-D and the last command exited with non-zero: the shell will exit
    -- with non-zero and the terminal would hang until the window is closed manually.
    exit_behavior = "Close",

    -- Pad window to avoid the content to be too close to the border,
    -- so it's easier to see and select.
    -- window_padding = {
    --   left = 3, right = 3,
    --   top = 3, bottom = 3,
    -- },

    color_scheme = "nord",
    window_background_opacity = 0.88

    -- cf the original issue (mine): https://github.com/wez/wezterm/issues/478 solved for me but not for everyone..
    -- cf the addition of this flag: https://github.com/wez/wezterm/commit/336f209ede27dd801f989419155e475f677e8244
    -- OK BUT NO, disabled because it does some weird visual artifacts:
    --  * About cursor behaviors:
    --    When a ligature is a the end of the line & the nvim' window
    --    is a little bit larger than the text so that when the cursor comes
    --    closer to the window border (and on the ligature), the buffer does
    --    a side-scroll. Then the cursor does wonky stuff when moving w.r.t that
    --    end-of-line ligature.
    --
    --  * About some symbols display:
    --    The git above/below arrows on the right of my prompt.
    --
    -- experimental_shape_post_processing = true,
}

-- Colors & Appearance
------------------------------------------

-- local cfg_colors = {
--   colors = require("cfg_bew_colors"),
-- }

-- Font
------------------------------------------

local cfg_fonts = require("cfg_fonts")

-- Key/Mouse bindings
------------------------------------------

-- Key bindings
local cfg_key_bindings = require("cfg_keys")

-- Mouse bindings
local cfg_mouse_bindings = require("cfg_mouse")

-- Merge configs and return!
------------------------------------------

local config = mytable.merge_all(cfg_misc, -- cfg_colors,
cfg_fonts, cfg_key_bindings, cfg_mouse_bindings, {} -- so the last table can have an ending comma for git diffs :)
)

return config

Expected Behavior

Unciode utf-8 characters should be well displayed and consistent.

Logs

2021-10-18T20:58:02.538Z INFO wezterm_gui::termwindow > OpenGL initialized! Mesa Intel(R) HD Graphics 620 (KBL GT2) 4.6 (Compatibility Profile) Mesa 21.0.3 is_context_loss_possible=false wezterm version: 20210814-124438-54e29167

Anything else?

No response

[bew]

Hehe I see you're using my multi-file config (with a few tweaks) :)

However I don't think all the config is necessary to reproduce the problem, can you try with no config? (with: wezterm -n) or with a more minimal config? (to configure the font only maybe? since it might be a problem with font loading) (with: wezterm --config-file ./minimal-config.lua) The idea is to find the minimal config necessary to reproduce the problem, and help find the root issue.

[wez]

I'm a bit preoccupied with moving house at the moment, so haven't had time to really dig in. If you could minimize the amount of configuration necessary to trigger the issue, that would help!

My first thought is that reloading the config is causing the font directory to be rescanned to search for a font with appropriate fallback glyphs; if you have a lot of fonts in there then it may take some time for the codepoint coverage to be evaluated. While a proper codepoint->glyph remains unresolved, wezterm substitutes a glyph from https://github.com/unicode-org/last-resort-font

If you start wezterm like this: WEZTERM_LOG=wezterm_font=trace,info wezterm, stderr should show a lot of diagnostics about font loading, including statistics about computing coverage.

If your fonts are resolvable via fontconfig, then you may wish to remove the cfg.font_dirs line from your config; wezterm will then use the available coverage information from fontconfig and won't need to build its own font database.

[no-response[bot]]

This issue has been automatically closed because there has been no response to the request for more information from the original author. With only the information that is currently in the issue, there isn't enough information to take further action. Please reach out if you have or find the answers we need so that we can investigate further.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.