search

wez / wezterm

A GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust
https://wezfurlong.org/wezterm/
Other
17.77k stars 795 forks source link

An Investigation into Unresponsiveness and Performance Regression in WezTerm related to Font shaping #5280

Open vvvvv opened 7 months ago

vvvvv commented 7 months ago

What Operating System(s) are you seeing this problem on?

macOS

WezTerm version

WezTerm Nightly 20240405-180910-cce0706b

Preface (feel free to skip to the main issue):

A few months ago I encountered performance issues in WezTerm characterized by unresponsive and laggy behavior, but I wasn't certain of the cause.
Having recently purchased a new MacBook Pro running a beta version of macOS and taking this opportunity to rewrite my nvim config there were several potential factors for these issues.
My primary use of the terminal meant I noticed these performance problems there, but a quick online search led me to believe that this issue stemmed from the new macOS version as numerous users across various applications were experiencing performance issues with the new macOS version.
Periodically I checked WezTerm's issues for similar reports, finding some, but pinpointing the exact cause was challenging due to its elusive nature and lack of reproducibility.
@wez made some educated guesses proposing fixes that seemed to help.
After applying these fixes to my config the problem seemed mitigated but given the issue's elusive nature it was difficult to test their effectiveness fully and I eventually encountered the same problems again. A couple of days ago I encountered this unresponsiveness again while working in nvim and started to investigate thoroughly. After deactivating nvim plugins step by step I identified the offending plugin.
Profiling nvim however, showed no unusual spikes anywhere and no unresponsiveness directing my attention towards WezTerm.
Through extensive testing, debugging, and profiling, I might have discovered the rather unexpected cause of the issue.

The Issue:

I found that configuring WezTerm to use fonts supporting ligatures resulted in decreased responsiveness, even when ligatures were disabled.
Surprisingly certain Unicode symbols (which aren't ligatures to my knowledge) worsened the responsiveness even more.
One such symbol is the ⋅ (U+22C5).
My profiling efforts focused on this symbol but I have a hunch other symbols might cause similar problems. The issue becomes more pronounced in maximized windows and the lagginess is visible even without profiling.

Methodology:

I experimented with various freetype load flags and harfbuzz features (calt=0, clig=0, liga=0) but the issue persisted.
Testing multiple fonts (JetBrainsMono Nerd Font, FiraCode Nerd Font Mono, Hasklug Nerd Font Mono) all supporting ligatures did not resolve the problem.
However, switching to Hack Nerd Font, which lacks ligature support, eliminated the issue.
I wrote a bash script which outputs a bunch of random strings, prefixed by U+22C5 to trigger the issue persistently and a minimal WezTerm config.

Profiling Results:

Profiling exposed that substantial CPU time was spent in wezterm_font::shaper::harfbuzz::HarfbuzzShaper::do_shape function, particularly in the call chain involving do_shape -> call_metric -> FT_Load_Glyph -> tt_glyph_load -> load_truetype_glyph -> TT_Hint_Glyph -> TT_Runins.

1. JetBrainsMono Nerd Font Mono - reproduce.sh default args:

2. JetBrainsMono Nerd Font Mono - reproduce.sh --print-inline:

3. JetBrainsMono Nerd Font Mono - reproduce.sh --line-prefix "":

4. JetBrainsMono Nerd Font Mono - reproduce.sh --timeout 60:

5. FiraCode Nerd Font Mono - reproduce.sh default args:

6. JetBrainsMono Nerd Font Mono - reproduce.sh default args - wezterm.lua 'calt=0', 'clig=0', 'liga=0':

7. WezTerm bundled font JetBrains Mono - reproduce.sh default args:

8. Hack Nerd Font (does not support ligatures) - reproduce.sh default args:

Notice how in run 3 which didn't print U+22C5 still spends 53% inside wezterm_font::shaper::harfbuzz::HarfbuzzShaper::do_shape versus how run 8 which uses a font without any ligature support only spends 24% there.
Notice also how run 7 which uses the bundled font spends only 50% in wezterm_font::shaper::harfbuzz::HarfbuzzShaper::do_shape even though it supports ligatures.

All profiling was done on a MacBook M3 Pro 2023, 36GB RAM, macOS Sonoma 14.4 using WezTerm nightly 20240405-180910-cce0706b.

To Reproduce & Configuration

I created a repo that contains reproduce.sh, the minimal wezterm config, and the tar.xz profiling which requires macOS Instruments (ships with XCode) to be viewed.

I'm sorry I abused the feature request form but the predefined bug issue form was too restrictive to share my findings.
If I can help further please let me know.
I didn't have time to dive into the codebase yet, but I'm not sure if I'd be of any help here as rust is far from my expertise.

vvvvv commented 7 months ago

They say once you explain your problem, you know the answer.

I took a closer look at ⋅ (U+22C5) and realized I had mistaken it for · (U+0387). (Gotta love it when after spending hours hunting down this behavior, it always comes down to some stupid oversight) It appears that only Hack Nerd Font supports U+22C5, unlike any of the other fonts mentioned above. U+0387 is supported and doesn't cause any issues.
So my guess is that when using e.g. JetBrains, U+22C5 falls back to a system font, which according to core_text.rs is Menlo.

From reading the codebase, I understand that glyphs are being cached, but since this glyph is reshaped repeatedly my guess is it doesn't get cached.
Is this the code path responsible for why it doesn't get cached?

Edit: This line hasn't changed in 3 years, so it's probably not the culprit. I haven't used WezTerm on macOS for the past 10 months or so, but I hadn't had this issue before that.

wez commented 6 months ago

Hey, thanks for taking the time to dig into this!

If you're in a position to build from source, could you try with this small change to see if not unloading unused fonts helps?

diff --git a/wezterm-font/src/shaper/harfbuzz.rs b/wezterm-font/src/shaper/harfbuzz.rs
index 9da18d524..d8e4d4983 100644
--- a/wezterm-font/src/shaper/harfbuzz.rs
+++ b/wezterm-font/src/shaper/harfbuzz.rs
@@ -542,7 +542,7 @@ impl HarfbuzzShaper {

         if !shaped_any {
             if let Some(opt_pair) = self.fonts.get(font_idx) {
-                if direct_clusters == 0 {
+                if direct_clusters == 0 && false {
                     // If we've never shaped anything from this font, and we didn't
                     // shape it just now, then we're probably a fallback font from
                     // the system and unlikely to be useful to keep around, so we

Regarding caching: the way that shaping works is context dependent, so we cache based on the line rather than individual cells, because the combinations are what impact shaping. If the lines are not changing in your test, then caching should work very well. If they are changing continually, then the cache will be thrashed and of limited use.