Terminal.app is very slow with Pragmata fonts

[cedeber]

Both of Pragmata (normal and mono) make Terminal.app very slow on macOS. Everything runs perfectly with SF Mono for instance, so it doesn't look like that it is a generic performance problem (I reset the Terminal to be sure) Strangely, PragmataPro Mono Bold is not affected and the Terminal runs smoothly.

[cedeber]

@kelas the squares render correctly on the term, it’s a big fat right arrow. I guess it’s simply not in the monospace font the browser uses.

[carloscabanero]

@kelas I can speak for Blink. We are a heavily modified HTerm and use WebKit for this calculation. It is possible that this is not a very optimized path within the rendering engine, but at the same time it is a very specific gesture (pinch to zoom) that “feels” slow just because you are used to a smoother experience. Not a gigantic deal and no one complained, but I feel it.

That’s why I was curious to know if there may be something else to be optimized on either side. Because whatever calculations take place, they only take this long with Pragmata.

[kelas]

@cedeber Please cat the contents of this file into a fresh term window and let me know if any specific areas seem to produce worse lag than the others during scroll. All_chars.txt

[dop]

All_chars.txt worked fine with Pragmata Pro, but had some small lag with Pragmata Pro Mono where the glyphs were missing (rendered as question marks or native macOS emojis.)

I think I just figured what makes Terminal.app slow for me. It's colors! I set my prompt to have blue color so that I can distinguish it easier from the output and it makes the Terminal.app crawl.

This reproduces it for me:

#!/bin/sh

BLUE="$(tput setaf 27)"
RESET="$(tput sgr0)"

for i in `seq 1 100`; do
    echo "${BLUE}>${RESET} $i"
done

[kelas]

@dop bravo, this connects. I get the same.

[kelas]

@dop And now the fun part. if you disable ANSI colors and anti-aliasing in the profile, what do you get? :)

[dop]

It still scrolls terribly slow. Is that expected?

[kelas]

None of this is expected. No wonder this went uncaught for so long. It is not just colors, it is specific sequences!

This kills the scroll:

#!/bin/sh

BLUE="$(tput setaf 27)"
RESET="$(tput sgr0)"

for i in `seq 1 100`; do
    echo "1${BLUE}0${RESET}"
done

And this doesn't:

#!/bin/sh

BLUE="$(tput setaf 27)"
RESET="$(tput sgr0)"

for i in `seq 1 100`; do
    echo "${BLUE}10${RESET}"
done

But anyway — we finally have a reproducible case, and can apply divide and conquer as @schriftgestalt suggests.

@dop very nice catch, amazing intuition.

[kelas]

Happy to let everyone know that the issue is finally isolated.

Those who think Terminal.app is still a good idea in A.D. 2018, expect good news soon :)

[schriftgestalt]

I pinged a contact at Apple and he told me that you should file a bug. If you do, post the radar number that I can send it to him.

[kelas]

@schriftgestalt The horrors I have seen in the past few hours looking at Terminal's process stack samples trying to understand what is it burning 100% CPU on while scrolling a trivial terminal window set in Pragmata or Iosevka, make me think Terminal authors are the last people I wanna deal with :) Hope they are still around, because the software looks a bit summer 2000.

The issue seems pretty clear, Terminal is badly confused by the fact that Pragmata is not monospaced and has additional substitution features.

Still have to discuss with @fabrizioschiavi, but I think there is a relatively painless solution. @dop 's amazing insight came right on time :)

And thank you!

[kelas]

@carloscabanero sorry for the wait. I'm Blink's paying customer for a while now, always thought mosh was a strong idea, but I've been out of touch with mobile terminals lately.

So basically I understand you're having issues achieving a responsive zoom of a text viewport set in PPro. As you figured, I've seen plenty of this lovely font lately, including some major performance bottlenecks in a scenario similar to yours — only the surface had a bit more pixels and the target frame rate was a bit higher. Responsiveness is never easy, but there is always a hack which is sometimes hard to spot. And it is never the font's fault, I realized :) You're not necessarily completely bound by Webkit's text layout performance, and what you're seeing with Pragmata could be a hint at a more general potential improvement.

[schriftgestalt]

But as you have a proper test case, filing a bug is easy. And it might actually get fixed.

[kelas]

@carloscabanero Only when everything else fails trying to achieve something truly responsive, people finally quit trying to make sense of countless Apple's deprecated APIs and remember that SGI gave us OpenGL sometime around mid 90's.

Sublime Text, for example, had a major difficulty keeping their viewport responsive when Retina popped out, so pretty much the only thing you see these days when you scroll a document in Sublime Text on a Mac is an OpenGL front buffer (although not without some non-portable, poorly documented IOSurface in the back).

[kelas]

@schriftgestalt It is all up to @fabrizioschiavi anyway. I only drive the bus :)

[kelas]

Why am I seeing random/weird colors?

Most likely because your terminal does not support 24-bit RGB color escape sequences. This in turn probably means that you are using macOS' default Terminal.app, which is the only major terminal emulator still missing that feature. [...] Switch to a terminal that supports true color escape sequences. A well-researched list of such terminals can be found here.

[kelas]

I’ve noticed that Terminal.app slows dramatically when outputting non-latin unicode ranges. I’m aware of three things that might cause this: having to load different font pages, and having to parse code points outside of the BMP, and wide characters. The first probably boils down to a very complicated mix of lazy loading of font glyphs, font fallback calculations, and caching of the glyph pages or however that works. The second is a bit speculative, but I would bet that Terminal.app uses Cocoa’s UTF16-based NSString, which almost certainly hits a slow path when code points are above the BMP due to surrogate pairs.

I might add that in case of colour output NSString becomes NSAttributedString, which makes matters an order of magnitude worse.

Sorry about the spam, but all of this just might save someone a lot of time. We get to suffer, but the next guy doesn't have to.

[kelas]

@dop Building on your idea, here is a fun way to benchpress a terminal emulator with a given font:

curl -OJ https://github.com/haasn/interpolation-samples/raw/master/60fps/native.mkv
TERM=ansi
HD="--vo-tct-width=96 --vo-tct-height=54"
PLAY="mpv --vo tct --vo-tct-256=yes --vo-tct-algo=plain"
$PLAY $HD native.mkv

$PLAY $HD native.mkv > stream.txt
cat stream.txt

The above is best enjoyed at silky 60fps in iTerm2 and snappy 2000fps in Alacritty. Back at Terminal.app, we take comfort in counting dropped frames and drawing comparisons:

On a more serious note, a modified version of @dop's script can be used to show that any font, ligature or not, monospaced or not, can render Terminal.app unusable. At 10pt in a 134x45 window running on macOS Mojave, the output of the following script causes completely substandard latency with Fira Code, Hasklig, Iosevka and Menlo. @fabrizioschiavi I now agree with @schriftgestalt this should be reported as defect. A modern terminal emulator doesn't have to be this slow, really.

#!/bin/sh

TERM=ansi

bold=$(tput bold)
r=$(tput setaf 1)
g=$(tput setaf 2)
b=$(tput setaf 4)
R=$bold$r
G=$bold$g
B=$bold$b
X=$(tput sgr0)

# disable colors
#unset r g b R G B X bold

# ppro, iosevka, fira
L1="->"; L2="<-"; L3="!=" ; L4="0>="; L5="1<="; L6="2=="

t="${r}R${X}${g}G${X}${b}B${X}${r}${L1}${X}${g}${L2}${X}${b}${L3}${X}${r}${L4}${X}${g}${L5}${X}${b}${L6}${X}"
T="${R}R${X}${G}G${X}${B}B${X}${R}${L1}${X}${G}${L2}${X}${B}${L3}${X}${R}${L4}${X}${G}${L5}${X}${B}${L6}${X}"

for i in `seq 1 500000`; do
   echo "$i $T$t$T$t$T$t$T"
done

#//:~

To measure raw terminal sink speed without the overhead of echo, the output wants to be cached as shown below. We can ignore 500mb SSD read time because the bottleneck is definitely elsewhere:

$ ./nuketty.sh > payload.txt
$ time cat payload.txt

These timings are not useful for any precise calculations — it is easy to see that under high bitrate Terminal.app drops to a lower FPS and begins to skip large chunks of input stream. As there seem to be some hardcoded limits to that (I'm guessing 5000 lines/2 FPS), time-per-frame eventually begins to impact stream processing speed; in turn, characteristics of the font affect time-per-frame, so, all other things being equal, total time is a useful measure of the font's relative performance in Terminal.app.

By itself, adaptive frame rate is a perfectly valid way to save a lot of CPU cycles and make non-interactive streams (cat /dev/urandom | hexdump -n 1000000) flush much faster while still giving reasonable visual feedback. iTerm2 is using a similar mechanism by default, only it is optional and fully configurable. Alacritty seems to be less worried about such things, and relies on sheer power of a simple and efficient OpenGL pipeline.

Back at Terminal.app, adaptive frame rate heroics no longer help when the user starts to scroll the buffer using a precision HID such as Apple trackpad. In that case, poor time-per-frame can no longer be swept under the rug as it becomes an integral part of sensory feedback loop. Human brain, as slow as it is, easily picks up latencies above 15ms, and is not easily fooled — normally you simply have to render fast enough to provide feedback before 15ms deadline. With that in mind, a few seconds of difference on 500,000 lines of synthetic input might still not seem like a big deal, but when 0.827 comes out I strongly recommend to try this benchmark with Terminal.app and compare your personal scrolling experience using different fonts. Be ready for surprises.

What is really amusing about benchmarking Terminal.app this way (attn @carloscabanero) is that disabling ligatures actually results in measurably worse performance. The only explanation I can think of is that because we have twice as many glyphs to be rendered and UTF-16-encoded NSAttributedStrings are longer, ligature processing simply ends up being cheaper. Sure enough, this is can only be true for contrived inputs that consist mostly of ligatures.

In reality, complex contextual substitution will surely slow things down, although the relative impact on the overall text layout performance is clearly not what is commonly believed. On the other hand, it is perfectly possible (and very easy!) to produce a substitution table that will utterly crush user experience — that's why, as with any software, profiling and regression testing of OpenType features on carefully chosen test data is absolutely essential.

@schriftgestalt Your articles are awesome. Thinking out loud, there is absolutely no shortage of OT syntax manuals in the wild, but it proves to be impossible for a newcomer like me to find a single public resource about the actual design of substitution tables. Given that some of these state machines are designed by people who have little clue what an FSM is, it really seems the world could use a good read on this — most importantly, the basic pitfalls to be avoided. Although it could be true that the vast majority of all CPU cycles in the Universe are spent on rendering OpenType text, it is not immediately clear how much of those are spent on meaningful work :)

[kelas]

Problem:
=======

Performance of Terminal.app’s rendering pipeline fails to deliver acceptable UX of flushing/scrolling/typing into the buffer filled with the output of the attached shell script.

Steps to reproduce:
==================

1. Ensure your hardware has a Retina display and is running Mojave Beta 6
2. Launch Terminal.app, clone the default profile called “Man Page” and modify the copy as follows:

  * Default font (SF Mono Regular 11pt)
  * No antialiasing
  * Don’t use bold fonts
  * Disallow blinking text
  * Display ANSI colours
  * Use bright colours for bold
  * In “Background” dialogue, set Opacity 100%, Blur 100%
  * Window size 150*55
  * Limit scrollback to 1000 rows
  * Everything else by default

2. Launch a new window using the new profile and run the provided shell script as follows:

$ ./nuketty.sh > payload.txt
$ time cat payload.txt

Observed behaviour:
==================

1. Total sink time (screenshot of the final frame attached):
real    0m44.786s
user    0m0.012s
sys 0m7.182s

2. CPU load around 120-145% during the entire execution (Terminal process sample attached)

3. No dramatic impact on malloc (goes from default 70Mb to 130Mb and back)

2. Subsequent attempts to scroll the trailing 1000 lines of buffer using a trackpad produce a clearly noticeable frame drop and response latency well above 15ms.

Expected behaviour:
==================

1. Total sink time of the same file not above 2.0s
1. Frame rate not below 60 FPS
2. No more than 20ms motor feedback latency

These expectations are met by all major alternatives to Terminal.app. Hope something can be done - rendering a monospaced font in 16 colours doesn’t have to be this slow.

Thanks
k.

File Uploads

* System Information report
* System Diagnostics (sysdiagnose)
* nuketty.sh
* Final frame.png
* Sample of Terminal at 140 pct CPU.txt

[fabrizioschiavi]

With the great help of @kelas PragmataPro is now one of the quickest typefaces in absolute. Please download version 0.827 to see the changes

[cedeber]

@fabrizioschiavi Hi, did you update the fonts on MyFonts? I don't see any updates. Thanks

[cedeber]

It works like a charm. Thanks a lot 🤘