Poor performance when terminal+canvas renderer is on a very wide container

[goenning]

Details

• Browser and browser version: Safari
• OS version: 12.5.1
• xterm.js version: 5.0.0

Steps to reproduce

1. Render the terminal (using canvas addon) into a container that has 20k+ pixels
2. use the fitAddon.fit
3. Start writing some lines, doesn't even need to be that long, just a 100 chars on each.
4. The whole windows begins to perform poorly to the point I need to force kill it

Note: This is somehow related to https://github.com/xtermjs/xterm.js/issues/2544

While it's understandable how horizontal scrolling is outside scope of xterm, is there anything that can be done to make it usable on a very wide container?

This is especially useful for viewing logs.

Thanks

[goenning]

Without using the fitAddon and just resizing it manually, it seems to start degrading after I set to 2500 cols.

[jerch]

@goenning

2500 cols? :scream:

Not sure where to start here - maybe lets make it blunt: this sort of perf degradation is kinda to be expected with canvas renderer for very huge displays.

Reasons: The canvas renderer does 2D context drawing, which is CPU bound and synchronous. Furthermore the renderer uses a bitmap-like char placing from prerendered atlas entries and does that one by one. Now doing the some maths - for only 4k displays in fullscreen mode this already means ~32MB of data transfer from CPU to GPU for every frame, at 60 FPS we are at ~2GB/s to be pumped. Sounds like cake? It certainly is for modern machines, but remember - it gets not transferred in big chunks, but in tiny char size portions. Since your display is much bigger than 4k - outch. At this point I think any software not doing hardware accelerated drawing will struggle here. Maybe try the webgl renderer? Secondly JS has a processing penalty of being 2-4x slower than a compiled language (the numbers are just rough values pulled from my perf measures), so you prolly will see the perf impact much earlier than with a C/C++ terminal also doing the CPU bound rendering. Lastly xterm.js has to do parsing and rendering in one thread currently, which furthermore steals processing time from constant render/parse context switches.

I am really interested in some perf diagnostics on such a big display. Care to run the demo in chrome and make some screenshots from the devtools->profiler tab? (Similar to these: https://github.com/xtermjs/xterm.js/issues/4112) Also the difference canvas to webgl would be interesting.

@Tyriar I wonder if there is a BG drawing bottleneck somewhere? This sounds suspicious:

Start writing some lines, doesn't even need to be that long, just a 100 chars on each.

I also wonder if we'd get some speedup by doing more double buffering (e.g. on line level before placing it on target canvas).

[goenning]

Hi @jerch thanks for the fantastic explanation. Here's the minimum repro I used: https://github.com/goenning/xterm-cols-repro/blob/main/main.js

xterm-addon-canvas

With 1000 cols, pretty much no activity

Same with 1500 cols

But with 2000, we seem to be spending a lot of time on the for each cell loop

And this is noticeable on the UI. When I change from 1500 to 2000 it becomes unusable. I'm on a Macbook M1 Pro btw

Note 1: There's this on the console Canvas2D: Multiple readback operations using getImageData are faster with the willReadFrequently attribute set to true. See: https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently

---

xterm-addon-webgl

I also tried webgl, performance looks great even with 3000 cols, but the text gets blurry/stretched. Everything is fine with rendering up until ~2000 cols, but starts having issues with rendering at that stage, it becomes worse as the col size increases.

Regarding the use case

It's a logging viewer, so it's impossible to know how wide the log lines will be. I understand 2500 is a lot and I guess most users are unlikely to scroll that much. Is there a way to force xterm to crop at a certain col and avoid breaking into another line?

[goenning]

I did another test with canvas and 2000 cols, but this time writing every 10 seconds instead of every second.

Seems like every write operation takes 7000 seconds of scripting to complete, mostly on the drawForeground function.

And then another test with canvas and 1500 cols, again writing every 10 seconds.

Only 42ms spent on scripting 😳

I'm at the stage that I think I'm doing something wrong, but I'm really just changing the number of cols, nothing else.

[jerch]

Hmm thats interesting, xterm.js does not do anything different for different col sizes, maybe thats some sort of cache thrashing in the browser/JS engine? Do other browser engines (webkit/Firefox) show a similar behavior? (Its alot harder to get nice profiling data on those, maybe you are still able to spot it from "interactiveness"?)

On a sidenote: From tests in the image addon I remember that drawImage on canvas shows a weird perf degradation at a certain size in chrome (dont remember - it was either around 2^13 or 2^14 pixel width...)

Edit:

Is there a way to force xterm to crop at a certain col and avoid breaking into another line?

Not really, as the print action is not hookable. There are tricks possible like disabling DECAWM, which would avoid auto-wrap writing all excess chars into the last cell of a row. Not sure if thats good enough for you. Another idea would be to preprocess the data with a headless xterm.js instance, and to cut off the excess chars. But thats quite involved to get done.

[Tyriar]

We haven't really optimized the renderers for that large, in my brief experience looking at this I think we run into problems with maximum texture when it is (eg. the stretching webgl texture). It may also be slow simply because we're iterating over all columns, not just the line length here:

https://github.com/xtermjs/xterm.js/blob/master/addons/xterm-addon-canvas/src/TextRenderLayer.ts#L83

Remember though that the canvas renderer should be a fallback for the webgl renderer, which is superior in pretty much every way. I think it's most useful to instead fix the webgl renderer's stretching problem here. Something I looked into recently was bringing the webgl renderer to the monaco text editor (https://github.com/microsoft/vscode/issues/162445) which requires supporting both partially scrolled lines as well as very long non-wrapping lines.

Instead of sizing the canvas to the number of columns, the best thing to do here is to size it to the viewport on screen and only render what's actually on screen. We'd need to change how xterm.js works wrt element size/horizontal scrolling, but we could do very long rows by adding an x offset uniform and then only draw from there. As is done in my prototype to accomplish partial lines with the y offset, we'd add an xoffset uniform to allow partial line rendering (clipped cells on the left/right), and then only draw viewport cols + 1 x rows.

[goenning]

@jerch i haven’t checked Firefox yet, but I’ve seen the same behaviour on Safari. It’s usable at 1500, and becomes unresponsive at 2000 cols

@Tyriar performance on webgl is fantastic, that’s definitely the one ill be starting with. Is the stretching a known bug? Or should I open a separate one for it.

[Tyriar]

@goenning I don't think it's tracked outside of my head yet 😉, yes please create an issue. That should also make CPU usage much lower for webgl by only rendering what's on screen