webgl regression: may not render as typing

[Tyriar]

VS Code issue: https://github.com/microsoft/vscode/issues/190195

I think this started happened in the last few weeks. Will need to review recent changes/optimizations to webgl

[tisilent]

Yesterday, I also discovered this issue in the internal version. One of the two devices had this issue, so I didn't pay much attention The display is only displayed after entering a few characters.

[tisilent]

On Monday, I'll see if the device can still reproduce. 😆

[tisilent]

If there is a time interval, this situation will occur, and if you directly term.write, there will be no such exception. 😵‍💫

[Tyriar]

@tisilent what do you mean by time interval?

[tisilent]

hi @Tyriar. Not much progress. The clues discovered now.

1. I have generated vscode for version 1.81.1 and master, and 1.81.1 is normal.
2. This situation can also occur when using earlier versions.

https://github.com/xtermjs/xterm.js/assets/48614781/0ca1f91e-14be-4c0b-9061-b5e798d81a1c

3. This is a virtualization device.

[jerch]

This situation can also occur when using earlier versions.

Does it go away if you test with an older release, like 5.2.0? From there on it would be good to pin it down to a certain beta build to get a hold of the commits in question. And if much older releases like 5.1.0 still show the glitch, then it might be a good idea to repeat the testing with an older chrome version and/or other webgl capable engine like firefox to rule out engine version issues.

[tisilent]

@jerch I tried two combinations,

1. xterm 5.2.1 , webgl 0.15.0
2. xterm 5.3.0-beta.7 , webgl 0.16.0-beta.2 (vscode currently in use) go away............. vscode 1.81.1 is normal,I can try using chromium 108.

[Tyriar]

I can't reproduce this unfortunately

[tisilent]

https://github.com/xtermjs/xterm.js/assets/48614781/3218fdf8-8ec3-4c33-afe4-10fc1e57188c left: electron 22.3.18, chromium 108 right: electron 26.1.0, chromium 116

[tisilent]

I don't know how to track it at the moment. 😭

[jerch]

Well I havent seen the issue once here, but I am still on chromium 112 with my tests (linux here). Which makes me wonder, if the recent WebGPU additions in chromium might have to do with this (maybe the drawing/update cycling got affected by those, temporary context loss, things like that...)

[Tyriar]

@deepak1556 we're starting to think this webgl issue might be related to the Electron update

[Tyriar]

@tisilent since you can reproduce, could you try it on https://insiders.vscode.dev/github/xtermjs/xterm.js in a recent Chrome version? This will help tell if it's a Chromium problem or an Electron problem.

[tisilent]

Okay, I'll try it later.

[jerch]

Cant repro with chromium 115 either (newest build available here). Could this be a windows only / gc-driver issue?

Below parts of chrome://gpu (currently using intel integrated graphics on ~Haswell~ Skylake). Note that it has deactivated WebGPU here due to some "blocklist".

[jerch]

Also tested on Chromium 116 (cheated a bit by using custom chrome binary from newest playwright) - cant repro. Also same output from chrome://gpu as above.

If there is a specific pattern to trigger it, I can test it again. What I tested was xterm.js demo (master) and xtermjs.org with switching browser tabs and stealing focus in between. All fine here.

[jerch]

Also note that in your demo repro video in https://github.com/xtermjs/xterm.js/issues/4665#issuecomment-1691387657 the cursor on the right side still moves & updates on every input, it is just the char glyphs that come up lazy at once. Hmm.

[tisilent]

I saw that fps has fluctuations and is processed, but the characters are not displayed. At first, I checked the code and found that the model output was consistent. This device is Virtualization. emmm.

[tisilent]

Verified, older versions of Chromium will not encounter this issue.

[jerch]

@tisilent Can you also check, if chrome://gpu --> "Graphics Feature Status" shows significant differences between your tested versions, e.g. WebGPU enabled here but disabled there?

My best guess so far - chromium devs did some fundamental rework on the graphics internals with recent versions, mainly to get WebGPU finally shipped. This may include shifts in handling certain driver/gc constellations (moved from HW to SW support, or vice versa, I think they update their blocklists regularily).

Also saw only windows reports so far, not sure if its platform related as well.

In summary - from xterm.js side there is nothing we can do about that (beside tracking and writing a bug report, if we have enough evidence).

[jerch]

This hw acc support issue in browsers is really annoying, it pops up every now and then for us. I think that xterm.js is just a surface vector here, where ppl experience it pretty early / more directly, but in fact is a deeper problem with the engine's hardware support.

At this point I wonder how other popular canvas/webgl driven libraries handle that issue (like d3, three etc), if there are any best practices to collect issues and report upstream.

@tisilent I feel your pain, such a regression is hard to swallow when coming from a flawless working system. If you are really up to find the culprit code change in chromium, I can only suggest to try out the nightly chromium builds one by one until you find the first defective build. From there you can dig deeper into the commits...

[deepak1556]

If you are really up to find the culprit code change in chromium, I can only suggest to try out the nightly chromium builds one by one until you find the first defective build. From there you can dig deeper into the commits...

If there is a reliable repro in Chromium browser, then I recommend to use the following bisect tool https://www.chromium.org/developers/bisect-builds-py/ to narrow down the commit range. It will be very useful when filing upstream bug report.

[tisilent]

@jerch There are too many features, some are different..other..

The Graphics Feature Status is the same.

[Tyriar]

We looked into this and here's the tracking Chromium bug https://bugs.chromium.org/p/chromium/issues/detail?id=1476475. See https://github.com/microsoft/vscode/pull/191795 for how to workaround this in Electron in the meantime (still needs to be verified once merged).

[jerch]

Slightly offtopic (still prolly related to chromium's recent work on the graphics internals) - my sixel-bench test yields these fps numbers (on the same machine/system):

• chromium 108 - 3.38s (35.09 MB/s, 202.69 fps)
• chromium 116 - 8.97s (13.22 MB/s, 76.34 fps) (WTH?)
• epiphany 16.4 Safari/605.1.15 - 3.64s (32.56 MB/s, 188.07 fps)
• firefox 117 - 5.83s (20.36 MB/s, 117.58 fps)

Btw chromium started at ~42 MB/s 2ys ago, but the recent drop to 13 MB/s is really hurting. Firefox was always that slow, Webkit is stable at its fast pace.

[jerch]

Yes something fishy is going on:

• chrome --use_angle - 3.53s (33.64 MB/s, 194.29 fps) (almost back at old values, maybe old default)
• chrome --use_angle=egl - 4.68s (25.35 MB/s, 146.41 fps)
• chrome --use_angle=gl - 8.67s (13.68 MB/s, 79.0 fps) (prolly the new default?)

Idk the right values for angle, I basically just guessed around, and found an empty value to be the fastest. :thinking:

Edit: And runs really slow with chrome --use-gl=angle --use-angle=swiftshader - now at 25.89s (4.58 MB/s, 26.46 fps). I guess thats mostly software emulation now? Are these settings explained anywhere?

[deepak1556]

Are these settings explained anywhere?

I haven't found a doc for these settings, it was mostly from reading the code https://source.chromium.org/chromium/chromium/src/+/main:ui/gl/init/gl_display_initializer.cc;l=26-170

[jerch]

Was finally able to restore the old throughput numbers for the sixel test by disabling vulkan in flags and using the empty --use-angle switch. This penalized a different code path now (png-blob --> bitmap creation), but not by a bothersome amount (only 20% slower).

Its really astonishing how fragile chrome's graphics settings are, while firefox and webkit didn't change fps numbers at all across different updates. Also I dont have to mess with undocumented switches there. (Note that webkit just added alot of stuff in the graphics section like offscreen canvas and bitmap support...)

[Tyriar]

Since this is a browser issue I'm closing this off

[deepak1556]

Issue has been addressed in the vulkan backend, refs https://bugs.chromium.org/p/chromium/issues/detail?id=1476475#c14