[MEGATHREAD] AtlasEngine bugs and missing features (`experimental.useAtlasEngine`)

[zadjii-msft]

Bugs

• [ ] Desktop stutters on Nvidia GPUs due to G-SYNC switching on/off when two MPO planes are active (MSFT:37283316)
• [ ] Zooming (font changes in general) are much slower than DxEngine 👉 Doesn't occur with a local dev build. What's wrong with our release builds? 🤔
• [x] Atlas acts like IntenseIsBold is always on
• [x] Remaining gridlines (left, right, top, bottom, double underline)
• [x] Missing cursor when experimental text rendering engine is enabled in HDR enabled monitor #12507
• [x] Changing transparency from the command palette doesn't change the transparency until committed / pressing enter

Missing features

• [x] ClearType
• [x] Links not underlined with experimental.useAtlasEngine enabled #11871
• [x] A more conservative GPU memory usage
• [ ] Shrinking the backing texture if possible
• [x] Double-height and double-width line support
• [x] #13853

New features

• [ ] Raster fonts
• [x] Soft fonts
• [x] Investigate alternative ways to handle box drawing/block elements #5897
  • Closes: experimental.useAtlasEngine breaks draw-box rendering #12512
  • Possibly also #6161
• [x] Arrow (➜) character in prompt is rendered small #6864

Cleanup

• [x] Deduplicate font fallback selection during UpdateFont(). Both DxEngine and AtlasEngine fall back to Consolas, etc. separately.

Things we think were fixed by this renderer...

...Or at least, this renderer got out of the way so that these can be fixed now.

• [x] #800 Ligatures do not render under cursor column in Vim
• [ ] #3075 Processing large data is extremely slow (in VMware)
• [x] #3515 Extremely sluggish behavior when in full screen on a 4K monitor
• [ ] #4974 [v0.10] When two characters are ligatured, the first one becomes invisible
• [x] #6738 Some of the extended Unicode characters are rendered with incorrect size and placement
• [x] #6974 Extremely lower performance when displaying too many box-shading glyphs
• [x] #7147 Support renderer fast presents
• [x] #7927 Frames and special characters from bitmap TTF font are not visible
• [ ] #10462 Extremely slow performance when processing virtual terminal sequences
• [x] #13527 Unicode Box drawings characters not rendered correctly (disappear) with Japanese fonts
• [ ] MSFT-38159826

Not really a bug

• [x] #12349

[DHowett]

• [ ] Atlas acts like IntenseIsBold is always on

[o-sdn-o]

Alpha blending is broken in grayscale AA mode (dark moire around glyphs with high RGB-contrast between foreground and background). Apparently the transition between linear/sRGB is missing somewhere.

My setup: Current main branch. Windows 10.

pwsh:

"🥵🦚`e[38;2;255;0;0;48;2;0;0;255mtest`e[38;2;0;255;255mtest`e[m"

Grayscale AA mode:

Cleartype AA mode:

Expected grayscale AA mode blending:

[lhecker]

Your issue is definitely weird. The blue text background in your two screenshots is #120efd and not #0000ff. Why's that? The red "test" text is also #ff0302 and not a clean #ff0000. Are you sure your screenshot wasn't chroma subsampled (I mean like YUV 4:2:2), or something similar?

In any case, I've tested this with Visual TrueType just now. First I've reset my ClearType settings so that I get a clean 1.8 gamma:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Control Panel\Desktop]
"FontSmoothing"="2"
"FontSmoothingGamma"=dword:00000000
"FontSmoothingOrientation"=dword:00000001
"FontSmoothingType"=dword:00000002

[-HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics]

[-HKEY_LOCAL_MACHINE\Software\Microsoft\Avalon.Graphics]

Then I configured VTT like so:

• Display > Size to 18pt
• Display > Resolution to 144 dpi (this matches your screenshots)
• Display > General Options: Here I made sure only "Follow gasp settings" and "ClearType gray" are checked at the end
• Tools > Options: Set "Gamma correction" to 1800

This got me this:

which is almost identical to what BackendD3D renders for me (if you zoom into the "t"):

[o-sdn-o]

The blue text background in your two screenshots is #120efd and not #0000ff. Why's that? The red "test" text is also #ff0302 and not a clean #ff0000. Are you sure your screenshot wasn't chroma subsampled (I mean like YUV 4:2:2), or something similar?

The screenshots were taken via RDP, perhaps there is some kind of post-processing.

Here is a screenshot from localhost:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics]

[HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics\DISPLAY1]
"PixelStructure"=dword:00000002
"GammaLevel"=dword:00000898
"ClearTypeLevel"=dword:00000064
"EnhancedContrastLevel"=dword:00000032
"GrayscaleEnhancedContrastLevel"=dword:00000064
"TextContrastLevel"=dword:00000000

[HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics\DISPLAY3]
"PixelStructure"=dword:00000002
"GammaLevel"=dword:00000898
"ClearTypeLevel"=dword:00000064
"EnhancedContrastLevel"=dword:00000032
"GrayscaleEnhancedContrastLevel"=dword:00000064
"TextContrastLevel"=dword:00000000

My gamma is 2.2 (GammaLevel=2200), since sRGB-gamma is 2.2 on my local monitors with sRGB mode enabled (PHL234E5, ATI HD 3870).

This got me this...

If you convert your image to shades of gray, you will see overdarkening that, in theory, should not be there:

VTT does not take into account gamma correction with grayscale AA blending (or do it wrong):

Off: VTT has a memory leak: +100 MB of RAM on every (back and forth) window resizing step with "Segoe UI Emoji" open.

Off2: VTT does not allow the gamma correction value to be set to 2250, even though the error message states that the valid range is 1050-2250.

Here is my test image (expected grayscale blending):

Off3: Github breaks colors on pasted screenshots. Pay attention to the red "test" - in both screenshots it is the same image. At low resolution, compression changes color.

• 
• Original:

[o-sdn-o]

https://en.wikipedia.org/wiki/SRGB https://en.wikipedia.org/wiki/Alpha_compositing

[o-sdn-o]

I found that the deprecated DxRenderer (if I understand it correctly) GDI-renderer (with UseDx=0) (thanks to @tusharsnx) is not affected by this issue:

Make sure system-wide cleartype is disabled:

[tusharsnx]

@o-sdn-o Nah, that's the GDI renderer (unless you have used useDX from the registry).

I do see similar result in conhost as you

Conhost with system-wide Cleartype off:  Windows Terminal with Grayscale AA:  Windows Terminal with ClearType AA: