Better support for `{xterm,tmux}-direct` terminals

[quark-zju]

termwiz currently defines PaletteIndex as u8, with the "256 color" intention. But the *-direct variant terminals use the index as a 3-byte true color index. The one-byte index affects only the blue byte. This means PaletteIndex users will be surprised using the PaletteIndex to render index within 8..256 using those terminals.

The main problem is that the setaf is defined as:

TERM='tmux-direct' infocmp | grep setaf
    setaf=\E[%?%p1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m,

which only special treats the first 8 colors and treats everything else as a "direct" RGB color index.

The ncurses terminfo comments mentioned "color index" but it does not seem universally supported. There is an RGB capability that tput RGB works but infocmp does not show it. The "max colors" is now 16M.

Note there is force_terminfo_render_to_use_ansi_sgr but it might have unwanted side effects. @chadaustin reported ^O around some other color renderings. So it might not be a practical solution.

Describe the solution you'd like

Given that terminfo setaf uses low-level "color index" (which is ambiguous - rgb(0,0,0..8) conflicts with basic colors) while termwiz already provides high-level types for different kinds of colors. I think the following might be a reasonable solution.

• When MaxColors is 16M (which should set ColorLevel to TrueColor but that's a separate issue), do not use setaf to render 256 or 16 colors.
• Maybe rename TrueColorWithPaletteFallback to TrueColorWith256Fallback, since the Palette might depend on context while 256 is less ambiguous.
• Maybe the force_terminfo_render_to_use_ansi_sgr can be a bitflag to control when to fallback more precisely. Its initialization can be "smarter" by running testing setaf with a memory buffer to figure out how the first 8, 8-16, 16-256 colors are rendered, and choose sgr fallback smartly.

[chadaustin]

I have not done a full audit of the other -direct terminfo files, but it looks like there are several these days:

$ toe -a | grep -- -direct
xterm-direct256 xterm with direct-colors and 256 indexed colors
xterm-direct2   xterm with direct-color indexing (old)
xterm-direct16  xterm with direct-colors and 16 indexed colors
xterm-direct    xterm with direct-color indexing
vscode-direct   Visual Studio Code with direct-colors
vte-direct      VTE with direct-color indexing
st-direct       simpleterm with direct-color indexing
mlterm-direct   mlterm with direct-color indexing
mintty-direct   Cygwin Terminal direct-color
nsterm-direct   nsterm with direct-color indexing
iterm2-direct   iTerm2 with direct-color indexing
konsole-direct  konsole with direct-color indexing
kitty-direct    KovId's TTY using direct colors
alacritty-direct        alacritty with direct color indexing
tmux-direct     tmux with direct-color indexing

[wez]

• Maybe rename TrueColorWithPaletteFallback to TrueColorWith256Fallback, since the Palette might depend on context while 256 is less ambiguous.

It's still early here and I'm not fully awake yet, so can you help me understand when this might be an issue? It seems like if we are using a -direct terminal we'd have ambiguous resolution of PaletteIndex, but for TrueColorWithPaletteFallback and a -direct terminal, we shouldn't need to use the fallback?

It seems like ColorAttribute::PaletteIndex might be the problematic variant in that situation?

• Maybe the force_terminfo_render_to_use_ansi_sgr can be a bitflag to control when to fallback more precisely. Its initialization can be "smarter" by running testing setaf with a memory buffer to figure out how the first 8, 8-16, 16-256 colors are rendered, and choose sgr fallback smartly.

Interesting idea! I think this is technically an imperfect solution because, in theory, setaf can produce arbitrary output that we may not understand, but in practice it will almost always be one of a handful of known outputs that we can parse and reason about.

Having a Capabilities flag (or bitfield) to indicate whether each of 8/16/256/direct color is renderable via setaf sounds reasonable to me, although I think this should probably be a tristate to represent Unknown (we didn't understand the setaf output), Yes and No (eg: broken output as you described in the OP).

[quark-zju]

It's still early here and I'm not fully awake yet, so can you help me understand when this might be an issue? It seems like if we are using a -direct terminal we'd have ambiguous resolution of PaletteIndex, but for TrueColorWithPaletteFallback and a -direct terminal, we shouldn't need to use the fallback?

It seems like ColorAttribute::PaletteIndex might be the problematic variant in that situation?

I think the problem is solved by the PRs. This is now more about documentation. Developers who are aware of *-direct terminals might expect PaletteIndex to be the "raw" (3-byte) color index. I think the document can be updated to mention that termwiz's PaletteIndex (in both structs) is always the 8/16/256 color index even in *-direct cases.

• Maybe the force_terminfo_render_to_use_ansi_sgr can be a bitflag to control when to fallback more precisely. Its initialization can be "smarter" by running testing setaf with a memory buffer to figure out how the first 8, 8-16, 16-256 colors are rendered, and choose sgr fallback smartly.

Interesting idea! I think this is technically an imperfect solution because, in theory, setaf can produce arbitrary output that we may not understand, but in practice it will almost always be one of a handful of known outputs that we can parse and reason about.

Right. I think even information like the length of the output can provide insights. For example, if max_colors == 16M and setaf(index=255) produces a same length as setaf(index=(255 << 8)) then we can conclude that setaf does not support 256 color index. The "smartness" might also be implemented by parsing the raw setaf code and interpreting it with special care (ex. when we see the "if index < 8" condition we know the "then" is for 8 colors and "else" is for 256 colors). terminfo might provide more accurate information about whether : or ; should be used (related: #2723).

True color rendering today never uses setaf. If there is a case that setaf produces a different true color rendering it seems nice to have this "smartness", since setaf does not render rgb(0,0,0..8).

Having a Capabilities flag (or bitfield) to indicate whether each of 8/16/256/direct color is renderable via setaf sounds reasonable to me, although I think this should probably be a tristate to represent Unknown (we didn't understand the setaf output), Yes and No (eg: broken output as you described in the OP).

Got it. It seems it could be a few Option<bool>s on the "hint" struct, and then convert to non-option values during initialization.

[quark-zju]

Thinking about it, a more flexible solution might be splitting "setaf" into multiple versions, with the ability to manually override (Option<code> instead of Option<bool>):

• 0-8 color setaf
• 8-16 color setaf
• 16-256 color setaf
• true color setaf

For example, when we got setaf like:

setaf=\E[%?%p1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m,

We can technically detect the "if index < 8" condition, and extract the "then", "else" clauses (done by ChatGPT, I haven't checked manually):

setaf_8_color=\E[3%p1%d
setaf_true_color=\E[38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d

Then "infer" the missing setaf_16, setaf_256 based on the above information. The SGR fallback today can be changed to only affect the "infer" process.