Add support for parsing Underlined and Doubly Underlined to the parser

Discussed with @DHowett-MSFT, as a follow up of #2554. See also #2915.

In #2554 I'm not adding support for parsing these additional extended states, because their implementation seems closely tied to more complicated existing implementations.

~Faint cannot be turned on at the same time as Bold~
Underlined is a trinary state
We're currently using LVB_UNDERSCORE to store "underlined"

Due to the timing with the upcoming 20H1 release, I felt it too risky to try and implement those as well. I've left room in the ExtendedAttributes enum for these states, but currently, Doubly-Underlined and Faint are totally ignored.

We'll need to parse and store them in the buffer correctly. It's another task entirely to support rendering these things correctly. I believe the rendering for underline state will be satisfied with #2915. ~Faint might just be okay too, considering it's like the opposite of Bold.~

Since you did move bold into ExtendedAttributes (but not Underline), and you did update the IsBold getter to query the ext. attrs., is Faint easier than Underscore here? What's the danger in doing that for 20H1 but not doing Underline?

Re underline:

The Kitty terminal emulator came up with the awesome idea of supporting curly and colored underlines, with the obvious intent of supporting user-friendly spell checking in terminal-based text editors. The choice of the escape sequence was coordinated between Kitty and VTE. The feature was then implemented so far at least in Kitty, VTE, Mintty, Hterm, probably a few more as well, and feature requests are filed to even more, including iTerm2, Konsole, xterm.js. Some have even added dotted and dashed underlines, too.

It would be lovely if you also considered these extensions.

(A bit of technical info: With truecolor support I assume you already have like 25 bits for the foreground and background color each (in order to be able to store the 256 legacy palette values as well as "default", in addition to 24 bit RGB). At least this is how we do in VTE. And we didn't want to waste another 25 bits for this rarely used feature. So we approximate truecolor underline colors to 4+5+4 bits of R, G, B, respectively. This way all the color information of a charcell fits in an int64.)

Re faint:

(I should really get a Windows and try out your terminal :D It's definitely on my todo list. Sorry in advance if I make assumptions about Windows Terminal that aren't correct.)

"Bold" has a lot of legacy confusion whether it's actually "bold", "bright" or both. With truecolor support in many terminals nowadays, and still no unambiguous way of encoding bold, Kitty and VTE decided to push for cleaning up this legacy mess, making "bold" (SGR 1) only stand for bold and not tamper with the colors. This is Kitty's only supported operation, and we also made this mode the default recently in VTE / GNOME Terminal. xterm and urxvt also support this behavior. (Let's also note that the popular Solarized color scheme requires this mode.)

"Faint" is even worse. While most terminals allow the user to configure the 16 basic colors (the 8 most basic ones, plus their ~bold~ ahem bright counterparts), they don't offer a way to configure their faint versions, it's usually computed using a hardwired formula. And then should all these 16 or 256 palette colors undergo such conversion? Should even truecolors? Or only the first 8? There's direct access to the 16 basic colors (normal and bright ones – the latter ones through the aixterm extension SGR 90..97, 100..107, without enabling bold typeface), but no direct access to the faint ones. Also, apparently it isn't clear whether faint is supposed to be mutually exclusive to bold or not. "Faint might just be okay too, considering it's like the opposite of Bold." – you said; but the opposite of which interpretation of bold?

With our separation of bold vs. colors, the faint property doesn't fit in this picture. Kitty doesn't implement it for this reason, and in VTE we also thought about deprecating/removing it. In fact, the behavior that would fit in this new model is to use an even thinner font than the default, without tampering with the color. This is limited by only few fonts supporting thin typeface, whereas for ones that do (like variable fonts) this kind of asks for a generic extension towards supporting even more weights. What I have in my mind is something along the lines of SGR 1:1 .. SGR 1:9 corresponding to CSS Fonts Module Level 3's fonts weights 100 .. 900, SGR 1 being a compatibility alias to 1:7 (bold), SGR 0 to 1:4 (normal weight), and SGR 2 (aka. faint) to maybe 1:2. This is yet to be discussed and agreed upon across some terminals.

Please read our more detailed Thoughts about faint (SGR 2) for juicy details how it's related to the cathode ray and how it all doesn't make any sense with dark-on-bright color schemes.

What I recommend you to do is:

If you haven't already, please implement a mode where SGR 1 does bold typeface only, but not brighter colors. Solarized users need it. Users who want to leave behind the legacy confusion need it.

I also encourage you to make this mode the default, like we did in VTE, but of course it's something that you need to evaluate yourselves if you're ready for this, or if you'd prefer to stay a more backwards-compatible terminal by default.

Treat "faint" with low priority, I honestly hardly ever saw any project using this attribute, and consider going with a thinner font if feasible. But don't feel bad if you don't support this property at all :-).

Kitty and VTE decided to push for cleaning up this legacy mess, making "bold" (SGR 1) only stand for bold and not tamper with the colors.

Personally I think this is a mistake, and I really hope Windows doesn't follow this approach. Interpreting SGR 1 as an increased intensity or brightness has been the expected behavior in terminals and terminal emulators going back decades now. Dropping that support is just going to cause things to break (the Kitty issue you linked to being a case in point).

And if we eventually do support the concept of bold as a font weight, I hope we have the option to turn that off, because again I think that will be an undesired behavior in legacy apps that aren't expecting it.

Whoops, I did not notice that WT does not support bold typeface yet.

If WT intends to be a modern terminal, with a feature set comparable to let's say GNOME Terminal, Tilix, Konsole, iTerm2 etc. (as far as I understand it does intend) then adding support for bold (and while at it, italic) typeface is a must. (Without that, my suggestion would be to make SGR 1 a no-op, this is clearly not what I'm suggesting.)

An option for SGR 1 doing bold only, and not altering the color, is an absolute necessity for the users of certain color schemes, including the popular Solarized. It expects the 16 colors to be absolutely independent from each other, never as a side effect of another operation switching from one to another. Also, its "bright" colors are not at all bright counterparts of the "normal" ones.

This option is also desireable for those who want a saner environment, e.g. want to access a bold but not bright variant of the 8 basic colors.

Interpreting SGR 1 as an increased intensity or brightness has been the expected behavior in terminals and terminal emulators going back decades now.

This sounds pretty subjective to me.

Most graphical terminal emulators do bold+bright, so I might as well argue that SGR 1 doing bold is expected, yet WT doesn't do it.

The popular screen drawing library ncurses calls the relevant attribute A_BOLD, no mention of brightness in its name. That is, anyone coding against ncurses probably expects bold.

Expectation is one thing, common behavior is another, the official standard is yet another. I know it's generally hard to decide what to do if they don't quite match.

I'd argue that "bright" or "bold+bright" behavior is a leftover from the days with limited hardware capabilities, including limited color palette. Even with the introduction of the aixterm 16-color palette (the addition of SGR 90..97/100..107) it no longer makes sense, let alone if you extend to 256 or 16M colors. Also, if it's meant to do "bright", why does it only do so for the basic 8 colors, why not for the rest of the 16- or 256-color palette entries, or even truecolor ones? (Nitpicking: and why bright foreground only and not bright background too? There's not a word about it in the specs.)

Dropping that support is just going to cause things to break (the Kitty issue you linked to being a case in point).

Yes, introducing this behavior breaks things. For example, as mentioned in one of the VTE bugreports, the "make menuconfig" step of compiling the Linux kernel doesn't draw borders now. Taking a look at its source, it makes not one but two faulty assumptions that happened to work due to testing with only one particular interpretation of the specification. (If a spec says "this or that", you shouldn't write code against it expecting that it'll always be "this", correct?)

We've changed VTE's behavior more than half a year ago, and roughly 50% of terminals emulator users on Linux use some VTE-based one, meaning that this change has already reached a pretty significant market share. Yet this is the most serious bug we've heard about. The generic perception of the change was pretty good, follow my comments at Alacritty 2776 and 2779 for details.

And most importantly: any such new bug like the Linux kernel "make menuconfig" borders is fixable from the application's side.

Contrary, the legacy behavior also has bugs, but those bugs are not fixable from the application's side. Solarized color scheme becomes pretty much unusable (or at least you need to completely avoid bold typeface). There's no access to bold+dark. With dark-on-bright default colors the two concepts of bold and bright that are enabled by the single SGR 1 actually work against each other, defeating any sane purpose SGR 1 might have. No one knows which of the two colors to make brighter in combination with the "reverse" attribute. And so on...

And if we eventually do support the concept of bold as a font weight, I hope we have the option to turn that off, because again I think that will be an undesired behavior in legacy apps that aren't expecting it.

Compatibility is usually pretty important. In some cases, when it stands in the way of further evolution of the ecosystem, I personally believe that generally modernization and bugfixing has to triumph over compatibility. With common sense and proper care being taken, of course. Here we're "just" talking about visual representation of some data, somewhat changing it is unlikely to severely break things, it's rather just causing a less pleasant look in some cases.

As for standard Linux terminal emulation, the expectation is that SGR 1 switches to (at least) bold typeface. This is the behavior of most terminal emulators, and as I said, the library that most apps use also calls it "bold". You say "legacy apps that aren't expecting it", but I seriously doubt there's any such application anywhere in the Linux world.

As for Windows applications, if they request let's say a bright cyan via API, you should just translate that to SGR 95 rather than SGR 1;35 and you made it independent of this entire story: it'll always be bright but not bold.

Supporting bold (and italic by the way) typeface, as well as adding an option to make SGR 1 bold only and not bright, is a feature absolutely expected from any decent modern terminal emulator.

Making the "SGR 1 is bold but not bright" the default is at least as much of a political decision as a technical one, so I can only weakly encourage WT developers to make this bold (pun intended) step. The experiences from VTE's switch are pretty good, there are only a few minor visual annoyances, nothing serious. I believe there are no Windows compatibility issues to worry about, you can just map bright colors to bright colors, entirely avoiding the ambiguous SGR 1. In this new world the issues are fixable, as opposed to the old world where they were not. And for anyone not liking it or running into real issues, there'd still be this option to revert to the legacy behavior.

I understand all the arguments for wanting a way to get a bold typeface without bright colors, but that could just as easily have been achieved by creating a new SGR code specifically for that bold style, with less chance of breaking backwards compatibility.

Interpreting SGR 1 as an increased intensity or brightness has been the expected behavior in terminals and terminal emulators going back decades now.

This sounds pretty subjective to me.

Individual expectations may be subjective, but we can make a reasonable prediction of the expectations of most users on aggregate. If the majority of terminals in use today support SGR 1 as an increased intensity (regardless of whether they also support a bold typeface), and that has been the case since at least the early 80s, then I think it's safe to say that that is what most people would expect.

Or are you saying the majority of terminal emulators don't support that any more? I'll admit I'm not particularly knowledgable about the current state of terminals other than XTerm.

The popular screen drawing library ncurses calls the relevant attribute A_BOLD, no mention of brightness in its name. That is, anyone coding against ncurses probably expects bold.

I think you're interpreting the word "bold" to mean what you want it to mean. When referring to a font, it may mean a thicker typeface, but when referring to a color it just means a more vivid appearance, and bright is considered a synonym. The VT100 technical manual explicitly defines bold as "increases the intensity of the display".

Here we're "just" talking about visual representation of some data, somewhat changing it is unlikely to severely break things, it's rather just causing a less pleasant look in some cases.

Except when you have something like bright white on a regular white background (which is not that uncommon a choice), the text suddenly becomes invisible.

As for standard Linux terminal emulation, the expectation is that SGR 1 switches to (at least) bold typeface.

Except if you're working in the Linux text mode console where a bold typeface is simply not an option.

And if we eventually do support the concept of bold as a font weight, I hope we have the option to turn that off, because again I think that will be an undesired behavior in legacy apps that aren't expecting it

Here we're "just" talking about visual representation of some data, somewhat changing it is unlikely to severely break things, it's rather just causing a less pleasant look in some cases.

I'm not saying it's going to severely break things; just that some people might prefer to turn that option off (not an uncommon preference, e.g. see here or here).

And it's hard to argue that these aesthetics don't matter while also arguing that it's absolutely essential to have SGR 1 not alter the color for people using the Solarized color scheme.

As for Windows applications, if they request let's say a bright cyan via API, you should just translate that to SGR 95 rather than SGR 1;35

Not all Windows applications would be using the API to set colors though. Legacy apps are quite likely to be using the simple escape sequences that were supported in ANSI.SYS and ANSICON. There may not be a lot of them, and there may not be any desire to support them any more, but let's not pretend that they don't exist.

Anyway, as long as these things are preferences that the user can change, I don't think it matters that much. But I understand you're not even allowing SGR 1 brightness as an option in VTE any more, and I'd strongly object to us doing the same thing in Windows.

but that could just as easily have been achieved by creating a new SGR code specifically for that bold style, with less chance of breaking backwards compatibility.

Going for a brand new SGR code sure could have been another approach. "just as easily" – I don't think so. That approach would have needed buy-in from more key players of the game, some of which I have good reason to assume that wouldn't have been supportive, and I assume would have led to much slower adoption. Anyway, discussing how it could have been done differently in the past won't take us forward.

Or are you saying the majority of terminal emulators don't support that any more?

I'm not saying that. I'm saying that it's no longer the default in a pretty significant fraction (soon to be in the ballpark of 50%, concerning market share within Linux).

I think you're interpreting the word "bold" to mean what you want it to mean.

I have to admit I haven't come across VT100's definition as "bright". I'm solely using "bold" in the sense that modern graphical applications use it: increased font weight.

Except when you have something like bright white on a regular white background (which is not that uncommon a choice), the text suddenly becomes invisible.

During the last half a year, we haven't received any report about this (except for the dialog border I've already mentioned). So it may be a bit less common than you think, at least across Unix apps.

Except if you're working in the Linux text mode console where a bold typeface is simply not an option.

Do people use the Linux text mode console for productive work (I mean, not firefighting or disaster recovery)? I can't really imagine why anyone would do so.

some people might prefer to turn that option off (not an uncommon preference, e.g. see here or here).

That purple screenshot made me cringe. Get a terribly looking font and then complain that it's terrible? Get a nice font (see e.g. https://github.com/microsoft/terminal/issues/3226#issuecomment-542749741) and it's no longer really an issue. :-)

But again it's mostly a question of project vision: where does WT want to go? Is it legacy and backwards compatibility, or being a decent modern future environment that is more important? Does it want to play safe, or be brave? And why would be the terminal emulator the only piece of software where the user can disable bold fonts? (In gnome-terminal we demoted this user-visible setting to a hidden one about two years ago, I can't recall anyone asking where it went, and just removed now in the development series to see if people really care.)

Anyway, I'm absolutely fine if WT adds bold typeface support along with a kill config option. Or maybe a tri-state: SGR 1 is bold, bright, or both. I most definitely do not want WT to remove the legacy compatibility behavior thereof.

But I understand you're not even allowing SGR 1 brightness as an option in VTE any more

This is not true. We do have that option, we just flipped the default.

Thinking further about this...

Legacy apps are quite likely to be using the simple escape sequences that were supported in ANSI.SYS and ANSICON.

I have to admit I completely missed this.

I think we should conclude that for SGR 1 the expectation is:

In the Linux world most users expect it to be bold typeface (a few users don't like that), and there's a huge confusion and ongoing change around brightness (and a heavy demand for both behaviors).
In the Windows world, users expect it to be bright, and not bold.

When sshing from Windows (Powershell or Cmd) to Linux, there is some magic going on. At least the charset is switched to UTF-8 (and back when logged out). I don't know which component does it, but (according to a script log of sshing from Linux to itself) it's not the remote side using the \e[%G/\e[% escape sequences. I suspect it's some local Windows magic. (I mean "magic" to me at this point, hopefully a well designed solution.)

I'm wondering... wherever this magic happens, could it also switch the behavior of SGR 1? Either via some API-like thingy, or via a newly invented "allow/disallow bold" escape sequence? Could this be a reasonable solution to support both worlds?

For what it's worth I've extended this a little, and my programs support the following subparameters:

single
double
(short wavelength) curly
(closely spaced) dotted (8 dots per cell)
(short) dashed (4 eighth-width dashes per cell)
long dashed (2 quarter-width dashes per cell)
extra long dashed (1 half-width dash per cell)
medium spaced dotted (4 dots per cell)
widely spaced dotted (2 dots per cell)
long wavelength curly

FYI, I've been doing a bit of research into how these attributes are implemented in other terminals, and the results are rather interesting.

Faint cannot be turned on at the same time as Bold

This isn't actually true in most cases (at least amongst my terminal collection). When bold and faint are applied at the same time, the font weight typically remains bold (if supported), and the bright palette entry for ANSI colors is still used, but that "bright" color is then algorithmically dimmed.

The effect of the color changes is most obvious in a palette like Solarized, where "bright green" is actually a shade of gray. So when you have faint combined with bright green, you get a fainter shade of grey, rather than the dark green.

Underlined is a trinary state

Yes and no. There aren't many terminals that support double-underline for real (most treat it as a synonym for single underline) so there's no obvious consensus. In VTE it's a trinary state, where the last choice (single or double) takes precedence. However in XTerm, underline and double-underline are actually separate attributes, and double-underline takes precedence when both are set, regardless of the order they were applied.

I'm not sure it matters that much, but I'd probably be inclined to follow the XTerm approach.

I moved the kitty extended underline discussion to #7228, since this is going to be closed by #7223. I'm working on collecting the points and opinions about bold/bright/intense/bold into a separate issue to track as part of #6879.

:tada:This issue was addressed in #7223, which has now been successfully released as Windows Terminal Preview v1.3.2382.0.:tada:

Handy links:

microsoft / terminal

Add support for parsing Underlined and Doubly Underlined to the parser #2916