Request: The default color scheme should stick mostly to the base 16 colors (0-15) and greyscale (232-255)

[mattmc3]

Request: The default 8-bit color scheme is harmful to the user experience, and ble.sh by default should stick (mostly) to the base 4-bit colors (0-15) and grayscale (232-255):

---

I know color scheme preferences are highly subjective, so let me preface with what this request is not:

• It is not asking for ble.sh to adopt some new popular color scheme
• It is not a critique of the existing color choices - only of the range of color codes chosen to represent them
• It is not without full recognition that people can already configure the colors however they choose

I'll try to stick only to objective points.

I understand that the colors are configurable, which is awesome! People can do their own thing, and ble.sh's color choices will never make everyone happy. That said, using so many colors outside the user-defined 0-15 range (and the 232-255 grayscale range) by default is likely to make ble.sh's highlighting really difficult to see/read for many users. For a project like ble.sh that does a wonderful job making Bash a significantly more user-friendly shell, using colors outside this range undermines that usability. As an example, all the fg=92 colors needed to be changed for me because they were unreadable with my terminal theme. I couldn't try out ble.sh without additional effort.

By switching the defaults to the terminal's user-defined color ranges, ble.sh gains the following benefits:

• You no longer need to worry about dark or light terminal backgrounds. The user has already picked the 16 colors that work well in that terminal - changing the defaults means that ble.sh then simply respects those choices
• The initial user experience of ble.sh is vastly improved because a user should never have to go configure things to fix colors to make ble.sh usable with their setup. This is why picking good defaults are really important.
• ble.sh will naturally adapt as users change terminal color schemes without the need for additional user configuration
• Accessibility improves for color blind users and for users that need high contrast. They picked colors for their terminal that work for their vision impairment, and ble.sh should respect those by default

I appreciate all the hard work that went into this project. It's really clever, and a much needed re-think of readline's limitations. Thank you for considering this request!

[akinomyoga]

I think we can reconsider those colors, but I would avoid using some of the basic 16 colors (such as 4, 5, 10, and 11) and still use some of XTerm's 256 colors after consideration.

• You no longer need to worry about dark or light terminal backgrounds.

This is false in my experience. If you look at the default settings of the basic 16 colors of different terminals, you'll realize that the basic 16 colors are not necessarily more visible than the selected colors in 16-255. Actually, this is "the reason" that I intentionally use some colors from outside the basic 16 colors.

For example, the characters with the foreground colors 4 (dark blue) and 5 (dark magenta) are sometimes not visible in the terminals with a dark background. This is the case with the most traditional 16-color palette, the 4-bit VGA colors. Some Windows consoles (Command Prompt and PowerShell) and terminals in Windows (PuTTY, Mintty, etc.) still use relatively dark default colors for the first 8 colors. The foreground colors 10 (light green) and 11 (light yellow) are usually invisible in the terminals with a light background. For example, XTerm's default is very bright 10 and 11 with the white background.

• The problem is that the basic 16 colors depend on the settings, and we cannot really make sure that they would be rendered in visible colors. Even if they are visible in some terminals or some themes, they are not ensured to be visible with different themes, particularly the default palettes of different terminals.
• On the other hand, the colors >= 16 are rendered in the pre-determined color (in mostly all the modern terminals with the 256 color support).

Of course, there are still other factors such as the display settings and also the individual visual perception, but they affect both the basic 16 colors and xterm's 256 colors anyway.

The user has already picked the 16 colors that work well in that terminal

I also have to doubt this assumption. Of course, some users would pick the 16 colors so that any foreground colors can be visible. However, I don't think all the users would carefully adjust the basic 16 colors. I think most of the users would use the terminal's default.

---

As an example, all the fg=92 colors needed to be changed for me because they were unreadable with my terminal theme.

Specifically for fg=92, it corresponds to #8700D7 [≡ rgb(135 0 215)] in the de facto standard palette (XTerm's 256-color palette). Personally, it appears to be visible in the dark background, and it is definitely brighter than the "dark blue" in many terminals (VGA, WniXP console, Terminal.app, PuTTY, mIRC, Mintty, urxvt, GNOME Terminal):

• https://en.wikipedia.org/wiki/ANSI_escape_code#3-bit_and_4-bit
• https://github.com/mintty/mintty/wiki/Tips#changing-colours
• https://auewe.hatenablog.com/entry/2014/05/30/042151

This means that even if we constrain the colors within the basic 16 colors, in those terminals, you wouldn't be able to see the dark blue characters (which is usually used for the directory names by ls, etc.).

Or maybe do you use a purple background for the terminal?

[akinomyoga]

@mattmc3 Just in case, let me confirm your situation.

• Q1: What is the background color of your terminal?
• Q2: Also, does your terminal have the standard 256-color palette?

FWIW, I know you are talking about the default settings of ble.sh, but if you want to force ble.sh to use the basic 16 colors, you can configure it by a single option:

# blerc
bleopt term_index_colors=16

This forces ble.sh to use the basic 16 colors, i.e., all the index colors above 16 are mapped to the closest basic 16 colors.

[mattmc3]

I use iTerm2 on macOS with various color schemes from https://github.com/mbadolato/iTerm2-Color-Schemes. Even though this site is labeled for iTerm2 themes, it has all the same ones for many other terminals - Alacritty, Terminal.app, PuTTY, kitty, Windows Terminal, etc. I think Terminal.app is the only one with 8-bit colors - all the rest support modern 16-bit true colors. Perhaps I have a faulty assumption that modifying 0-15 is how other theming engines work since that's how this one does.

FWIW, I know you are talking about the default settings of ble.sh, but if you want to force ble.sh to use the basic 16 colors, you can configure it by a single option:

I'm not looking to use the limited terminal default color palette so much as I'm looking for ble.sh to respect when that palette has already been themed. So, as an extreme example, when 1-red is defined to be visually green like with the Hulk theme, it stays green.

[akinomyoga]

I use iTerm2 on macOS with various color schemes from https://github.com/mbadolato/iTerm2-Color-Schemes. Even though this site is labeled for iTerm2 themes, it has all the same ones for many other terminals - Alacritty, Terminal.app, PuTTY, kitty, Windows Terminal, etc.

This site seems to provide many themes "to" many terminals, but it doesn't mean it covers all the themes "from" all the other terminals. Although the themes on this site seem to be designed so that all the basic 16 colors (except for the one that matches the background one) can be safely used as a foreground color, it doesn't mean other themes that do not appear in this site also consider it. As I explained in my previous reply, there do exist many basic 16-color palettes where not all 16 colors are intended to be used as foreground colors.

I think Terminal.app is the only one with 8-bit colors - all the rest support modern 16-bit true colors.

I assume you are talking about the SGR color specification of 38;5 / 48;5 (index colors) and 38;2 / 48;2 (RGB 24-bit colors), but I'm not sure how it is related to the present discussion. The color specification used in the theme configuration and the SGR color specification used by terminal applications are usually separate. Or, do we need to use the SGR codes to configure the themes in Terminal.app (I don't have macOS, so I don't have a way to check the theming interface of Terminal.app)?

Perhaps I have a faulty assumption that modifying 0-15 is how other theming engines work since that's how this one does.

Could you describe it in a bit more detail? What is the assumption?

---

We probably want to make a list of safe colors. Let us assume that the background color is black or white (or almost white) because if we take into account a gray background or colorful background, no colors are safe.

After looking at different themes, I realized that 14 (bright cyan) can also be invisible with light background themes. Of course, the colors that can typically be a background one (i.e., white 15 and black 0) are unsafe. In addition, the brightness of grays 7 and 8 also varies, so they are not safe either. It appears that only seven colors (1, 2, 3, 6, 9, 12, and 13) out of the basic 16 colors are safe for the light and dark background colors.

The 24-level grayscale colors (232-255) are similar. The gray colors that are too close to black or white are unsafe. To my eyes, only the gray around 247 seems to be safe for both the light and dark backgrounds. We may additionally use "gray"s around 247, such as 246 and 248, but they are essentially similar to 247 so it's subtle that I would count them as distinct colors.

Honestly, 7 or 8 is too small for the number of available colors considering the variety of the highlighted elements supported by ble.sh. I still would like to consider a part of the 6x6x6 color cube (16-231) as available colors. Some colors from the 6x6x6 color cube are too dark or too light to be safe, but not all the colors in 6x6x6 are unsafe. They do not automatically change to the colors linked to the theme that the user has chosen, but it's usually irrelevant to the brightness, which has a direct impact on the character's visibility.

[pallaswept]

I had some trouble recently with the colours, when I had a buggy update that killed my GUI, and I was forced to use TTY. The 256-colour scheme caused me a few problems there with invisible characters.

I ended up with a section in my init.sh which sets 4 bit colours first, then uses if [ `tput colors` == "256" ] to set 8 bit colour if I can.

I was just passing by and saw this issue, and that experience seems relevant.

[akinomyoga]

I had some trouble recently with the colours, when I had a buggy update that killed my GUI, and I was forced to use TTY. The 256-colour scheme caused me a few problems there with invisible characters.

This sounds like the situation is slightly different from the topic in this discussion (the original discussion is for the case where the 256 index colors are correctly shown in the expected color, yet the characters are invisible due to the small contrast to the background color). The operating system console (the one that can be still used when GUI is dead) usually doesn't have support for the 256 colors, so the 256 index colors will not be shown in the originally expected colors. Then, there are two possibilities for the situation you experienced.

• One possibility is that ble.sh failed to detect the correct number of colors supported by the console and outputted the SGR codes for the 256 index colors. In this case, all the color codes will be wrongly interpreted by the console and causes unexpected behaviors. However, I think this is unlikely because ble.sh uses tput colors to detect the number of supported colors. The only exception is the case where TERM starts with xterm or kterm, where the 256 index colors are assumed to be supported even if tput colors reports a less number. However, I hardly think the console sets TERM=xterm or TERM=kterm.
• In other cases, ble.sh automatically maps the 256 index colors to the closest basic 16 colors. In this case, some colors can be mapped to the dark blue, dark magenta, etc., which would result in invisible characters. In particular, 4-bit VGA colors are probably used in the console, which has too dark blue and magenta for the character's visibility.

In this sense, the story by @pallaswept seems to suggest that the dark colors in the basic 16 palettes are actually unsafe. If one sets the colors in the basic 16 colors, one needs to avoid 4 and 5 at least.

[akinomyoga]

@mattmc3 @pallaswept I'd like to know the brightness range of the foreground colors that can be visible in light and dark backgrounds. Could you see the following picture and answer the questions below the picture? The blue number indicates the brightness (i.e. $Y' = (0.2126R^\gamma+0.7152G^\gamma+0.0722B^\gamma)^{1/\gamma}$ for people free from color blindness, where (R,G,B) are the sRGB intensities). The black and white parts show strings in the corresponding grayscale foreground color.

The appearance should depend on the display setting and the characteristics of the individual visual perception.

• What is the minimum acceptable brightness in the black background for you?
• What is the maximum acceptable brightness in the white background for you?

[mattmc3]

The answers are going to be really dependent on whether I'm in a dark room at night with only light from a dim laptop screen, or at full brightness in daylight. It's also going to depend on whether I'm fresh at the start of the day, or my eyes are tired after 10 hours on a screen. So this is far from scientific, but I'll offer an opinion if you think it will help your research.

For something like autocomplete where it's strictly informational and I'd only passively read it, I could probably go as low as 087 on black, but 096 might be a bit more comfortable especially if it's something I need to read often. Similarly, on white, I wouldn't want anything brighter than 198, but 174 would be more comfortable for longer text.

[akinomyoga]

The answers are going to be really dependent on whether I'm in a dark room at night with only light from a dim laptop screen, or at full brightness in daylight. It's also going to depend on whether I'm fresh at the start of the day, or my eyes are tired after 10 hours on a screen.

Yeah, those are also important factors. In addition, the font size would also largely affect the legibility in my experience. The larger the font size is, the less the contrast is needed. So I understand that these are just fragile references anyway, yet I guess it would still be better than deciding the range based on only my experience. I also would like to know how much it would fluctuate depending on the environment, the person, and the situation.

So this is far from scientific, but I'll offer an opinion if you think it will help your research.

Yes, I expect and welcome such opinions!

For something like autocomplete where it's strictly informational and I'd only passively read it, I could probably go as low as 087 on black, but 096 might be a bit more comfortable especially if it's something I need to read often. Similarly, on white, I wouldn't want anything brighter than 198, but 174 would be more comfortable for longer text.

Thank you. Maybe we can prepare two levels of color lists for stricter ones (such as $96 \le Y' \le 174$) and more permissive ones ($87 \le Y' \le 198$). FWIW, my personal limits in my current environment are something like $100 \le Y' \le 200$.

---

Here's the list of colors whose brightness satisfies $100 \le Y' \le 174$:

The black numbers in the first column show the color index (16..255). The second column shows the corresponding color. The third to fifth columns show the color emulation with different color blindness (The third column is for protanopia, the fourth is for deuteranopia, and the fifth is for tritanopia). The conversion was performed by the formulae explained in this page, (though I'm not sure if these are really trustable). The blue numbers Y=??? from the second to fifth columns show the brightness of the colors. The two black numbers in the sixth column show the perceptual color distances from black and white, respectively. The color distances are calculated by using CIE76 $\Delta E^*_{ab}$ based on colorspace CIELAB.

I calculated those values for all the 240 colors beyond the basic 16 colors (16..255) and filtered only the color indices where all of the color and corresponding color blind versions have brightness in the range $100 \le Y' \le 174$. Only 93 colors remained after filtering. For example, fg=92 mentioned in the original post was filtered out because it was too dark. I'll probably remap the default colors in ble.sh faces as 26 -> 33, 92 -> 99, 94 -> 100, 124 -> 166, 4 (navy) -> 63, and 5 (purple) -> 133.

I also tried filtering by the color distance $\Delta E^*_{ab}$ from white and black (and other definitions of distances), but they didn't seem to be closely related to the characters' visibility. Currently, the filtering by the color distance is not performed.

[pallaswept]

This sounds like the situation is slightly different from the topic in this discussion

I agree. My apologies, I did not intend to hijack or interrupt.

I thought that maybe this trick of using 'tput colors' might be helpful, but I did not realise that ble.sh already does this by default. Of course, I should know by now that you have thought of this already :D Since I have read how it works in this thread, and I saw the 256 colour codes in use in my TTY, I believe that my experience is due to my own config being wrong, so I should fix that here.

The info in this thread will be very helpful in fixing my issue, so I thank you both, and apologise again for interrupting.

[akinomyoga]

This sounds like the situation is slightly different from the topic in this discussion

I agree. My apologies, I did not intend to hijack or interrupt.

Sorry, I didn't mean it was hijacking or interrupting! It was actually helpful and gave insights from another perspective.

I thought that maybe this trick of using 'tput colors' might be helpful, but I did not realise that ble.sh already does this by default.

Yeah, the result of tput colors (or tput Co in BSD tput) is stored in the shell variable $_ble_term_colors, and it is used to determine the default color numbers in this block.

Since I have read how it works in this thread, and I saw the 256 colour codes in use in my TTY, I believe that my experience is due to my own config being wrong, so I should fix that here.

Hmm, maybe it's not your fault; ble.sh's logic to determine the use of 256 colors might still have a bug or flaw. What is the result of the following command in TTY?

echo "$TERM, $_ble_term_colors, $_ble_color_index_colors_default"

Although ble.sh changes the behavior based on the terminal's color support obtained by tput colors, I was too lazy to prepare dedicated face settings for the reduced colors. Currently, ble.sh only has the default face settings designed for 256 colors, and for the terminals with a small number of colors, ble.sh just converts the face settings for 256 colors to 16 colors by picking "the closest 16 colors". This color reduction may not work well. I should maybe prepare a separate set of the default face settings tailored for 16 colors.

[pallaswept]

Hmm, maybe it's not your fault

I did not want to bother either of you with details, but with my new understanding of this, I am almost certain that it is my fault. It's the way I set a few of my custom colours in my init.sh, they had no opportunity for handling by ble.sh. I had unintentionally hard-coded them to 256-color mode.

Just in case:

In Konsole:

> echo "$TERM, $_ble_term_colors, $_ble_color_index_colors_default"
xterm-256color, 256, 256

In TTY: linux, 8, 8

Seems correct.

I stay subscribed to this thread anyway, because when I fix my init.sh I will use the features discussed here - so thank you both very much for looking into this!

[akinomyoga]

I added changes in the latest push.

• In commit e4cce0eaa578e0da91d8fd6af7b94a7fbacf13cb, the default fg colors of the faces that use too dark colors are replaced with similar but brighter colors.
• Also, I added an option bleopt color_scheme to allow users to select a preset color scheme from the contrib/scheme directory. Currently, only default and base16 are prepared, but the user who doesn't want any 256 index colors can specify bleopt color_scheme=base16.
• In https://github.com/akinomyoga/blesh-contrib/commit/7c6f4e9dc523eba562baed6de5146ede9b7f7a2b, I added bleopt colorglass_base16_palette and bleopt colorglass_blindness. If your terminal supports the 24-bit colors, after running ble-import colorglass, you can set bleopt colorglass_blindness to simulate the colors with color blindness. Also, if you set bleopt colorglass_base16_palette, you can replace the base16 colors with the 24-bit colors in the specified palette. The list of base16 palettes available for colorglass_base16_palette is found at List of base16 palettes (I wanted to create a color sample page using Markdown, but it turned out that Markdown (GFM) doesn't have a way to show the specified 24-bit color (outside the Issue comments. For this reason, the color samples are shown with HTML in GitHub Pages). You can check the colors with the current colorglass settings by running ble palette.
• The color samples used to adjust the default color scheme can be shown using the script make/color.sample.sh. The brightness sample in comment 2292604626 can be shown by the command make/color.sample.sh show-brightness-sample. The list of safe colors in comment 2293331282 can be shown by the command make/color.sample.sh list-safe-colors. The HTML version is also available here.

[pallaswept]

I am seriously impressed

[mattmc3]

🤩 Awesome!!!

I think there's a bug, however. When I run bleopt color_scheme=base16 after my config loads, it works great. But when I put that line in my ~/.config/blesh/init.sh file, I get this error:

ble-face: face 'region_@' not found
ble-face: face 'disabled' not found
ble-face: face 'overwrite_mode' not found
ble-face: face 'vbell' not found
ble-face: face 'vbell_@' not found
ble-face: face 'syntax_@' not found
ble-face: face 'command_@' not found
ble-face: face 'filename_@' not found
ble-face: face 'varname_@' not found
ble-face: face 'argument_@' not found
ble-face: face 'prompt_status_line' not found
ble-face: face 'cmdinfo_cd_cdpath' not found
ble-face: face 'auto_complete' not found
ble-face: face 'menu_filter_fixed' not found
ble-face: face 'menu_filter_input' not found
ble-face: face 'menu_desc_default' not found
ble-face: face 'menu_desc_type' not found
ble-face: face 'menu_desc_quote' not found
ble-face: face 'argument_error' not found
ble-face: face 'auto_complete' not found
ble-face: face 'cmdinfo_cd_cdpath' not found
ble-face: face 'command_directory' not found
ble-face: face 'command_function' not found
ble-face: face 'disabled' not found
ble-face: face 'filename_directory' not found
ble-face: face 'filename_directory_sticky' not found
ble-face: face 'filename_orphan' not found
ble-face: face 'filename_setgid' not found
ble-face: face 'filename_setuid' not found
ble-face: face 'menu_filter_input' not found
ble-face: face 'overwrite_mode' not found
ble-face: face 'prompt_status_line' not found
ble-face: face 'region' not found
ble-face: face 'region_insert' not found
ble-face: face 'region_match' not found
ble-face: face 'region_target' not found
ble-face: face 'syntax_brace' not found
ble-face: face 'syntax_comment' not found
ble-face: face 'syntax_document' not found
ble-face: face 'syntax_document_begin' not found
ble-face: face 'syntax_error' not found
ble-face: face 'syntax_expr' not found
ble-face: face 'syntax_function_name' not found
ble-face: face 'syntax_glob' not found
ble-face: face 'syntax_history_expansion' not found
ble-face: face 'syntax_param_expansion' not found
ble-face: face 'syntax_tilde' not found
ble-face: face 'syntax_varname' not found
ble-face: face 'varname_array' not found
ble-face: face 'varname_empty' not found
ble-face: face 'varname_export' not found
ble-face: face 'varname_expr' not found
ble-face: face 'varname_hash' not found
ble-face: face 'varname_number' not found
ble-face: face 'varname_readonly' not found
ble-face: face 'varname_transform' not found
ble-face: face 'varname_unset' not found
ble-face: face 'vbell_erase' not found

I removed my ~/.local/share/blesh and re-ran make in case that would help, but it didn't. If I have bleopt color_scheme=default in my config, it does work though. Something with that base16 load is breaking.

[akinomyoga]

Ah, thanks. Confirmed. This would be an easy fix.

[akinomyoga]

@mattmc3 Thank you. I fixed it in commit 5f5554a, tested, and pushed. You can now update ble.sh by ble-update.

[mattmc3]

Works (and looks) great! I really appreciate having a base16 option that changes with the terminal theme. With that as a starting point, it means way fewer custom ble-face overrides are needed. Thanks again @akinomyoga!