Color support and ANSI terminal detection on Windows (broken for CMD.exe and Powershell.exe)

[e-d-n-a]

Description

I get weird and wrong color/ANSI-detection and progressbar output containing unprocessed ANSI-escape sequences in a CMD.exe or powershell.exe console on Windows, now that I have updated to the latest version (v4.4.2).

The main issue is, that some tools of mine that use progressbar2 now output ANSI-escape sequences with default/no color settings set on the ProgressBar-instance, although the console doesn't support it. It seems also NOT possible to globally deactivate color support for a console with an environment variable (you can only forcefully activate it). I don't want to change all my initialization code to deactivate colors and I would want to actually see colors with progressbar2 using the Windows-API (as seen in the module windows.py).

Output of code from below with 'enable_colors=None' (with NO colors visible):

isatty: True
Version: 4.4.2
Progressbar(line_breaks=False, is_terminal=True, is_ansi_terminal=True, enable_colors=16777216)
←[38;2;255;0;0m  0%←[39m ←[38;2;255;0;0m(0 of 5)←[39m |
←[38;2;255;111;0m 20%←[39m ←[38;2;255;111;0m(1 of 5)←[39m |#########
←[38;2;255;167;0m 40%←[39m ←[38;2;255;167;0m(2 of 5)←[39m |##################
←[38;2;247;255;0m 60%←[39m ←[38;2;247;255;0m(3 of 5)←[39m |###########################
←[38;2;190;255;0m 80%←[39m ←[38;2;190;255;0m(4 of 5)←[39m |####################################
←[38;2;0;255;0m100%←[39m ←[38;2;0;255;0m(5 of 5)←[39m |#############################################
#| Elapsed Time: 0:00:00 Time:  0:00:00

Expected output as with 'enable_colors=False' (with NO colors visible):

isatty: True
Version: 4.4.2
Progressbar(line_breaks=False, is_terminal=True, is_ansi_terminal=True, enable_colors=16777216)
100% (5 of 5) |##############################################| Elapsed Time: 0:00:00 Time:  0:00:00

Note that even with 'enable_colors=False' I get 'enable_colors=ColorSupport.XTERM_TRUECOLOR' anyways somehow, but no escape sequences! I would expect it to be 'enable_colors=ColorSupport.NONE' though, as it is when initialized in a Python REPL-session (see below)! Also 'is_ansi_terminal' should be False, when ANSI-codes are not supported!?

>python
Python 3.9.17 (main, Jun  8 2023, 18:39:20) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import progressbar as pb2
>>> pb = pb2.ProgressBar()
>>> pb.enable_colors
<ColorSupport.XTERM_TRUECOLOR: 16777216>
>>> pb.is_ansi_terminal
True
>>> pb = pb2.ProgressBar(enable_colors=False)
>>> pb.is_ansi_terminal
True
>>> pb.enable_colors
<ColorSupport.NONE: 0>

I don't have the time right now trying to fully understand the new terminal/color detection/support logic and this described behavior I get. I can only share what I noticed so far.

Questions

1. Is there a way to deactivate colors globally or can it be made possible? I would expect the value of an environment variable like 'PROGRESSBAR_ENABLE_COLORS' to actually be used and not just its existence. (see in bar.py)

2. Why is the instance sometimes showing a non-0 value for enable_colors, but no escape codes are outputted? This way I cannot verify, that colors have really been deactivated. Also it adds confusion for me on how the code works. Shouldn't it be 'ColorSupport.NONE' if I set 'enable_colors=False' on the constructor!? (see in _determine_enable_colors in bar.py)

3. The Windows-API code (print_color, set_text_color) in windows.py is currently NOT used at all, because of the buffering issue!? (see my comment below)

4. Shouldn't the ANSI-support detection on Windows be based on the flag 'ENABLE_VIRTUAL_TERMINAL_PROCESSING'? In env.py i can see, that a test for the flag 'ENABLE_PROCESSED_OUTPUT' is responsible for setting the color-support on Windows and with default settings, but this is unspecific to ANSI-code support. Or this rather only decides on the amount of colors available!? I would expect the code to verify that 'ENABLE_VIRTUAL_TERMINAL_PROCESSING' is actually set for the console, before it outputs ANSI escape sequences. Am I wrong? I can see that you use this flag for setting up the console on Windows.

EDIT: See also this answer on SuperUser on 'Windows console with ANSI colors handling'.

Code

import sys
import progressbar as pb2
import time

print('isatty:',sys.stderr.isatty())

print("Version:", pb2.__version__)
pb = pb2.ProgressBar()
print(f'Progressbar(line_breaks={pb.line_breaks}, is_terminal={pb.is_terminal}, is_ansi_terminal={pb.is_ansi_terminal}, enable_colors={pb.enable_colors})')

pb = pb2.ProgressBar(max_value=5, enable_colors=None)
# pb = pb2.ProgressBar(max_value=5, enable_colors=False)

with pb:
    for i in range(5):
        time.sleep(0.2)
        pb.update(i)

Versions

• Python version: 3.9-x64
• Python distribution/environment: CPython
• Operating System: Windows 7-x64
• Package version: 4.4.2

[e-d-n-a]

Well, I only later realized that all the Windows-API code is currently unused, while I searched for the use of the 'print_color'-function.

I've now seen this comment, that non-ANSI consoles are still not really supported atm (or won't be anymore!?). It's still unfortunate, that it's not easy to make the ANSI-detection work or deactivate colors globally via an environment variable. If you still want to make colors work on older consoles as well using the Windows-API, then I could try implementing that.

I don't yet understand the buffering issue. Where is an example for the need to defer the output to the console? I guess, it is more relevant for file output!?

The abstract and clean way to implement it would probably be to take over the job of ANSI-support for the console and interpret ANSI-sequences in the output stream, but this feels wrong and overly complicated.

[wolph]

First of all, thank you for the very detailed bug report! That is always greatly appreciated :)

Description

The main issue is, that some tools of mine that use progressbar2 now output ANSI-escape sequences with default/no color settings set on the ProgressBar-instance, although the console doesn't support it. It seems also NOT possible to globally deactivate color support for a console with an environment variable (you can only forcefully activate it).

That's definitely a bug, it should be possible to both activate and deactivate it using an environment variable so I'm looking into that.

I am curious what tools you are using though, I have tested on half a dozen different windows systems in cmd, powershell, pycharm and vscode, and I haven't seen any issues with the current implementation so I'm wondering which edge case is breaking it.

If color support in windows remains buggy I might have to disable it by default, which would be a shame because it was quite a bit of work to get it to the current state :(

Output of code from below with 'enable_colors=None' (with NO colors visible):

None means autodetect so that should work in all cases. For all of the edge cases where it does not work I would like to know if there's any way for me to detect when it cannot work.

Note that even with 'enable_colors=False' I get 'enable_colors=ColorSupport.XTERM_TRUECOLOR' anyways somehow, but no escape sequences! I would expect it to be 'enable_colors=ColorSupport.NONE' though, as it is when initialized in a Python REPL-session (see below)! Also 'is_ansi_terminal' should be False, when ANSI-codes are not supported!?

The thing is, both cmd and powershell in modern windows versions are ansi compatible enough for the purposes of this library. But... it is hard to detect the actual capabilities of a windows terminal, the library current tries some auto detection (and even correction of settings) on windows platforms but it seems even that is not enough yet.

Questions

1. Is there a way to deactivate colors globally or can it be made possible? I would expect the value of an environment variable like 'PROGRESSBAR_ENABLE_COLORS' to actually be used and not just its existence. (see in bar.py)

That is indeed the way, if it does not work that's simply a bug that needs to be solved. It appears to be a windows only bug since I can't reproduce it on other platforms.

To make it easier for the future I'll try to collapse the environment variables into a single overridable variable in progressbar.env so it is easier to enable/disable it globally.

2. Why is the instance sometimes showing a non-0 value for enable_colors, but no escape codes are outputted? This way I cannot verify, that colors have really been deactivated. Also it adds confusion for me on how the code works. Shouldn't it be 'ColorSupport.NONE' if I set 'enable_colors=False' on the constructor!? (see in _determine_enable_colors in bar.py)

I'm honestly not sure why that would happen. You are indeed correct in your assumptions as to how it should work

3. The Windows-API code (print_color, set_text_color) in windows.py is currently NOT used at all, because of the buffering issue!? (see my comment below)

Correct. There's no easy way to incorporate that with the rest of the code which is why I'm not using it. I'm not ruling out the possibility of using it in the future but for now it's dormant code.

4. Shouldn't the ANSI-support detection on Windows be based on the flag 'ENABLE_VIRTUAL_TERMINAL_PROCESSING'? In env.py i can see, that a test for the flag 'ENABLE_PROCESSED_OUTPUT' is responsible for setting the color-support on Windows and with default settings, but this is unspecific to ANSI-code support. Or this rather only decides on the amount of colors available!? I would expect the code to verify that 'ENABLE_VIRTUAL_TERMINAL_PROCESSING' is actually set for the console, before it outputs ANSI escape sequences. Am I wrong? I can see that you use this flag for setting up the console on Windows.

You are correct, but there's another thing to note. It's not only possible to read that flag, but it can be manipulated as well which is what the progressbar currently does.

After enabling the virtual processing flag it assumes that it can safely use ANSI support. Perhaps that's what is going wrong in your case, but I can't be sure without further testing.

From your results I gather that you are using Windows 7? I didn't try that since it's been permanently EOL for well over a year now.

[wolph]

After quite a bit of testing I can't reproduce the issue with the environment flags at all. When I set PROGRESSBAR_ENABLE_COLORS=false it properly sets color support to ColorSupport.NONE.

With this script in test.py:

import progressbar
pb = progressbar.ProgressBar()
print(f'{pb.enable_colors=}')
print(f'{pb.is_ansi_terminal=}')

for i in pb(range(10)):
    import time
    time.sleep(0.2)
    pass

And the first one has colors, the 2nd is just black and white

[e-d-n-a]

I tested it again now and it works for me too! :|

I must have spelled something wrong with the environment variable in my 1st tests and then only reasoned based on the code in bar.py, that I interpreted wrongly. I thought, it doesn't evaluate the value correctly, but it does and now it's obvious to me. This is embarrassing and I'm really sorry for these errors on my part.

I would've commented anyways, that this is the most important aspect to get working, so it's fine for me now. At least this issue cleared up all the other questions I had and that it's probably not worth getting colors to work in old consoles using the WinAPI anyways.

[wolph]

Well... the thing is that the library currently supports both truecolor (24 bits colors) and 8 color mode on windows, that work is already done :)

The only thing that remains is detecting when to use which mode and from your results I'm guessing that the detection does not work. I'll see if I can find an older windows vm to reproduce the issue. It would be nice to automatically detect the color modes in a better way. I want the library to work out of the box without any blocking issues for anyone.