Support for terminal attribute control (ANSI ESC sequences and terminal detection)

[mintty]

Apparently winpty transforms ESC characters on purpose (https://github.com/rprichard/winpty/issues/47 and e.g. https://github.com/Microsoft/vscode/issues/22616#issuecomment-288280866).

However, this raises problems with native Windows tools that try to apply ANSI features, esp. colour attributes (e.g. https://github.com/rprichard/winpty/issues/104#issuecomment-287016774 or https://github.com/git-for-windows/git/issues/1470#issuecomment-365618938).

The intrinsic use case for winpty is to use it in a traditional terminal and support terminal features for native tools. If such a tool thinks it’s running within such a terminal, it may employ ANSI escapes on purpose. Therefore I think winpty should not filter ESC or unset TERM (https://github.com/rprichard/winpty/issues/106), at least not by default.

I also don’t understand your statement in https://github.com/fusesource/jansi/issues/64#issuecomment-249695306:

the escape sequence may be unreliably interpreted by the terminal in which winpty runs

because it’s the nature of the terminal to interpret ANSI controls, and also it’s in contradiction to your previous stance in https://github.com/rprichard/winpty/issues/35#issuecomment-224837490:

winpty is designed to output escape sequences normally

So as suggested from my description, I’m asking to please revert this transformation and pass through ESC controls as well as the TERM environment variable transparently by default; an option may change both simultaneously.

[rprichard]

winpty scrapes a console's 2D buffer and renders what's visible into terminal escapes. If the Windows console doesn't interpret an escape sequence, then it is drawn into the 2D buffer, and by the time winpty sees an escape character drawn in a cell, it is too late to interpret a command sequence.

For example, suppose we have:

1. a Windows console app, running on Win7
2. a console of size 16x2 (16 cols, 2 rows)

The console app:

1. clears the screen
2. sees TERM=xterm and decides to print VT escape sequence (e.g. because some of its code is cross-platform)
3. writes This text is \x1b[32mGreen\x1b[0m. to the console where \x1b is an escape character

When the app is run in an ordinary Windows console, the window looks like this (a ? is actually U+001B, and | is the position of the cursor):

line 1: [This text is ?[3]
line 2: [2mGreen?[0m.|   ]

Both escape sequences occupy cells in the 2D buffer, and the first one has wrapped around. With winpty, this console window still exists, but is normally hidden. (It can be made visible by setting the environment variable WINPTY_SHOW_CONSOLE to 1 and reinvoking winpty.)

One simple failure mode is that winpty sets the cursor position to line 2, column 13. If it doesn't escape the escape character on line 2, then the cursor will appear to be several cells after the end of the 2mGreen. text.

The bigger problem is that winpty also inserts control characters / escape sequences before/after each line. I suspect it would insert a clear-rest-of-line sequence before the 3 on the first line, and it probably inserts a CRLF sequence between lines 1 and 2. Escape sequences can't be nested.

winpty generally tries to make the terminal look like the underlying console buffer, so if programs have broken output, like the ones you've linked to, then the programs are usually at fault. They will also have broken output when they're run in a normal console. Fixing them is usually straightforward. The conventional fix is to use SetConsoleTextAttribute instead of escape sequences. (A common approach is to use escapes internally and include a layer that translates them to SetConsoleTextAttribute.) Starting with Windows 10, a program can set ENABLE_VIRTUAL_TERMINAL_PROCESSING and write escapes like with Unix.

The intrinsic use case for winpty is to use it in a traditional terminal and support terminal features for native tools. If such a tool thinks it’s running within such a terminal, it may employ ANSI escapes on purpose. Therefore I think winpty should not filter ESC or unset TERM (#106), at least not by default.

The child program isn't directly running under the terminal, though. It's running inside a Windows console. winpty translates terminal input into INPUT_RECORD, and it scrapes the 2D CHAR_INFO buffer and paints it using terminal escapes.

[mintty]

Thank you for elaboration, understood. I was imaginating that winpty would intercept invocations of WriteConsole* which would be a different base situation. I wonder though how cygwin handles this, as some applications show the mentioned problem under winpty but get their escapes interpreted when run directly from cygwin. About the Window 10 terminal mode, I assume that would be handled before filling the screen buffer, so winpty handles that implicitly?

[rprichard]

I was imaginating that winpty would intercept invocations of WriteConsole* which would be a different base situation.

Intercepting the APIs would let winpty do a better job (and have lower CPU consumption and latency). I'm not sure there's a reliable way to do that.

I wonder though how cygwin handles this, as some applications show the mentioned problem under winpty but get their escapes interpreted when run directly from cygwin.

With Cygwin, the escape interpretation happens in the Cygwin/MSYS DLL, so the problem can happen if the program running under winpty isn't a Cygwin executable.

About the Window 10 terminal mode, I assume that would be handled before filling the screen buffer, so winpty handles that implicitly?

Yes. A program may need to be modified to enable the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode, but as long as the mode is enabled, Windows interprets the escape sequences and does not write them into the visible buffer, so winpty doesn't see them when it calls ReadConsoleOutput.

[perlun]

@rprichard @mintty I stumbled into this issues because of some issues I'm seeing in my console-based app (more details in https://github.com/perlang-org/perlang/issues/285). The short version:

• System.Console.ReadKey in .NET doesn't work when program is being executed via Git Bash (because stdin is redirected).
• I added detection for this in my code to relaunch with winpty <myprogram> <args> if stdin redirection is detected (on Windows only).
• When running via winpty, ANSI colour sequences no longer work. I am testing this on Windows 10.0.19044.1526.

What you write about here:

A program may need to be modified to enable the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode

...could this be (part of) the solution in my case? 🤔 As noted, the ANSI colour sequences work correctly when executed outside of winpty but perhaps I'd need to do some special tricks anyway to make the Windows console subsystem enable these features? (this is way out of my own area of expertise, so any help/suggestions are highly regarded 🙏)

[mintty]

If you run a current version of cygwin on Windows 10, you shouldn't need winpty anymore. If you run MSYS, I suggest to enable ConPTY support by setting MSYS=enable_pcon before starting a terminal; I will add a hint to the wiki.

[perlun]

If you run MSYS, I suggest to enable ConPTY support by setting MSYS=enable_pcon before starting a terminal; I will add a hint to the wiki.

Thanks, I tried enabling this now but it doesn't seem to make any difference. I still get the same .NET exception (Cannot read keys when either application does not have a console or when console input has been redirected. Try Console.Read.) when trying to call System.Console.ReadKey().

The Git Bash I'm using seems to use mintty 3.1.0 (x86_64-pc-msys). It's a fairly old Git for Windows I think (2.24.1.windows.2), so perhaps upgrading it would give me a slightly newer mintty as well.

Do note though (second screenshot in https://github.com/perlang-org/perlang/issues/285) that ANSI colors actually work already, but I guess the "does not have a console" issue should also go away if I manage to get ConPTY support properly enabled? 🤔

---

A side note which I've experienced is that my app seems to behave differently when compiled as a regular debug/release build with "AnyCPU" vs when I compile it in Release mode, targeting x86-64 specifically and enabling more "release-oriented" flags in the compilation. In the latter case, I don't seem to be detecting the fact that stdin is redirected correctly, so I wonder if something weird is going on here...

[mintty]

I do not know whether ConPTY catches a low-level attempt to do I/O in a way deliberately incompatible with terminals, you'll need to try. I also don't know whether you environment supports the invocation MSYS=enable_pcon mintty already. Check uname -a, if it reports a version of at least 3.1.0, it should work.

[perlun]

Check uname -a, if it reports a version of at least 3.1.0, it should work.

3.0.7-338.x86_64, so apparently too old for that. I upgraded Git for Windows to a more recent version and it even provided a GUI for setting this flag now:

With that in place, everything works perfectly in Git Bash. No more WinPTY hack needed! 🎉

However, when executing Git Bash in the Windows Terminal, ANSI sequences are not being parsed correctly for some super-odd reason:

Anyway, this is in no way related to MinTTY, as this is just C:/Program Files/Git/bin/bash.exe -i -l being executed in the Windows Terminal. I'll try to find the appropriate repo to write a comment about this in. (feel free to let me know if you have any ideas though.)

Thanks for your help, it was really helpful. 👍

[perlun]

(Hmm, my problem could be a permutation of https://github.com/git-for-windows/git/issues/2483. I'll ask there for advice.)