[Discussion] System.Console re-design

[adamsitnik]

The initial design of System.Console was mainly driven by Windows OS capabilities and available APIs.

When .NET became cross platform, a number of issues arose (click on the details button below to see the full list) as there was no good way of mapping some Windows-specific concepts to Unix. A good example are all Console.Window* APIs: on Unix the Window is owned by the Terminal app, the .NET Console App has no control over it.

The current design makes no distinction between Console and Terminal, which is also a source of plenty of issues (mostly for apps that have redirected output).

Moreover, System.Console is a static class and there is no common abstraction that would allow not only for testing but also for integrating projects like spectre.console and System.CommandLine.Rendering.

By creating this issue I would like to start a public and transparent discussion about what is wrong with the current design of System.Console and how we could fix that. Based on the results of this discussion, I would like to create a proposal for .NET 7.

cc @jonsequitur @KathleenDollard @patriksvensson @colombod @Keboo @howardvanrooijen @daxian-dbw

* `Console.ReadKey`: * `Console.ReadKey` returns incomplete information for some key sequences on Linux #802 (.NET 6) * `System.ConsoleKeyInfo` can not handle Unicode surrogate pair and Emoji Sequences #27828 * `Console.ReadKey()` doesn't work for keys from the numeric keypad in the PuTTY ssh client #25735 * [Windows Terminal] `Console.KeyAvailable` causes the next Unicode character input 'EN DASH' to be skipped in console #38966 * .NET 5 fails to read the HOME key on ubuntu16.04 #44621 * Console.ReadKey and pressing SHIFT+END returns invalid escape sequence on WSL/Ubuntu #45597 * System.Console.ReadKey fails in MSYS' console mintty #49210 * Let System.ConsoleKeyInfo able to represent keypress in the Unicode world #51085 * `Console.ReadKey` throws in unexpected circumstances https://github.com/dotnet/runtime/issues/59059 * Support language input keys #63034 * Redirection: * console redirection is broken #22314 * Don't dup stdin & stderr as this can break some std io redicection code #25394 * Console.CursorLeft, Console.CursorTop, Console.WindowWidth, Console.WindowHeight and Console.WindowTopLeft doesn't throw an IOException for invalid values on Windows 7 x86 when Input is redirected #43138 * Colors: * Bright console colors don't work correctly on Linux #23241 * Implement a way to enable color in console even when the output is redirected #33980 * Overload Console.Write and Console.WriteLine to take ConsoleColor as a parameter #39746 * Developers using Console can employ VT/ANSI APIs (eg to use embedded color codes) #44513 (.NET 6) * Add overloads to Console.Write that also take a ConsoleColor parameter (#61731) * Console.Foreground does not correctly set the color in every Terminal (#62186) * Terminal: * MacOS 10.13.6 Console.SetWindowSize() exception #27216 * Terminal hardware is always left in application mode (keypad_xmit) after running a console program #27626 * Console.Clear() doesn't clear the scrollback buffer on Unix-like platforms #28355 * Setting Console.CursorVisible property does not change visibility in Linux #31063 * Cannot use left/right arrow keys to edit text in Console.ReadLine on Unix #38051 * System.Console.Read() needlessly changes terminal settings on linux #49129 * Console issues with multithreaded c# app only when run on Linux #49301 * Encoding: * Changing console's OutputEncoding on linux to unicode generates garbage #29735 * Consider changing the default console encoding on Windows to UTF-16 #31466 * Console.OutputEncoding by default returns a wrong value on Windows Nano Server #42959\ * Console UTF-8 input is misbehaving on Windows #43295 * `Console.CancelKeyPress`: * Third party code may remove cancel keypress handler #30464 * Setting Console.TreatControlCAsInput to true causes text to be removable #44667 * .NET 5 apps can no longer intercept SIGINT signals (receive CancelKeyPress events) when running under Docker #51221 * Other: * FreeBSD: System.Console is not working right #23653 * Console.Write prints a new line on iOS #36440 * Console.In.Peek() always returning EOF after the first read line on Windows #40735 * Console.KeyAvailable throws when input isn't redirected on Windows 7 x86 #43015 * "System.UnauthorizedAccessException: Access to the path is denied." from Console.WriteLine on Ubuntu 20.10 #48654 * Performance: * `Console.OpenStandard*` few times slower on Linux #31396 * New APIs: * We need a cross-platform way to detect key-down state #24314 * Enhance Console.ReadLine() to return at Tab #800 * Add API for detecting broken pipe when write to stdout(or stderr) #30540 * Proposal: Async Console I/O APIs #299 * Test issues: * [mono] Test failed on windows: WindowAndCursorProps.Title_Set_Windows #34454 * System.Console tests failing on iOS #36878 * System.Console.Tests fail on Android #37465 * CancelKeyPressTests.HandlerInvokedForSigQuit test failing on OSX in CI #38998 * Docs: * System.Console: Backport MS Docs documentation to triple slash #48959

[ghost]

Tagging subscribers to this area: @carlossanlop See info in area-owners.md if you want to be subscribed.

Issue Details

---

The initial design of `System.Console` was mainly driven by Windows OS capabilities and available APIs. When .NET became cross platform, a number of issues arose (click on the details button below to see the full list) as there was no good way of mapping some Windows-specific concepts to Unix. A good example are all `Console.Window*` APIs: on Unix the Window is owned by the Terminal app, the .NET Console App has no control over it. The current design makes no distinction between `Console` and `Terminal`, which is also a source of plenty of issues (mostly for apps that have redirected output). Moreover, `System.Console` is a `static class` and there is no common abstraction that would allow not only for testing but also for integrating projects like [spectre.console](https://github.com/spectreconsole/spectre.console) and [System.CommandLine.Rendering](https://github.com/dotnet/command-line-api/tree/main/src/System.CommandLine.Rendering). By creating this issue I would like to start a public and transparent discussion about what is wrong with the current design of `System.Console` and how we could fix that. Based on the results of this discussion, I would like to create a proposal for .NET 7. cc @jonsequitur @KathleenDollard @patriksvensson @colombod @Keboo @howardvanrooijen @daxian-dbw

* `Console.ReadKey`: * `Console.ReadKey` returns incomplete information for some key sequences on Linux #802 (.NET 6) * `System.ConsoleKeyInfo` can not handle Unicode surrogate pair and Emoji Sequences #27828 * `Console.ReadKey()` doesn't work for keys from the numeric keypad in the PuTTY ssh client #25735 * [Windows Terminal] `Console.KeyAvailable` causes the next Unicode character input 'EN DASH' to be skipped in console #38966 * .NET 5 fails to read the HOME key on ubuntu16.04 #44621 * Console.ReadKey and pressing SHIFT+END returns invalid escape sequence on WSL/Ubuntu #45597 * System.Console.ReadKey fails in MSYS' console mintty #49210 * Let System.ConsoleKeyInfo able to represent keypress in the Unicode world #51085 * Redirection: * console redirection is broken #22314 * Don't dup stdin & stderr as this can break some std io redicection code #25394 * Console.CursorLeft, Console.CursorTop, Console.WindowWidth, Console.WindowHeight and Console.WindowTopLeft doesn't throw an IOException for invalid values on Windows 7 x86 when Input is redirected #43138 * Colors: * Bright console colors don't work correctly on Linux #23241 * Implement a way to enable color in console even when the output is redirected #33980 * Overload Console.Write and Console.WriteLine to take ConsoleColor as a parameter #39746 * Developers using Console can employ VT/ANSI APIs (eg to use embedded color codes) #44513 (.NET 6) * Terminal: * MacOS 10.13.6 Console.SetWindowSize() exception #27216 * Terminal hardware is always left in application mode (keypad_xmit) after running a console program #27626 * Console.Clear() doesn't clear the scrollback buffer on Unix-like platforms #28355 * Setting Console.CursorVisible property does not change visibility in Linux #31063 * Cannot use left/right arrow keys to edit text in Console.ReadLine on Unix #38051 * System.Console.Read() needlessly changes terminal settings on linux #49129 * Console issues with multithreaded c# app only when run on Linux #49301 * Encoding: * Changing console's OutputEncoding on linux to unicode generates garbage #29735 * Consider changing the default console encoding on Windows to UTF-16 #31466 * Console.OutputEncoding by default returns a wrong value on Windows Nano Server #42959\ * Console UTF-8 input is misbehaving on Windows #43295 * `Console.CancelKeyPress`: * Third party code may remove cancel keypress handler #30464 * Setting Console.TreatControlCAsInput to true causes text to be removable #44667 * .NET 5 apps can no longer intercept SIGINT signals (receive CancelKeyPress events) when running under Docker #51221 * Other: * FreeBSD: System.Console is not working right #23653 * Console.Write prints a new line on iOS #36440 * Console.In.Peek() always returning EOF after the first read line on Windows #40735 * Console.KeyAvailable throws when input isn't redirected on Windows 7 x86 #43015 * "System.UnauthorizedAccessException: Access to the path is denied." from Console.WriteLine on Ubuntu 20.10 #48654 * Performance: * `Console.OpenStandard*` few times slower on Linux #31396 * New APIs: * We need a cross-platform way to detect key-down state #24314 * Enhance Console.ReadLine() to return at Tab #800 * Add API for detecting broken pipe when write to stdout(or stderr) #30540 * Proposal: Async Console I/O APIs #299 * Test issues: * [mono] Test failed on windows: WindowAndCursorProps.Title_Set_Windows #34454 * System.Console tests failing on iOS #36878 * System.Console.Tests fail on Android #37465 * CancelKeyPressTests.HandlerInvokedForSigQuit test failing on OSX in CI #38998 * Docs: * System.Console: Backport MS Docs documentation to triple slash #48959

Author:	adamsitnik
Assignees:	adamsitnik
Labels:	`Design Discussion`, `area-System.Console`
Milestone:	Future

[adamsitnik]

@mdh1418 @MaximLipnin is there any chance that you could share your experience from https://github.com/dotnet/runtime/pull/41184 https://github.com/dotnet/runtime/pull/50931 ?

[mdh1418]

@adamsitnik The work in #41184 and #50931 was primarily done to leverage the platform compatibility analyzer and warn users when a particular API was unsupported on the specific platforms. I believe we had taken note of which APIs failed with PlatformNotSupportedException when running library tests on Browser-wasm Android and also iOS +tvOS (which also had PlatformNotSupportedException for System.Console APIs, and we proceeded to mark those APIs with the Unsupported attribute

[phillip-haydon]

Microsoft needs to give @patriksvensson a ton of money and replace with spectre.

[HowardvanRooijen]

@phillip-haydon If you notice, Patrik is CC'd on the first message in the thread, because we've been having fortnightly calls for many months on the subject! The point of this thread is to mobilise "an army of the willing" who want to see these improvements invested in, and delivered! My original request was "please can we interop between System.CommandLine and Spectre?" 😁

[phillip-haydon]

I saw, I'm just totally blown away at spectre, been following it on twitter, we need to give Patrik more recognition that he's built something amazing :D (and the contributors to the project too)

[HowardvanRooijen]

100% I've said that Spectre delivers the "high level productivity API", that enables me to build impressive things quickly. It's like having the WebControls / WinForm Controls / WPF Components for the CLI, but there are also the lower-level APIs that are needed to deliver that experience consistently across all the platforms that .NET Supports. Hence my tagline of:

Spectre.Console + System.CommandLine = ❤

[maloo]

Please make sure it works great with https://github.com/migueldeicaza/gui.cs

[adamsitnik]

@migueldeicaza would you like to provide some feedback based on your experience with building https://github.com/migueldeicaza/gui.cs?

[alexrp]

FWIW, I'll throw this into the mix: https://github.com/alexrp/system-terminal

There are some aspects of that library I wouldn't necessarily repeat if I were to start from scratch, but I think it provides a mostly reasonable API surface that could serve as inspiration for a System.Console re-design.

[i-am-shodan]

I maintain dotnet-shell so a lot of the thing I’m doing with Console are less about UI and more about input handling.

Unordered list of problems with Console / features I often require

• async support
• Colour support. I’d like an inbuilt ColorString in the console namespace but Write having an optional color param would also work
• Input handling needs a rethink for Linux. There is a ‘hack’ for Control-C but Control-Z is not possible on Linux without pinvoking.
• Events for when important console parameters change
• Perhaps a separate Terminal implementation where WindowTitle etc can be set would be good. Included implementations for Windows Terminal and POSIX/xterm
• I’d like Console to be an interface that can be easily overridden and replaced for mocking etc
• Consideration / best practice for unit testing console apps.

[xela-trawets]

Tek4014, I told Santa.

[xela-trawets]

Or just a bitmap party space Invite

[migueldeicaza]

I agree that System.Console is too tied to Windows, and there are things that are hard to emulate on Linux (and generally on terminals that rely entirely on a stream of bytes).

The list of challenges above seem to be a blend of limitations in the current implementation, with some design challenges - I am not convinced that they are all design flaws (take for example the Color ones, or the Android/iOS ones that really are not speaking to a real terminal but a log).

It might be useful to compare the bugs above against the Mono version (the official version on Linux, that emulates .NET Framework 4.7) as some of those seems like limitations of the current .NET Core implementation, rather than design flaws.

If I were to design a new system, I think that I would do a few tiers of work:

(a) Put the terminal in raw mode/cooked mode, byte-at-a-time vs line-at-a-time (b) Colors, positioning

Then I would provide a split "WindowsConsole" with all the capabilities that Windows has (there are a bunch of things missing from System.Console, gui.cs shows a few of the things missing), and then also add an "XtermLineageTerminal" where there is a progression of capabilities from the simplest black and white VT100 all the way to modern terminals which support quite a number of capabilities.

The models are different, in Windows, folks can assume a framebuffer exists, while in vt100-derived terminals you must rely on escape sequences to get the job done. In the old days, it made sense for a library to interpose itself between the application and the terminal to optimize which command should be sent based on the changes made to the screen - while there is a mild case to be made for this, nowadays, connections and terminals are fast enough that it does not really matter all that much - so layers like ncurses might be overblown for things like this.

While gui.cs supports using the plain System.Console as a driver, it also has a Windows driver (for access to things like mouse events and additional modifiers) and a curses-driver, for Unix systems.

[BDisp]

While gui.cs supports using the plain System.Console as a driver, it also has a Windows driver (for access to things like mouse events and additional modifiers) and a curses-driver, for Unix systems.

But unfortunately, Windows driver mouse events aren't supported on Windows Terminal. So, maybe the solution is using a combination of WindowsDriver (https://github.com/migueldeicaza/gui.cs/blob/main/Terminal.Gui/ConsoleDrivers/WindowsDriver.cs) and NetDriver (https://github.com/migueldeicaza/gui.cs/blob/main/Terminal.Gui/ConsoleDrivers/NetDriver.cs), as discussed here https://github.com/migueldeicaza/gui.cs/issues/332#issuecomment-821297044.

[chucker]

Possibly slightly OT, but I would like better handling of buffers. It was somewhat unintuitive to me that running a command through System.Console has different buffering behavior (despite BeginOutputReadLine) than a Terminal would.

[ericsampson]

Splitting it up into base/Windows/XTermLineage as Miguel mentioned makes sense to me, similar to how .NET 5 has platform-specific TFMs.

[ericsampson]

• async support would be nice. Writing to console is not exactly fast currently, from what I remember.

[Pxtl]

Standardize a Warning stream. It's frustrating working with Powershell and Azure DevOps automation and whatnot how all of them support the same stdout and stderror outputs but they all implement their own distinct flavor of Warn and they're not compatible.

Even if the implementation of the warning stream on most consoles is done as "color the text orange" instead of doing smart things like warning-specific redirection like Powershell offers, having a standard API for logging warnings would simplify a lot.

I want "Console.Warn".

But that probably requires some OS support that doesn't exist, I assume.

[iSazonov]

From PowerShell experience I opened https://github.com/dotnet/runtime/issues/800 to get support for tab-completions. Also there was mentioned it is desired to have a featured line editor. I don't remember whether an issue exists for TermInfo support. Currently .Net uses TermInfo internally in limited way. It would be nice enhance this, maybe expose publicly (and add Windows Terminal in TermInfo database) so that we could use unified approach for most of terminals.

[jonsequitur]

@iSazonov Tab completions are available today. They're at a higher layer than System.Console. System.CommandLine allows tool authors to configure completions, and end users can enable these using dotnet-suggest. There's been some discussion of merging the latter's functionality into the dotnet CLI and improving the install experience.

[iSazonov]

@jonsequitur Thanks for pointing System.CommandLine but PowerShell uses low level API to implement tab-completions for cmdlets.

[waf]

One pain point for my libraries is CJK (Chinese, Japanese, and Korean) output. In the console, a single CJK character renders as two characters wide, versus English/Latin characters which are only one character wide. If you're trying to do any sort of fancy rendering where you care about character positions and alignment, it becomes pretty complex. If it's not designed for upfront, it's painful to add later on as it breaks a lot of assumptions.

Spectre.Console maintains https://github.com/spectreconsole/wcwidth/ to help retrieve this character width information, another C# implementation is this wcwidth gist.

[iSazonov]

One pain point for my libraries is CJK (Chinese, Japanese, and Korean) output. In the console, a single CJK character renders as two characters wide, versus English/Latin characters which are only one character wide.

Currently PowerShell PSReadline uses simple heuristics and it would be great to have native support the feature in .Net.

[jeffhandley]

Some issues to consider with this effort:

• https://github.com/dotnet/runtime/issues/33980 (which had a draft PR: https://github.com/dotnet/runtime/pull/47935)
• https://github.com/dotnet/runtime/issues/33980

[Pxtl]

Currently -whatif circumvents the console. That's a small bug I'd like to see fixed - get -whatif contents into the #6 info-stream same as write-host

[hamarb123]

Hi, just wanted to say that using Console.SetCursorPosition is terrible if you're trying to go back to a previous position on macOS because whenever the window adds more lines, all of the numbers are offset, unlike on Windows. eg.

var pos = Console.GetCursorPosition();
for (int i = 0; i < 100; i++) Console.WriteLine("Other line");
Console.SetCursorPosition(pos.Item1, pos.Item2);
Console.Write("Alternative text");

On windows this should put the text "Alternative text" on the first line of the output, but on macOS, it puts it at differing locations depending on how scrolled down you got.

[jeffhandley]

52807 is another issue to look at when we explore this space.

[BDisp]

I thought it was just with the ncurses that the horizontal scrolling wheel was not detected on Linux and now I also came across that the same is not detected by System.Console.ReadKey(true), using escape sequences. Is any solution foreseen for this?

Edit: On Windows it's returning the same code for horizontal left or horizontal right, the same as the wheel down. "\u001b[<65;col;row;M"

[deeprobin]

If we do a redesign I think we should leave the old System.Console API for now and create a new one (e.g. System.Terminal) so that we don't have any breaking changes in System.Console.

In this new API we can then modify all the sins we know of in System.Console without hesitation.

[KathleenDollard]

@deeprobin That's the expectation. We're not going to break System.Console. This is to collect info on what the 'sins" of System.Console are.

[hamarb123]

Hi, just wanted to also say that Console.Clear doesn't do what I'd expect on macOS. It should send \u001b[3J\u001b[0;0H (you can test this does what you want using by running printf '\u001b[3J\u001b[0;0H' in the terminal) - this clears the screen and the scroll-buffer and then puts the cursor at the top. Currently it sends something similar to \u001b[2J\u001b[0;0H (not exactly sure exactly what it sends) - this only clears the screen and puts the cursor at the top, it can also scroll down sometimes. I think there is a place for both types of clearing though, but currently we have only the less useful to me. Note: on Windows we only have the full console clearing currently, but I think the other is possible too using special escape sequences (possibly the same ones). Another thing that would be really useful (but I don't know how to do / if it's possible) is have control over the scroll-bar in terminal using an api, because if you go down a line past the end of what's show, the position is still the same according to the api, so having control over both would be great on macOS for restoring positions properly. Unsure if this can be legitimately detected & controlled though. Also, it would be very nice if we could detect dark mode easily on a platform like macOS where it changes all the colours in the terminal; I've implemented a basic version of this in one of my apps and could share the code if it's wanted. Also, I'm sure you're aware, but please keep in mind that there are kind of 3 sets of apis in the Console class: plain text ones (like WriteLine), extra features that all/most terminals have (like colour), features to control the window (like window size) - even some of the last one should be able to work on macOS (and probably Linux if possible), not just the first 2. Thanks!

[deeprobin]

Hi, just wanted to also say that Console.Clear doesn't do what I'd expect on macOS. It should send \u001b[3J\u001b[0;0H (you can test this does what you want using by running printf '\u001b[3J\u001b[0;0H' in the terminal) - this clears the screen and the scroll-buffer and then puts the cursor at the top. Currently it sends something similar to \u001b[2J\u001b[0;0H (not exactly sure exactly what it sends) - this only clears the screen and puts the cursor at the top, it can also scroll down sometimes. I think there is a place for both types of clearing though, but currently we have only the less useful to me. Note: on Windows we only have the full console clearing currently, but I think the other is possible too using special escape sequences (possibly the same ones).

@hamarb123 I think that bug does relate to the current API. Can you create a new Issue for this?

Another thing that would be really useful (but I don't know how to do / if it's possible) is have control over the scroll-bar in terminal using an api, because if you go down a line past the end of what's show, the position is still the same according to the api, so having control over both would be great on macOS for restoring positions properly. Unsure if this can be legitimately detected & controlled though.

I am not sure if this is possible. I guess this is something that belongs to the conhost on windows.

But there are ANSI control sequences for scrolling ^1 (but these are not supported in ANSI.SYS = no DOS-support):

Description	Sequence
Enable scrolling for entire display	<ESC>[r
Enable scrolling from row {start} to row {end}.	<ESC>[{start};{end}r
Scroll display down one line.	<ESC>D
Scroll display up one line.	<ESC>M

[zadjii-msft]

FWIW, no, controlling the scrollbar is not something that's broadly possible with VT/escape sequences. The "scroll up/down" sequences are used for shifting the contents within the active buffer (the "viewport" at the bottom of the scrollback), and account for things like scroll margins, but they don't move the viewport into scrollback.

Not all terminals have support for scrollback. There have been a few attempts across the ecosystem to try and roll one-off sequences for controlling the position of the viewport, but those aren't widely implemented (to the best of my knowledge).

[hamarb123]

Thanks, I tried some of those sequences and none of them scrolled back into the scroll-back on macOS. I will make an issue for the macOS clear function as well - thanks.

[webczat]

Well what I noticed was mostly things like Read/ReadKey. windows is the os where you can just ReadKey where on unixes it requires switching back and forth between terminal canonical mode, or what is done here, just staying in non canonical mode, so a dotnet console app, even a simple one reading lines, doesn't kinda feel native. Even though the fact itself that you can just ReadKey without doing os specific things is also useful. I am wondering, if a new api in any shape appears, will Console still be shown by default for simple things like WriteLine?

[hamarb123]

doesn't kinda feel native

That's another good point @webczat, something like ReadLine doesn't even let you press left and right to move the cursor - I've not seen any other console utility that does this on macOS (except things like vi where it's intentional in certain contexts) - also this doesn't mean they don't exist, I just haven't experienced them.

[webczat]

doesn't kinda feel native

That's another good point @webczat, something like ReadLine doesn't even let you press left and right to move the cursor - I've not seen any other console utility that does this on macOS (except things like vi where it's intentional in certain contexts) - also this doesn't mean they don't exist, I just haven't experienced them.

Well generally it's exactly the canonical vs non canonical mode. in canonical mode (default) dotnet is not supposed to handle things character by character, just ReadLine should read from terminal until newline, as only full lines are sent to it. In case of non canonical each character is sent and everything including ^u and ^k keystrokes and other such things, backspace etc needs to be directly handled by the program. I am on linux BTW.

[hamarb123]

@hamarb123 I think that bug does relate to the current API. Can you create a new Issue for this?

https://github.com/dotnet/runtime/issues/28355 - seems like I've commented here before also

[BDisp]

I'm having the follow error when I resize the console height to zero or almost zero and pointing the mouse to the available console space:

System.InvalidOperationException: 'Cannot read keys when either application does not have a console or when console input has been redirected. Try Console.Read.'

This is happening on Windows and WSL using cmd, pwsh and conhost. As I'm reading escape sequences from Console.ReadKey (true) in a console with a height equal to zero causes the System.Console.dll crash. The problem is I can't force resizing the console. Do I still need to using for Windows the Win32 API Console by enabling ENABLE_VIRTUAL_TERMINAL_PROCESSING to be able to use escape sequences or does System.Console allows that feature? Another question is many escape sequences request not available in Windows and WSL, is there any configuration I can use or is limitation? To test this issue please clone the https://github.com/gui-cs/Terminal.Gui repo and run the UICatalog project with the argument -usc, which will use the NetDriver using the System.Console. Thanks.

[adamsitnik]

@BDisp please create a new issue, this thread is dedicated to Console re-design