Console filters out bracketed paste input codes when virtual terminal input is disabled

Windows Terminal version

1.22.2702.0

Windows build number

Other Software

This issue applies broadly to all WIN32 software. In my case, I happen to be working on Cosmopolitan Libc.

Steps to reproduce

Use SetConsoleMode() to disable the ENABLE_VIRTUAL_TERMINAL_INPUT flag on GetStdHandle(STD_INPUT_HANDLE).
Use SetConsoleMode() to enable the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag on GetStdHandle(STD_OUTPUT_HANDLE).
Write the string "\033[?2004h" to standard output, to enable bracketed paste mode.
Press ctrl-shift-v or shift-ins to paste text with multiple lines into the terminal.
Reading from standard input will give you the pasted text (but it's supposed to surround the paste with "\033[200~" and "\033[201~")

If you're a Cosmopolitan Libc fan and have cosmocc installed from our GitHub releases page, then here's a simple program that can reproduce this without needing to write any WIN32 code.

void Write(const char *s) {
  write(1, s, strlen(s));
}

int main(int argc, char *argv[]) {
  Write("\033[?2004h");
  char buf[512];
  int rc = read(0, buf, 512);
  Write("\033[?2004l");
  printf("got %`'.*s\n", rc, buf);
}

See also https://github.com/jart/cosmopolitan/blob/master/libc/calls/read-nt.c if you're curious about our termios driver.

Expected Behavior

If I enable bracketed paste mode on the terminal, then I believe it should insert the "\033[200~" and "\033[201~" ANSI codes into the paste, even if ENABLE_VIRTUAL_TERMINAL_INPUT is disabled. Cosmopolitan Libc needs this. Cosmo is similar to Cygwin in that it implements POSIX on Windows. For example, common utilities like bash and emacs can be compiled with cosmo libc and then run in the command prompt. I've been very successful making this work well. This bug is the one problem I can't solve unless Microsoft fixes things. It is impossible to work around this issue.

Why is that? Cosmopolitan Libc always wants a virtual ANSI style Linux-like terminal. To do that, we always enable ENABLE_VIRTUAL_TERMINAL_PROCESSING on standard output. Virtual terminal output works fantastically with your tools. However Cosmo never uses ENABLE_VIRTUAL_TERMINAL_INPUT because it isn't very good. Virtual terminal input is a toy feature that makes important use cases impossible, such as polling console input. Cosmopolitan Libc always uses ReadConsoleInput() and GetNumberOfConsoleInputEvents() to read console input and then manually translates INPUT_RECORD to ANSI codes and implements things like canonical mode line editing because there's no other way.

However there's one and only one occasion where I actually DO want WIN32 to inject \033[200~ and \033[201~ ANSI codes into my input, and that's for bracketed paste mode. Windows should either do this, or it should generate INPUT_RECORD events that tell me where the paste begins and ends. But Windows does neither and that's a huge painful issue for me.

Actual Behavior

Windows makes it impossible to obtain the the \033[200~ and \033[201~ ANSI codes that tell me where pasted text begins and ends, unless ENABLE_VIRTUAL_TERMINAL_INPUT is enabled. It's not possible for me to enable this flag, because that flag is problematic and removes win32 capabilities when it's enabled, such as the ability to poll console input. If I've put the terminal into bracketed paste mode by writing the virtual ANSI sequences to the output handle, which does have virtual mode enabled, then the console should not filter out and remove those ANSI codes when they get sent to my input. The ENABLE_VIRTUAL_TERMINAL_INPUT flag presence should be ignored in this specific case.

Hey there! I'm glad to see you here--Cosmopolitan is such a cool project.

I'll gladly pore over the repro. I do want to get a couple things out of the way, though.

The console subsystem's stance is that applications get what they request--up to a point--and increasingly do not get what they do not expect[^1]. I bring this up only because VT-encoded input is one of those things an application has to tell us they're expecting.

Right now, the only exception we have for giving applications VT input even if they haven't requested it is in response to VT report requests. Applications requesting reports on their own output handle are probably--despite not setting ENABLE_VIRTUAL_TERMINAL_INPUT--expecting reports on their input.

Bracketed paste is not like VT reports, though. The requesting application may no longer be running by the time the user pastes content. In that way, it's similar to DECKPAM, DECCKM, S8T1C, Xterm mouse reporting and others.

In short, I am hesitant to give VT-encoded input to applications that haven't requested it.

Now, that brings me to the other thing.

Virtual terminal input is a toy feature ...

☹

We have legitimately not heard this one before, despite having been aggregating issues in the public roughly since it launched in ~2017. I'm sorry to hear that you feel that way.

To the best of my team's understanding, it should still work with polling. I would much rather root cause and resolve that, because PeekConsoleInputW and GetNumberOfConsoleInputEvents are APIs with some concrete guarantees around them.

There are issues, for sure, in VT input. It sucks for some pretty common use cases, like figuring out when the window has resized or when focus was lost or gained. But it still does work for those use cases.

Applications and support libraries have largely coalesced around still using ReadConsoleInputW even when using ENABLE_VIRTUAL_TERMINAL_INPUT. There's a direct translation from escape sequences into INPUT_RECORDs, and you don't actually need to use ReadFile or ReadConsoleW.

Cygwin does it after setting VT_INPUT (MSYS2 as well); WSL does it (I'd link the source if I could :eyes:); Win32-OpenSSH does as well (after having moved from their own internal INPUT_RECORD translator!); all of our other adapters do as well.

The nuance--and it is a nuance I believe is worth dealing with (I'll explain below)--is that:

Window resize and focus events come in as usual (not as dtterm window manipulation or xterm focus sequences)
VT input comes in the key events where bKeyDown is true
Unicode input comes in subsequent surrogate pairs in uChar (as with non-VT input) when bKeyDown is false[^3]

I tried to pull all of this behavior out into a reusable (C++, but I will admit it is not well-written) input handler as documentation. That lives here.

But why trust the Console subsystem at all?

... manually translates INPUT_RECORD to ANSI codes ...

VT_INPUT supports the following DEC-compatible input modes, as well as a couple Xterm-specific ones:

LNM to toggle whether Return produces LF or CRLF
DECANM to toggle VT52 input coding
DECARM to toggle key autorepeat
DECKPAM and DECKPNM to control "keypad" mode
DECCKM to control the encoding of cursor keys
DECBKM to control the encoding of Backspace as \x08 or \x7f (and Ctrl+Backspace)
S7C1T and S8C1T to control whether C1 controls are used in the input stream
Xterm mouse modes UTF-8 and SGR, as well as the three tracking modes (button, any event, default)
Xterm alternate scroll mode, which controls whether mouse wheel events are transformed into cursor up/down in the alternate screen buffer.

To adequately cover the needs of applications that port using Cosmopolitan while still maintaining your own translations, you'll probably end up writing a terminal emulator for your output handle... just like Cygwin and OpenSSH used to, and actively moved away from. :smile:

[^1]: I have long-term plans to make the input and output modes process-specific but still heritable; this work is not yet done, but will free us from some significant Windows-specific burdens[^2]

[^2]: Making the input mode process-specific doesn't help with VT input modes like DECCKM or DECKPAM or even xterm's mouse reporting modes, but... small steps.

[^3]: It's nearly 1AM, I actually don't recall why this is. I'll investigate in the morning. I mean, the "sun is up" kind of morning.

microsoft / terminal