ConPTY Passthrough mode

[be5invis]

Modern apps won’t read the hidden character grid and do everything in VT. So why not an API/console mode to tell Console Host to completely throw away that?

[BrianInglis]

They wouldn't even notice in pure pass-through applications. It might be necessary to solve an otherwise unresolvable dilemma.

Just as an FYI mintty is a port of putty to Cygwin using Windows API, which has added VT graphics and xterm extensions including Tek4010 and DEC ReGis sixel support and probably others.

[christianparpart]

They wouldn't even notice in pure pass-through applications. It might be necessary to solve an otherwise unresolvable dilemma.

Just as an FYI mintty is a port of putty to Cygwin using Windows API, which has added VT graphics and xterm extensions including Tek4010 and DEC ReGis sixel support and probably others.

What does this FYI add to the topic of ConPTY passthrough? Putty (and forks) work as ssh(/telnet/other-lines) clients and do not handle local processes, unless something has changed since I last used these. As such application you have 100% control over the communication line and it's easy to add things to hat ConPTY does not support. This is incomparable.

[BrianInglis]

They wouldn't even notice in pure pass-through applications. It might be necessary to solve an otherwise unresolvable dilemma.

Just as an FYI mintty is a port of putty to Cygwin using Windows API, which has added VT graphics and xterm extensions including Tek4010 and DEC ReGis sixel support and probably others.

What does this FYI add to the topic of ConPTY passthrough? Putty (and forks) work as ssh(/telnet/other-lines) clients and do not handle local processes, unless something has changed since I last used these. As such application you have 100% control over the communication line and it's easy to add things to hat ConPTY does not support. This is incomparable.

The mintty port talks to the conhost via Cygwin pty and Cygwin console drivers, or WSL console interfaces using wslbridge-{cygwin-frontend,linux-backend}, and uses Windows text/graphics rendering APIs with charset conversion where required, so transparent passthru supporting alternative non-"ANSI"/xterm-compatible graphic terminal modes are desirable. Screen/Tmux/SSH/X/Windows console (including cmd /c commands and batch/cmd and PowerShell scripts) and GUI apps may also be run (launched for Windows GUI and locally or remotely for X) from the console(s) in the mintty terminal window(s).

[christianparpart]

Thanks, @BrianInglis :)

[csdvrx]

They wouldn't even notice in pure pass-through applications. It might be necessary to solve an otherwise unresolvable dilemma.

Just as an FYI mintty is a port of putty to Cygwin using Windows API, which has added VT graphics and xterm extensions including Tek4010 and DEC ReGis sixel support and probably others.

mintty is no longer just a simple port of putty: in practice, these various additions make it better than xterm for most common day uses

[zadjii-msft]

Note to self: Although this was experimentally added a while ago, I think Niksa's intention was to keep both this thread and #10001 open in parallel. There's a LONG way to go before experimental.connection.passthroughMode is ready for daily driving, and I'm not sure if we've got a good mental framework for figuring out if that will be the satisfactory conclusion to this thread (#1173)

[mintty]

I have just submitted microsoft/WSL#9117 as I felt for the main use case of "passthrough mode", Windows Terminal is the wrong domain in the first place. There should be a decoupled WSL launcher, like wsl.exe but without any bundled Windows Terminal functionality, in order to support full VT-like connectivity.

[DHowett]

This is also the repository for the Windows Console Subsystem, and given the breadth of applications that want to run on Windows and emit/receive plain VT it is entirely within our charter.

[palves]

Hi! I got here because I've been working on running the GDB (GNU Debugger) testsuite on Windows. The GDB testsuite uses DejaGnu, which is based on Expect. My plan is to have DejaGnu ssh into the Windows machine, and run GDB there. This is something that has been working fine on Linux for decades. On Windows, with the new Virtual Console support, it sounds like exactly what we'd need to make it work on Windows as well, and be able to auto test GDB in an environment that behaves like a regular console environment. However, the fact that ConPTY is not passthrough makes it still unworkable, unfortunately. Let me explain a little.

For example, this is the output DejaGnu sees when connected to GDB on Linux (with TERM=dumb):

...
For help, type "help".\r\n
Type "apropos word" to search for commands related to "word".\r\n
(gdb) set height 0\r\n
(gdb) 

And this is what DejaGnu sees when connected to GDB on Windows (via Windows 10's built-in SSH)

 ...
 $ c:/gdb/build-windows/gdb/.libs/gdb.exe -q\r\n
 (gdb) s\r(gdb) set height 0\r\n
 (gdb) \x1b[12X

The odd line redraw in the first "(gdb)" line, and the extra escape sequences after the second prompt make this totally unworkable, because the testsuite is built on matching GDB output, and it isn't expecting any redraws or escape sequences (unless GDB sent them, e.g., for color, in which case the test will match them).

I am hoping that a passthrough mode would make the Windows output more like Linux one.

Can someone let me know what is the current status of passthrough mode, and how would one enable it? Is there some undocumented flag I can make GDB pass to SetConsoleMode, perhaps?

[lhecker]

Windows Terminal Preview has a hidden setting for this. If you open the settings.json file via the "Open JSON file" button in the settings tab, you can add this:

"experimental.connection.passthroughMode": true,

to any of your profiles including the "defaults". Afterwards you'll have to create a new instance/tab of that profile. If you use PowerShell you'll notice fairly quickly that it works, because output will be severely broken.

Passthrough mode is something that must be done at some point or another (the "rendering" that ConPTY does is effectively a hack and would require constant maintenance forever if we kept it), but right now it's not anywhere near useable yet. Its implementation is simply incomplete, and its current approach might not even be the right one. I personally hope that we could tackle that issue soon, but realistically (and frankly) speaking we're not staffed enough to do that in the near term. There's a significant interest in other features like session restore, Unicode, image support, "sudo", font fallback, tmux, and many many more and it's not quite clear what issue deserves attention the most. Put differently, while I'm personally interested in solving this, the reality is that this issue and any of those that would be fixed by it don't even show up in the first 2 pages sorted by 👍. To summarize: I'm sure we'll get to it, but just not right now. Sorry for the ramblings... 🙁

[BrianInglis]

You could consider Cygwin project mintty/wsltty originally derived from putty with serial support converted to Cygwin pty interfacing with Windows ConHost/ConPTY (or WSL over wslbridge/wslbridge2) running Cygwin and most Windows terminal shells. Its adaptations and many configuration settings plus support for Cygwin screen/tmux and ssh may allow you to do what you need. The current developer @mintty is responsive and helpful.

[DHowett]

I am hoping that a passthrough mode would make the Windows output more like Linux one.

At the core of this issue is a misunderstanding--on the part of the Win32-OpenSSH folks--of what it means to request a tty.

Even on Linux (supersets and derivatives), spawning GDB or any other application with a controlling terminal runs the risk of that application deciding that it should emit control sequences. This often happens even when TERM is set to dumb. A bunch of applications just check isatty(3) rather than consulting TERM!

On Windows, the equivalent of spawning a controlling terminal is establishing a pseudoconsole/console hosting session. sshd on Windows requests a console hosting session even where one would not be appropriate.

sshd on Windows should not be allocating a pty to host things for which sshd on Linux/BSD would not allocate a pty[^1]. That would get ConPTY and its rendering logic entirely out of the loop, and give you stdin/stdout from the spawned process directly and unmodified.

[^1]: such as when ssh is used in a pipeline or not itself connected to a tty/pty :)

[palves]

sshd on Windows should not be allocating a pty to host things for which sshd on Linux/BSD would not allocate a pty. That would get ConPTY and its rendering logic entirely out of the loop, and give you stdin/stdout from the spawned process directly and unmodified.

That is essentially what you get with "ssh -T", but it's not what I need -- I really need for GDB to be connected to a pty, not just a plain pipe, so that input works as if on a terminal, like cursor keys, and sending ctrl-c.

[DHowett]

Interesting. That makes sense; thanks for explaining it :)

[palves]

Windows Terminal Preview has a hidden setting for this. If you open the settings.json file via the "Open JSON file" button in the settings tab, you can add this:

"experimental.connection.passthroughMode": true,

Ah, thanks. I read https://github.com/microsoft/terminal/pull/11264/files and saw that that is plumbed down to a PSEUDOCONSOLE_PASSTHROUGH_MODE flag passed to the Virtual Console creation. Since I need it with ssh, I guess sshd would have to have its own way to enable this.

So it is not possible to activate the passthrough mode from the command line application itself, like how you pass ENABLE_VIRTUAL_TERMINAL_PROCESSING to SetConsoleMode ? I.e., something like a ENABLE_VIRTUAL_TERMINAL_PASSTHROUGH flag GDB itself could enable in SetConsoleMode? Is that something that you envision would be possible if/when this work is eventually completed?

[BatmanAoD]

Other terminals, such as Wezterm, also need a way to enable this option.

[mintty]

Other terminals also need a way to enable this option.

Yes, and that makes me repeat that this is actually not a Microsoft Terminal issue but a feature needed for the WSL launcher, wsl.exe, which should have a decoupled mode to not capture any terminal interaction; see https://github.com/microsoft/WSL/issues/9117

[j4james]

@mintty As has already been mentioned above, this is also the repository for the console subsystem, which underlies all console applications - that includes both Windows applications and WSL applications. And the whole point of conpty is to translate the Windows console APIs into VT sequences. WSL has got nothing to do with that. Windows Terminal has got nothing to do with that either, but it happens to share the same repository.

@BatmanAoD It's worth noting that any terminals that want to use this option would likely need to work on their VT support first. To translate Windows console APIs into equivalent VT sequences without a local buffer (which is a requirement for pass-through mode), we'll likely need VT features like rectangular area operations, which aren't that widely supported.

[mintty]

the whole point of conpty is to translate the Windows console APIs into VT sequences. WSL has got nothing to do with that.

Yes, but wsl.exe does, unfortunately, as it intercepts and filters terminal interaction. It shouldn't, in a passthrough mode, that's my point, not really caring about the underlying architecture.

[ferdinandyb]

Put differently, while I'm personally interested in solving this, the reality is that this issue and any of those that would be fixed by it don't even show up in the first 2 pages sorted by 👍. To summarize: I'm sure we'll get to it, but just not right now.

I've been browsing issues and wanted to point out, that sorted by 👍, the third issue is sixels where @j4james said, that

We either need the full passthrough mode working, or for the conpty renderer to be capable of regenerating the sixel on the fly so it can repaint areas of the screen that have been invalidated. Of the two, passthrough mode seems more feasible.

so it seems to me that there's at least one high profile issue that would be solved by this, and I wouldn't be surprised if there were more highly voted issues where this would be the solution (I'm guessing a lot of people are like me: not knowing much about terminal internals, just wanting a WSL terminal to feel like they are on a linux box).

[BrianInglis]

xterm implements almost all of the 700+ documented ECMA, VT, ANSI, and X control sequences, and is available under MIT/X permissive licences: so just "Use the Source Luke!"

[jerch]

We either need the full passthrough mode working, or for the conpty renderer to be capable of regenerating the sixel on the fly so it can repaint areas of the screen that have been invalidated. Of the two, passthrough mode seems more feasible.

I also think that the sixel-regenerating is a dead-end in the long run creating lots of maintenance burden (you'd basically need an "image recorder" with decoding and encodings clipping viewport areas, geez) while only solving this particular sixel issue. What about other image sequences later on? What about other sequences in general? They will still fail at the "not implemented here in ConPty" trap. On a sidenote - recently had a similar image forth and back encoding/decoding need for state serialization within xtermjs, and did not vote for sixel, but QOI (reasonably fast in wasm with good enough compression - much better than any PNG handling offered by the browser engines).

This prolly got stressed several times already - to me it seems that ConPty "puts the cart before the horse" with its interim terminal emulation to stay compatible to the far less capable WIN-console API. Wouldn't it be better to do it the other way around and to abstract the console caps into rectangular VT sequences? Yes, support is quite lousy for those across TEs, but even with scroll margins only one can simulate many aspects. I also think that TE devs would happily implement better rect support, if there is a serious need for those.

[j4james]

it seems that ConPty "puts the cart before the horse" with its interim terminal emulation to stay compatible to the far less capable WIN-console API. Wouldn't it be better to do it the other way around and to abstract the console caps into rectangular VT sequences?

@jerch I've been pushing for this for a while, and I think it's probably the right long term solution, but the current conpty approach was most definitely the correct decision at the time, and it may well remain the most practical solution for a long time.

The Windows console API has a lot more functionality than your average VT terminal, and even with the most advanced VT capabilities there will likely be things we just can't reproduce. To be honest, the more time I spend looking at this, the less convinced I am that it's worth the effort.

[mintty]

Windows console API has a lot more functionality

You mean like reading back the screen contents? An exotic feature rarely useful and certainly not relevant in the WSL domain.

less convinced I am that it's worth the effort

That's why I had tried to drag this discussion over to a WSL issue. For the Windows Terminal application, it may be dispensible to provide terminal transparency as it's a terminal itself. For WSL however, there is serious need for transparent terminal access, both local and remote, so passthrough mode without any Windows console legacy burden is essential for the WSL launcher.

[christianparpart]

The Windows console API has a lot more functionality than your average VT terminal, and even with the most advanced VT capabilities there will likely be things we just can't reproduce

@j4james hey, I am curious to know what Windows console API offers that doesn't exist as VT sequence or extension just yet. OTOH, I think if one can count the number of features that are available on the conhost side but not on the VT side, it might make sense to introduce VT sequence extensions for those few and still put conhost on top of VT. I do not want to mandate here anything, I am just curious to know what conhost is more advanced in. :)

On the "your average VT terminal" argument, I'd love to bring some of my main features of contour to Windows, which I can't until either ConPTY supports them, or a passthrough mode is available. currently my windows version of Contour will always be inferior than on other platforms, sadly. Not sure how to solve this without ConPTY passthrough in the future :)

[jerch]

@christianparpart I think the buffer read caps are on the annoying side of things - as far as I remember console API allows complete buffer access (in the sense, that the console app "owns" the console and thus the buffer state). Thats almost impossible with non rect-based VT mechanics or at least will be limited to active cursor area only prolly on all TEs. I still think that most buffer access primitives could be mimicked in VT. The hard unsolvable problem might be the exclusiveness, that console API kinda provides to apps with a quite strict process IO coupling/isolation - a thing that many ppl also wish for terminals in linux/macos, but it is just not possible there with the current TTY/PTY abstraction (though its not a matter of VT mechanics but POSIX terminal API).

[j4james]

@j4james hey, I am curious to know what Windows console API offers that doesn't exist as VT sequence or extension just yet.

You can find a list of the console APIs here: https://learn.microsoft.com/en-us/windows/console/console-functions

If you click through each of them, you should see a "Tip" section which explains which of them do or don't have a VT equivalent. If you think you can convert all of them into VT sequences, please do feel free to contribute PRs. A lot of the framework is already in place to do this.

it might make sense to introduce VT sequence extensions for those few

We aren't even at the point where most conpty terminals are supporting the standard VT sequences we would need. The chances of everyone agreeing to a bunch of Windows-specific extensions on top of that doesn't seem very likely.

I think what some of you really want is just a pass through mode for WSL. If that's the case, you can assumedly build your terminal as a Linux GUI app running on WSL. But ConPTY is specially for Windows console application support. If you aren't interested in that, this isn't the issue for you.

[palves]

There is a set of Windows console applications that want the feature (in my case, GDB (GNU Debugger) testing), which don't need anything from the console API that the supported subset of standard VT sequences doesn't already provide. IMO, there could be a console API that such applications could call to enable some restrictive/subset of the API, only. Call it "passthrough mode", or "no intermediate buffer mode" or some such, and when that mode is active, the problematic console APIs (like complete buffer access) would just fail with an error.

[zadjii-msft]

FWIW that's my preferred approach to solving this. A console mode that a client app can enable to say "I solemnly swear I am up to good". Something like WSL would opt-in, because it knows it's only ever going to do VT.

That doesn't work as well for mixed applications that want to do both. But that would let legacy console apps rely on conpty's regeneration for their own needs, and modern apps just use VT and all the new features that come with it.

[j4james]

If that's the way we want to go, that's fine, but it means the Windows shells don't gain any benefit from it. For example, you wouldn't be able to TYPE a sixel file, or ECHO a Contour-specific escape sequence and expect it to work. Personally I'd consider this a waste of time, but I can accept that's all some people might care about.

[jerch]

To avoid rewriting console API into wonky VT pendants, wouldn't the following work:

• make stream-IO/passthrough VT-mode default
• additionally make legacy console API fully abstract on both sides (process & TE side)
• a TE can decide whether to implement the legacy console primitives, if it does - processes can call into console API as before, if not - processes may not call into legacy console (may need special process flags?)
• TE operates as single source of truth for console buffer primitves - its the TEs resp. to get data from its internal terminal buffer feeding the TE-side of the console API
• provide a standard TE on OS side fully implementing legacy console primitives, so things like AttachConsole still works
• last step - retire ConPty :smiley_cat:

Ofc there are details to overcome, like the question whether the console primitives can be made blocking or need true memory sharing & possible security implications.

[csdvrx]

On a sidenote - recently had a similar image forth and back encoding/decoding need for state serialization within xtermjs, and did not vote for sixel, but QOI (reasonably fast in wasm with good enough compression - much better than any PNG handling offered by the browser engines).

@jerch , am I right to say that we had a few standards already existing (sixel, kitty, ...) and now we've got one more?

I think it's just sad that the desire for technical perfections goes before practical concerns for something that'd work about everywhere, if leveraging the existing ecosystem.

So like @j4james, I think not being able to reach some baseline level of functionality is waste of time:

If that's the way we want to go, that's fine, but it means the Windows shells don't gain any benefit from it. For example, you wouldn't be able to TYPE a sixel file, or ECHO a Contour-specific escape sequence and expect it to work. Personally I'd consider this a waste of time

Actually, I'd consider that not just a waste of time, but a sad waste of time.

[jerch]

am I right to say that we had a few standards already existing (sixel, kitty, ...) and now we've got one more?

Nope, that serialization format is for TE internal usage only, not intended for outside with an explicit sequence. QOI yields the best compression/performance ratio covering RGBA in 32bit. Since xtermjs is browser based, we cannot just store raw bytes somewhere on the disk for session restore, as other desktop TEs would do here.

[mominshaikhdevs]

what is the latest update on this?

[zadjii-msft]

Nothing to share at this time. We'll make sure to update this thread when there is. In the meantime, might I recommend the Subscribe button? That way you'll be notified of any updates to this thread, without needlessly pinging everyone on this thread ☺️