Arrows ignored by bash when it started in ConEmu

[Maximus5]

I'm the author of ConEmu - alternative terminal for Windows. My users have created the issue Maximus5/ConEmu#629. When bash is started in ConEmu console, it totally ignores all extended keys (Arrows, Fn, etc.), so users can't access command line history with Up/Down arrows.

What is the problem with bash? Up/Down arrows properly cycle cmd.exe history, when it's started in ConEmu, but bash ignores them.

ConEmu is GUI application, so it get keyboard events via WM_KEYUP/WM_KEYDOWN and forward them to the created (usually hidden) real console using WriteConsoleInput.

[Serubin]

@Tyriar You're right. Thanks!

[pinonpierre]

Any news about this issue ? It's very annoying !

It will be the Bug Anniversary soon !

[khards]

Weekly bump the bug bump

[zadjii-msft]

Okay so just to clarify, nothing has particularly changed since @miniksa's post from July 14, 2016:

[we] overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix. I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

Unfortunately, we have a lot of other things that are a lot higher priority than fixing this. Rest assured, when there's an update on our end, we'll post here. That way we can avoid sending weekly pings to the 27 people on this thread (and quite possibly more watching)

[bgshacklett]

@jackchammons For those of us not in the know, could you give a quick summary of why this is no-longer considered a bug?

[zadjii-msft]

@bgshacklett Right now it's being tracked internally as work item MSFT:7640020. It's a little bit more than just a simple "bug" fix - this is more "feature" level work. Internally, the distinction makes quite a bit of difference on how we're able to plan work. This should have the feature label now to make sure that this is labeled appropriately.

[nigurr]

@zadjii-msft Thanks for the update. it would be great if you can assign the milestone or tentative timeline for shipping this

[zadjii-msft]

@nigurr Unfortunately I don't have a tentative timeline for this fix at the moment, sorry.

[jackchammons]

@bgshacklett Just doing a little house keeping on the GH labels. It's still an issue that we plan on addressing but it's being tracked internally as a feature. Semantics, I know, but it helps us keep things organized.

[tobico]

Please consider prioritizing the fix for this. Having a wide variety of quality Terminal applications available is very important for WSL to be a useful feature of Windows.

[DropsOfZut]

solved?

[coolxxy]

Please consider fixing this. In ConsoleZ(another third-party terminator), Arrows doesn't work too. and The reason for using third-party terminator is for the terrible font setting for CMD and PowerShell in Chinese edition.

[i336]

That's a very good point - the new console subsystem needs to have really, really good Unicode support. Modern programming languages support unicode strings and expect to be able to use it for I/O, and there are lots of prompt hacks using unicode characters as well.

I figured I might as well ride the bit of traffic that's been generated right now to chime in and add that the community remains super interested in a series of blog posts on the architectural updates. An in-depth writeup that adds every last detail that's legally possible would be widely appreciated - both because architectural upgrade trivia is always really fun, and also because it'll clearly explain why everything took so long.

I personally completely understand that this bugfix is the tail end of a lot of planets aligning being strong-armed into place, and that it'll take a little while. The console subsystem probably still has design cruft in it from Windows 3.1 - the NTVDM code doesn't ship in most (64-bit) installations, but is still builds for 32-bit, so it's still in the codebase. Knowing Microsoft's obsession with backward-compatibility, fixing this will not be easy.

For people frustrated about this, I have an open question (I don't have any Windows-capable hardware at the moment, so I unfortunately can't try this out myself): Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL? If it can, that could be a very workable alternative.

For the MS devs working on this issue, when you see this update - I understand that speculating on timelines creates lots of avoidable fallout, so I won't ask for that. What I will ask for is a vague idea of the status of the internal tracking bug. How far have 7640020 and/or its dependencies/blockers progressed since the bug was initially filed (around August last year)?

I'm not trying to extricate an executive "I'm impatient, hurry up" summary. This is not ready yet, not much to do except face that. That's fine. But to know that there has been some movement would be cool.

Ah, if only you could share internal details. Then you could keep us in the loop about "the architecture, the customers, the issues, and the dependencies" :D and what the progress is with each. We wouldn't mind <3

[rprichard]

FWIW, winpty works around this issue by detecting when the console's ENABLE_VIRTUAL_TERMINAL_INPUT flag is enabled and, when it is, switching to an alternate system for generating console input:

• For most input, winpty replaces the keydown INPUT_RECORD with records representing the VT/100 encoding of the key. (See https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L722.)

• For navigation keys (arrows keys, Home/End), when no modifier was used (i.e. no Alt/Shift/Ctrl), winpty instead sends keydown/keyup window messages. This special case is necessary for Midnight Commander and probably vim too. (See https://github.com/rprichard/winpty/issues/90 and https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L684.)

Unfortunately, sending a window message doesn't work quite right, because the resulting console input records seem to use modifier state from the keyboard. winpty would like to synthesize the modifier state, but I don't think there's a way to do that using SendMessage. This issue also affects Ctrl-C handling, which is a more noticeable problem with winpty. (See https://github.com/rprichard/winpty/issues/116 and https://github.com/rprichard/winpty/blob/ce9239af5d751195ea6982a41c7d71356ac9201c/src/agent/ConsoleInput.cc#L361.)

[rprichard]

AFAIK, other console emulators on Windows (e.g. ConEmu and ConsoleZ) ought to be able to work around this issue the same way that winpty does.

@i336

Can urxvt or xterm from Cygwin running on a Windows X server talk to WSL?

AFAIK, one could run WSL urxvt or xterm connecting to any Windows X server, including Cygwin's. One could also run any Cygwin terminal emulator (including mintty, urxvt, xterm) and connect to WSL using https://github.com/rprichard/wslbridge (a program I wrote). wslbridge is also shipped as part of https://github.com/mintty/wsltty, a standalone package bundling mintty, wslbridge, and a Cygwin runtime DLL.

[Maximus5]

FYI ConEmu has implemented a workaround long long ago

[fpqc]

@Maximus5 I once got the msys2 bridge + wslbridge working together in tandem (without MinTTY), but it stopped working after like an hour. Have you done any testing with it?

[Maximus5]

Of course. It's working. Just use latest ConEmu version.

[zadjii-msft]

Hey everyone,

I finally have an update on this bug! I have a potential fix in code review right now.

To summarize the problem: We were only translating KEY_EVENTs into appropriate VT sequences when the events arrived from the window message loop. I moved the translation layer around, and now the INPUT_RECORDs written to the input buffer from the API (WriteConsoleInput) will also get translated appropriately.

It's not going to be available in the Fall Creator's Update, but it should be in the release that follows.

Thanks for your patience!

[jarun]

Good news. :+1: I just wasted an hour figuring out the issue and reached here.

[rprichard]

@zadjii-msft Good to hear. A few questions:

1. Does this also affect recognition of Ctrl-C or Ctrl-Break? That is, when ENABLE_PROCESSED_INPUT is on, will WriteConsoleInput send a signal for these key presses? It currently doesn't, and I'm guessing that won't change. (If it did, I might be able to fix https://github.com/rprichard/winpty/issues/116.)

2. Is mouse input affected? There's an analogous situation where mouse input should produce VT escape sequences, and the encoding is controlled by mouse mode flags (e.g. 1006 for the "SGR mouse encoding"). I haven't gotten around to testing the console yet, so maybe WriteConsoleInput calls already do this conversion.

3. Do you have suggestions on how programs can detect this behavior change (i.e. so that a workaround can be disabled)? For my situation (winpty), I think it'd be worth it to temporarily enable the VT input mode and test whether input is converted (by calling WriteConsoleInput then ReadConsoleInput). I'm guessing there isn't a better approach, though I suppose checking the OS version could be faster.

[ericblade]

That is great news (not so great that it won't make Fall.. oh well :-) ) .. just curious if this means that it ended up being significantly less work than initially expected? :-D

[zadjii-msft]

@rprichard

1. I don't believe it will. The change is pretty limited in scope to just the VT translator, so I don't believe any of the rest of the input is modified.
2. Mouse is unfortunately not fixed by this change. The keyboard events were easier to move than the mouse events unfortunately. I have a work item to address that issue before the end of the next release.
3. I don't have any good recommendations on that topic, sorry. Both the mechanisms you describe sound like they should work, though I have no idea which would be faster.

It was a bit easier than anticipated, but like I mentioned, mouse input is actually harder than anticipated unfortunately.

[bgshacklett]

Sorry, guys, I know you're being bombarded, but can you tell us how being on the Windows Insider fast track might affect the speed at which we see some of these changes?

[zadjii-msft]

@bgshacklett If you're on the fast ring, you'll see these changes pretty immediately after they hit winmain, which is usually a month or so after I commit the code on my end. Otherwise, if you're on just a non-insider's build, you'll have to wait for the entire release to be finished, which is about twice a year (based on previous Windows 10 releases)

[bitcrazed]

Thanks for the discussion.

Closing this issue since:

1. This issue is now fixed
2. This is the WSL issues repo, but this is an issue in Console which has its own Console GitHub Repo
3. GitHub doesn't allow issues to be moved between repos, preserving posters' identity :(

If you have further asks/issues, please file new issues on our Console GitHub Repo.

[maietta]

I DO NOT CARE WHAT YOU PEOPLE KEEP SAYING. THIS ISSUE IS NOT FIXED IN WINDWOS 11 AND WSL2.

Why is this still an issue in 2023?

[zadjii-msft]

@maietta This issue was closed 5 years ago. The console codebase has had a tremendous amount of churn since then. Whatever you're seeing now probably has a new root cause. I'd recommend filing a new issue over at https://github.com/microsoft/terminal/issues, so we can ID whatever's causing what you're seeing.

Thanks!

[nikhil-swamix]

for people running cli scripts in python: disable this

Terminal › Integrated: Windows Enable Conpty

key navigation will work, for packages like rich, fire, click cli tools...