Arrows ignored by bash when it started in ConEmu

[Maximus5]

I'm the author of ConEmu - alternative terminal for Windows. My users have created the issue Maximus5/ConEmu#629. When bash is started in ConEmu console, it totally ignores all extended keys (Arrows, Fn, etc.), so users can't access command line history with Up/Down arrows.

What is the problem with bash? Up/Down arrows properly cycle cmd.exe history, when it's started in ConEmu, but bash ignores them.

ConEmu is GUI application, so it get keyboard events via WM_KEYUP/WM_KEYDOWN and forward them to the created (usually hidden) real console using WriteConsoleInput.

[Maximus5]

I've created workaround in ConEmu build 160411. However, it would be nice to know, is it a bug or intended behavior.

[Maximus5]

Unfortunately, arrow keys still do not work in vim.

[sunilmut]

bash.exe has its own logic to read from console directly (this is as per the design today and subject to change in future releases). I am not a console expert, but I wonder if ConEmu interacting with the console directly is interfering with bash.exe's ability to process keyboard events?

[Maximus5]

ConEmu use official console API WriteConsoleInput to post keypressed into the conhost input buffer. That worked perfectly until "Bash on Ubuntu" appears.

[sunilmut]

Ok. I am not sure I fully understand the problem here. My understanding of this issue is that bash is not able to process arrow keys when launched from within ConEmu? Can you help clarify.

[Maximus5]

Short answer: Yes.

Many words below. Let's take for example UpArrow key.

When user press and release UpArrow, normal console application recieves two console key events: 1) bKeyDown = TRUE; wVirtualKeyCode = VK_UP; 2) bKeyDown = FALSE; wVirtualKeyCode = VK_UP; References: PeekConsoleInput, ReadConsoleInput, INPUT_RECORD, KEY_EVENT_RECORD.

But Unix console applications expect different key sequences. For UpArrow keypress either ESC [ A or ESC O A sequences are expected depending on requested ‘DECCKM’ mode. References: ANSI sequence ESC [ ? 1 h and ESC [ ? 1 l.

The bash.exe is Windows console application, therefore it's expected that this application would work properly with standard Windows console API keypresses (KEY_EVENT_RECORD, VK_UP). But, I'm not sure who, the translation is done when user presses UpArrow. And bash.exe recieves ESC [ A instead.

Well, that is rude workaround, but if bash.exe is started in ConEmu with -cur_console:p1 switch, ConEmu does this translation and arrows magically start working.

That is really rude, because ConEmu does not recieve ANSI sequences from linux processes. And, when user starts vim from bash prompt, ‘DECCKM’ is not changed. Therefore, ‘Arrows are not working in vim under ConEmu’. There are absolutely no way to determine that vim was started from bash, or vim has requested to change ‘DECCKM’ mode. Seems like linux processes (init, bash, vim) somehow write their console output directly to conhost, bypassing bash.exe. That's bad, because ConEmu has absolutely no control over linux processes (started from service), only bash.exe may be hooked to catch it's output and ‘fix’ input.

If you would like to check, download latest ConEmu build (alpha 160411 ATM), and run from Win+R:

ConEmu.exe -basic -cmd bash.exe -cur_console:p1

[Maximus5]

Sure, it is. Monday, 02 May 2016, 07:09PM +03:00 from Daniel Gordon < notifications@github.com> :

So Just to be clear here - you guys are talking about bash.exe - that is or is not the same thing as the new Ubuntu Subsystem in the most recent Windows 10 dev build? — You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub

[DanielGGordon]

@Maximus5 I deleted my comment because I figured it out on my own by looking at the top of this page 10 seconds after I posted my comment... Sorry for the confusion.

[dariobig]

Hi, console2 and consoleZ, two other terminal emulators, have the same problem with arrows not working on bash.

[Maximus5]

@bigonazzi Of course they are. Only ConEmu is able to send xterm-ish keyboard sequences, which LX kernel expects.

[xilun]

Just would like to add that IMO this is quite important to have WSL working properly with ConEmu, because some serious console users (myself included) are not going to be satisfied with the basic Windows Console (which is now quite better than the legacy one, but still far away from any random XWindow Console we are used to)

For now I'm doing some dev under WSL on the Windows Console (for now coping with the international keyboard bugs it has by switching to a US layout) but I will be seriously disappointed if I can't use ConEmu for a vast majority of WSL software, and that obviously includes vim...

@Maximus5 thanks for your excellent ConEmu btw :)

[dariobig]

@xilun +1

@Maximus5 I like conemu, all I'm saying is that the issue is bigger than just one program and it should be addressed with high priority.

[Darktex]

@xilun +1

The Windows default terminals don't have nearly enough features for any power user

[djmott]

@Maximus5 thank you for ConEmu. It's a huge productivity improvement. MS should ditch the POS default console, give you a few million USD and ship it with every install of Windows.

[sunilmut]

@Maximus5 - Given the popularity of ConEmu, you could also convert it to a UWP using “Project Centennial." and submit it to Windows Store for even a wider audience.

[nigurr]

+1 Please fix this

[miniksa]

Sorry, this is my bad. @zadjii-msft and I overlooked this portion of the design when we implemented the virtual terminal input handling in the console host such that it doesn't present itself properly for ConEmu and friends. Unfortunately because it's a design flaw, it's a little bit more work than a quick bug fix.

I have it tracked internally as 7640020. I can't commit to a date it'll be fixed, but know that it is in our tracker, it is understood, and it is being ranked/prioritized.

[dariobig]

@miniksa good to hear that you're listening to user feedback!

[Novynn]

Ah, just came across this issue. I hope we'll see some progress soon!

[Maximus5]

@Novynn What progress do you want to see? ConEmu had implemented workaround and arrows started working months ago.

[Novynn]

@Maximus5 I was meaning more towards on Microsoft's end, I wasn't aware of a workaround!

I've just started using Cmder today, which reports it's ConEmu as being 160710 stable (I see this version is the latest release on GitHub) and I am unable to use arrow keys.

I'm using cmd.exe in Cmder to start Bash on Windows to ssh into a screen session on a Linux machine. This is on the Windows 10 Insider Preview 14393.

[Maximus5]

ConEmu.exe -basic -run {bash}

[Novynn]

Ah, that makes a lot more sense! Works perfectly thanks.

I still hope that Microsoft resolves this issue so no workarounds are required on your end.

[pbecotte]

I have this issue trying to use the new bash terminal in Pycharm. They could probably implement the workaround listed here, but doesn't that still leave the terminal not working for things like vim?

[DanielGGordon]

Last time I tried it out, arrows worked on the command line, but not in VIM. Has this changed with a recent update?

On Tue, Jul 26, 2016 at 11:35 AM, Paul Becotte notifications@github.com wrote:

I have this issue trying to use the new bash terminal in Pycharm. They could probably implement the workaround listed here, but doesn't that still leave the terminal not working for things like vim?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/BashOnWindows/issues/111#issuecomment-235326658, or mute the thread https://github.com/notifications/unsubscribe-auth/ALRu8N6Xetakj1XAO9zQffA6NUfMrT7Nks5qZjdtgaJpZM4ID8Gx .

[pbecotte]

Pycharm has the parameters to call the executable with hard-coded in, so I can't just use the workaround out of the box (at least I think that's what's happening :) )

[gfairchild]

I'm experiencing the same vim issues another few users are reporting. Arrows work on the command line, but not in vim.

[Zardoz89]

htop looks that does weird things with the arrow keys. and Midnight Comander (mc), don't get any notice of arrow keys being pressed.

[nergdron]

I've found that with the workarounds in ConEmu, htop locally works as expected now, but remotely via SSH it still gives the weird broken default behaviour.

[meta-inf]

Neovim does not receive arrow keys properly with the {bash} profile.

[qrmn]

Same as nergdron, but also can confirm this is not a ConEmu issue: the exact same thing happens when using ssh under Bash in vanilla conhost.

It appears to be a WSL/conhost design fault that needs fixing if we're going to use ssh on Windows.

[i336]

I just found this bug linked from a random Hacker News comment, and while I'm not using Windows at the moment (hope remains for a future upgrade and dualboot at some point) I wanted to chime in and say something while this issue is actively being discussed.

As M$ is certainly well aware, the console subsystem in Windows has historically been incredibly difficult to work with. I'm admittedly unsure how much Powershell has fixed the situation (if at all), but going forward, I think that it would be 100% worth it to take the time (eg, a few extra months) to architect a solution that makes it possible to sanely manage console access for Linux apps, and possibly even Windows console apps too.

I cannot emphasize enough that there will be much appreciation and gratefulness if the race conditions (I understand ConEmu opens a hidden cmd process and constantly polls it for updates?) and other issues that plague Windows' current builtin console support can be sidestepped using a new API. It doesn't have to be perfect nor the easiest to work with; it just has to be predictable, consistent and well-documented. Fast would be nice too, so we can run find / without 100% CPU usage :P

Currently we've only got ConEmu, but now that Linux support is becoming more and more usable, I think bits of the Linux "scene" are slowly going to leak over to Win32, and it's possible that will include terminal emulators - either in the form of new Windows-specific apps, or hybrid codebases that can compile for Windows and Linux (and maybe macOS).

In my (possibly not completely thought through) opinion, a Really Good Way(TM) to fix this might be to make it possible to get at/create a Linux pty (ie what you'd get from a call to posix_openpt) from the namespace of a Windows process. Then a terminal application just needs to read/write the handle(s) it gets on the Win32 side of the fence to handle display and user input.

That only covers Linux tty access though. If there's some even lower-level approach you guys can use to fix it for Linux and Windows, I'm definitely in support of using these bugs to justify the development time :+1:

[dariobig]

+1 @i336

[miniksa]

@i336... I know. Everything you have said is on my mind and I'm considering ways we can solve it for Windows in the future. I'm quite proud of how far we've come in the Windows Console space in my short few years on this team, but I know we have a long way to go.

I can't speak in detail about planning and pre-production because that makes a lot of people cranky and means I have to spend my day making them all un-cranky instead of fixing actual problems. But do know that the spirit of what you've said is very well captured in my design plans.

I know the underlying console infrastructure on Windows is problematic. I know that other terminal emulators out there are better than what we've achieved for the Windows Console Host. I know that ConEmu's Maximus in particular has done an unbelievably good job at dealing with the BS that our API and console infrastructure is. I am trying my hardest to come up with a comprehensive solution that can make all of you happy, continues to advance the inbox Windows Console Host, allows other terminals to be very viable alternatives, and keeps my corporate overlords happy.

My team and I have a lot more in store for the console in the coming year. Please bear with us as we work. We look forward to pleasantly surprising you (hopefully!) as much next year with what we still have to deliver as we did this year with what we achieved in releasing Bash on Ubuntu on Windows.

---

Also... just as a little aside... The "M$" is not appreciated. I understand that sort of phrasing is common when referring to my company and its behavior especially in the past. Heck, I was guilty of it in college when I solely used Linux and hated on Microsoft products. But it's quite hurtful to me, my team, and all the engineers here like us who are working very hard every day to change how Microsoft operates and bring our products and community closer to yours. This company has come a long way in the past few years, and I'd like your help in embracing where we can go and what you and I can achieve together... not the missteps of former products and people here that might have colored your opinion. I wouldn't work here if I thought all that we did was for the money and nothing else. I hope that my team, my fellow Microsoft engineers, and I can change your opinion in that respect.

[Tyriar]

Just chiming in from Visual Studio Code, users that attempt to use bash.exe as the shell in the integrated terminal is also hitting this issue. It uses winpty to communicate with bash.exe.

[i336]

@miniksa: Thanks so much for your candid response (and so fast too!), it's awesome to hear it. I've had a vague awareness for some time that the console infrastructure is broken, and it's really really cool to be talking to someone on the team who's going to be working on fixing it <3

Based on what you've said, I get the impression things are the way they are possibly because of 3rd-party application compatibility? (Always technically curious. I reckon that insight on the current infrastructure and its technical evolution - posted to somewhere in-depth like The Old New Thing, which is always full of fun trivia - would make for a very interesting read for a lot of people. Perhaps once things are a bit different.)

Regarding your aside, I must admit my embarrassment at my subconscious choice of contraction of the word Microsoft - I actually intended it as a benign reference, and didn't think of the ever-present connotations behind the $. Thanks for reminding me (and helping clarify) that Microsoft is in a very different position today, with a different focus - s/M\$/Microsoft/g from me from this point forward :)

[miniksa]

@i336: It's cool. Thanks for understanding. :)

Compatibility is one factor of many for why things often stay the same or get extended in an interesting way. Most of the console's oddities are explained by organic growth of the code base over 27+ years, so you could call it evolution. It's a system with many, many customers and had a steady stream of different owners. It hasn't had a thorough dose of dedicated comprehensive love until my team formed a couple years back and started cracking the nut.

I have a wall outside my office full of post-it notes and yarn scraps of the architecture, the customers, the issues, and the dependencies. There is a lot more to it than most realize, so any change often affects all the other parts. ConEmu and similar tools are up there as the fifth pink (or "server-side customer") square in from the left under "Customers" and are connected to various of the orange "Issues" below, one of which is "Non-public server API" which makes Maximus do the polling as you observed. Trust me, it's all captured on that wall.

@bitcrazed and I have been discussing coming up with a series of blog posts about the good, the bad, and the ugly of the console system. I hope to get to doing that soon since it seems you and others would be interested.

[ericblade]

FWIW, I'm also interested in this bug, as a user of ConsoleZ, with 6 different command processors currently installed :-D

Also worthy of note, that running WSL in ConsoleZ results in obviously lagged screen output, which does not happen with other command processors.

[fpqc]

@miniksa Will you be starting a console blog like the WSL blog, with or without Seth Channel 9 videos?

[miniksa]

@fpqc We'll be starting something whenever I can wrangle our PM @bitcrazed into helping me get it all set up and approved. He's been very busy recently with another more time-critical scenario, so the blog/video/etc. plan has been on the backburner and the rest of us devs have been busy coding away on the console host in the meantime. We didn't forget, trust me. I nag him in stand up meetings several times a week for when he's going to be done with his other work so we can get to setting this up. :P

[jrock2004]

Wanted to add another twist. I am using zsh and my arrow keys don't work. It does not matter which terminal I use.

[zadjii-msft]

@jrock2004 Could you open a new issue for that? They should be working in conhost.exe, and a new issue would be the easiest way to debug your issue (without constantly pinging the other 22 people already in this particular discussion)

[bitcrazed]

@fpqc It's true ... @miniksa & @zadjii-msft and the rest of the team REALLY DO nag me at least once a day on a number of things I'm not helping him with!!

Yes, we do have a developer blog where we post articles about all things command-line - Console, Bash, and hopefully PowerShell too: https://blogs.msdn.microsoft.com/commandline.

We'll be posting more stuff about the Console's new features in a couple of weeks (honest Mike!), we've just got to get some high-pri stuff out of the way first ;)

[miniksa]

@bitcrazed I'll believe it when I see it. :P

[arcanis]

HyperTerm just released its first official Windows build, but unfortunately it also suffers from this issue.

https://github.com/zeit/hyper/issues/1127

[arcanis]

Worth noting that a relatively easy workaround is to start bash using wslbridge instead of a direct bash command. It also fixed a few other display issues I had on Hyper.

[Tyriar]

@arcanis also Hyper is in the process of adopting the new version of node-pty which should fix the issue without needing wslbridge due to a newer version of winpty.

[khards]

Would be good to see this bug fixed

[Serubin]

@Tyriar Do you know which branch that is of Hyper if there is one?

[Tyriar]

@Serubin latest Hyper should include the fix.