keyboard input randomly hangs when typing into an MSYS2 MINGW64 shell and newly established SSH remotes.

[ipaqmaster]

Windows Terminal version

1.18.1462.0

Windows build number

Microsoft Windows NT 10.0.19045.0

Other Software

Command line = C:/msys64/usr/bin/bash --login

Steps to reproduce

Open a new terminal tab, ssh to a remote machine. Randomly throughout the day these shells will just stop accepting keyboard input and rarely resuscitate after a long delay.

Expected Behavior

A terminal experience which doesn't hang so often.

Actual Behavior

Opening new shell tabs and sshing to hosts can sometimes hang with no way to type any further (Not even Enter+Esc+. will kill ssh when it hangs like this at the prompt of a remote).

[zadjii-msft]

Y'know, I think we've heard something similar to this over in #14019 before. From that thread:

I am not sure if it is a terminal problem, but other ssh client keeps connecting server.

What is the other SSH client that works for you?

Is this maybe relevant to your issue? superuser.com/questions/468584/ssh-connected-terminal-windows-hang-in-vm-after-waking-host-from-sleep As described here: http://manpages.ubuntu.com/manpages/jammy/en/man1/ssh.1.html#escape%20characters

I think it's not really that the Terminal is freezing, but rather that ssh doesn't realize it's disconnected and the remote host not responding to the TCP packets. I believe this requires setting keep-alive values to fix and is not specifically an issue with the terminal.

Thanks for your answer. I think you are right, I missed some SSH parameters

    ServerAliveInterval 300
    ServerAliveCountMax 2

Now I'll add it to my ssh config file and see if it's working.

Does that seem to help/?

[ipaqmaster]

I'm aware that those two will help prevent the underlying TCP connections from being considered dead and dropped by network devices between a user and their remote ssh server - but that isn't related in this case.

I'm connecting to remote machines and my keyboard input is sometimes Immediately becoming unresponsive.

Furthermore, I can do ctrl+shift+t to open a brand new Bash shell locally - and that also hangs immediately sometimes.

In both of these cases I have to ctrl+shift+t open a new Windows Terminal tab and X out of the other one while this one loads

New terminal windows and bash shells can be very slow to load compared to a native Linux install due to Defender's file, memory and process scanning overhead. But this complaint is outside the scope of this issue and when these terminal hang's happen - there is no CPU activity on the host - so I don't believe Defender is to blame for this issue either I don't think.

---

I can also confirm this isn't a case of Ctrl+S/Ctrl+Q either (Ctrl+S sends a control-character to stop printing to a terminal and Ctrl+Q resumes it. Remnants from the teletype consoles of the 70s/80s). Mashing Ctrl+Q doesn't suddenly allow me to type again. So it can't be related to these two classic control characters.

I'm really unsure what this is... but the next time it happens I'll poke around even more to try and find a cause or consistency.

[ipaqmaster]

Happened again this afternoon. I've SSH'd to a machine on the network and get a terminal prompt ~# which I cannot type into. Immediately as the session starts.

Funny thing - I can SSH into the same host in another tab and run echo test | wall and the message appears in the stuck terminal.

So when the terminal hangs - It's still running and receiving SSH traffic - it just cannot send/transmit its own traffic (/ keystrokes) back.

I also tried typing echo test | wall in the hung terminal - this did not work, confirming that the ability to accept keyboard input is simply hung for some reason or another. But the terminal can still receive and display text.

[ipaqmaster]

Furthermore - I packet captured filtering for tcp.port == 22. Typing into the stuck terminal SSH session does not show any new packets being transmitted. While typing into the working terminal SSH session did.

This helps eliminate that this is not a networking problem. The hung terminal can still display text from the remote if received (e.g. by calling echo test | wall from another terminal SSH'd into the same remote for the message to display to all logged-in users.

It's also worth mentioning that I can't <Enter>+<Tilde>+<dot> to signal to the SSH client of the input-hung terminal tab to end the connection - something which works in the non-hung tab just fine. This bug seems to be an entirely local keyboard input bug I keep running into with Windows Terminal. I just don't know how to consistently trigger it yet.

These tests confirm this is a local problem on the local machine with Windows Terminal Preview. The terminal tab just stops accepting keyboard input. I also couldn't Ctrl+S/Ctrl+Q it back into accepting keyboard input either. No amount of Ctrl/Shift/Ctrl+Shift key mashing could revive the tab. I wonder if there's another keyboard shortcut I've accidentally invoked when this happens?

[DHowett]

It might take some time to redact, but can you reproduce this under ssh -v -v -v?

As well, are you using msys' build of OpenSSH or Windows'?

If msys: if you use the one directly out of /mnt/c/Windows/System32/OpenSSH, do you see the same failure dynamics?

[ipaqmaster]

I'll try to reproduce the hang with -vvv specified.

Yeah this MSYS2 MINGW64 Shell is invoking its own build of ssh. ssh -V says its OpenSSH_9.4p1, OpenSSL 3.1.2 1 Aug 2023.

I'll also try to experience the problem using the Windows OpenSSH build.

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[ipaqmaster]

It's happened a few times since the last comment but I've been unable to use the native Windows ssh executable as it doesn't acknowledge my SSH agent which I need to shell into things securely with a key which would slow a day down to a crawl.

I also haven't been able to recreate the experience on purpose while using -vvv. I can't make an alias to leave it on all day because it makes the regular shell experience very noisy even between commands on established sessions.

This is proving difficult to track down a cause for. But it's worth mentioning that this can also happen on the local shell as a new Terminal Preview tab or window is spawned - just a rarely as the one-way SSH lockup. But I can't find a reproducible cause just yet. I think the only thing I've noticed is that it can happen when I'm doing things very quickly and concurrently - quicker than this office laptop can keep up with sometimes (Bogged down with Defender scanning every little thing plus two other modern agents also taking their piece of the cpu pie all the time). It makes me wonder if I'm accidentally triggering some race at times.

[ipaqmaster]

Ha of course that small speed comment was the important trick. I got it to happen consistently now and it seems to only happen when my inputs are buffered way before the terminal finishes loading and processing my bashrc.

If I Ctrl+Shift+T Windows Defender Advanced Threat Protection goes into high gear watching the new MSYS2 MINGW64 terminal load up. Because of this security policy it takes about 4.2 seconds before I can type anything in. I am not in a position where this can just be turned off.

If during this pre-prompt loading stage I type in ssh my-common-remote the terminal eventually finishes loading and immediately shells to what I typed in while it was loading. As is expected in any terminal software.

If I touch nothing else the shell to my-common-remote loads fine and can be used. But if I begin typing even more commands after the pre-typed ssh line .... the remote SSH shell cannot be typed into but maintains a full connection logging keepalives and all.

There's also some interesting ssh -vvv output if I happen to include the debug flags up to level 3:

debug2: shell request accepted on channel 0
debug2: channel 0: read<=0 rfd 4 len 0: closed
debug2: channel 0: read failed
debug2: chan_shutdown_read: channel 0: (i0 o0 sock -1 wfd 4 efd 8 [write])
debug2: channel 0: input open -> drain
Last login: Mon Sep 18 13:44:24 2023 from <Last login IP>
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed
my-common-remote ~ $ 

At the bottom there is the end of the SSH initialisation debug and appearance of a prompt which I cannot type into. That input drain open/close does not appear when SSH is left alone to finish loading before I type anything in.

[ipaqmaster]

This does not happen if I ssh to the remote and pre-type things from an existing local already-at-bash-prompt MSYS2 MINGW64 tab. Only new ones.

[carlos-zamora]

quick question: when your ssh session is in a hung state, if you press enter then ~ then . does it terminate? If it does, that confirms that the ssh client is receiving input from the terminal.

[ipaqmaster]

I use that quite often but it does not work when Windows Terminal gets in this stuck state.