microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.25k stars 812 forks source link

Terminal emulator gives blank screen when executing HTOP or SCREEN #7660

Closed AriESQ closed 2 years ago

AriESQ commented 2 years ago

Version

Microsoft 10 Enterprise LTSC 1809 [Version 10.0.17763.2268]

WSL Version

Kernel Version

Linux 4.4.0-17763-Microsoft x86_64

Distro Version

Ubuntu 18.04 LTS

Other Software

openssh.exe cmd.exe ssh htop screen ncurses

Repro Steps

1) Open an WSL instance, which drops you to a functioning command prompt. Execute HTOP, screen goes blank. Press Q to exit HTOP, and you will see a portion of HTOP at the top of your screen, confirming it was running.

2) ssh into a remote linux instance from the WSL environment. Execute SCREEN or HTOP, you will receive a blank screen, but keyboard input is still active.

3) open CMD, run openssh.exe to ssh into a linux machine. Execute Screen or HTOP, you will receive a blank screen, but keyboard input is still active.

4) While attempting to attach strace logs, I ran man strace and found that even man is broken, giving a blank screen. type man strace and you will have a blank screen, exit with q and you will have some scrollbck of the man page.

Because this behavior is shared between WSL and CMD.exe in terminal mode, I believe that the problem may lie in the underlying windows terminal emulation.

KB5006744 appears to be the breaking update. If I uninstall this update the problem goes away, if I re-install it the problem comes back.

I suspect the problem is that when the remote system or PTY grabs control of the terminal, such as by issuing a window resize, ASCII control code, or the like the window terminal emulator is misinterpreting it and painting a black screen.

I have troubleshooted by removing .bashrc and .profile custom PS1 prompt configurations that might be transmitting ANSI color codes. I have switched users to a bare root /bin/sh shell. I have tested on different local and remote linux installations.

The terminal in vs-code is not affected by this issue, and third-party terminal emulator ConEmu is also not affected by this flaw.

I may have traced things to default pager less. Or, less has the same problems experienced by htop and screen. Overriding the default pager via man -P more allows man to execute. So less is affected but more is not. top is unaffected but htop is.

I have attached strace but nothing jumps out at me as the culprit.

wsl.exe --status produces no output, however the binary has a modified date of 2020-03-10. I believe I have an older version of WSL1, perhaps installed via github? I mention this extensively because my version of windows is not supported by Windows Terminal.

Expected Behavior

The terminal application should not go to a blank screen when executing programs such as HTOP (Ncurses) or SCREEN (spawns a PTY).

Actual Behavior

When executing commands such as HTOP or Screen, the terminal emulator paints a black screen.

Diagnostic Logs

No response

StnVsl commented 2 years ago

Same issues with vim and mc on Windows Server 2019 Standard 1809 (build 17763.2300) after KB5007206 update.

josuenadal commented 2 years ago

This is happening to me when i try to run nano, vim or a Text-based User Interface (TUI) update screen on my debian machine through SSH on the Windows CMD or PowerShell (not WSL, just CMD & PowerShell).

Where normally i would see the Nano TUI or the VIM TUI i am seeing a blank screen with a cursor in the top corner that can recieve input, but does not show any change. There is nothing upon the screen until i CTRL+Z or CTRL+C my way out of the "open" application. Then i see that it was apparently open and some traces of the TUI are on the top of the screen as well as some of my input, scattered.

This is on Windows 10 LTSC, Version 1809, Build 17763.2300.

ligouras commented 2 years ago

I'm experiencing the exact same behavior as Atlas667. I am too using Windows 10 LTSC 1809, WSL1 + Ubuntu 20.04 (focal). This seems to have happened after a windows update reboot. The last two updates that applied were KB5007206 and KB5007298.

I also cannot use vim or htop when connected to a remote server through SSH. Though I'm invoking vim on the remote server, the terminal running on WSL still draws a blank screen.

sanikolov commented 2 years ago

+1 the writeup by ligouras matches my diagnosis exactly. I am seeing this on server 2019 dev essentials.

acanakoglu commented 2 years ago

+1 I am using Windows 10 LTSC 1809, WSL1 + Ubuntu 18.04 LTS.

I removed the updates KB5007206 and then KB5006744, the problem is resolved.

bouvetcj commented 2 years ago

+1 Windows server 2019 version 1809, WSL1 + Debian 10

imanabu commented 2 years ago

Also confirms that this happens on Windows Server 2019 with Ubuntu 20. Looks like some form of termcap issue to me, but I do not know enough (or time to figure that out yet) to diagnose this one. This does happen on vim and also emacs I am able to do this on WSL2 on Windows 11.

Primozz commented 2 years ago

Same on Windows 10 LTSC 1809, WSL1, Debian 9. Like imanabu I was also able to trace the issue down to termcap, but don't know enough to go further. Before finding this issue, I worked around the problem by setting TERM to one of the other available terminals from ls /lib/terminfo/*. export TERM=vt220 works for me. Not an alt terminal though, but still better than blank.

giofx commented 2 years ago

Hi, I got the same issue last week on Windows 10 enterprise LTSC 1809 17763.2183, WSL 1 and ubuntu 18.04 LTS. As suggested above, I removed KB5007206 (I had no KB5006744), and after reboot nano/vim started working as before

NikWP commented 2 years ago

Same on Windows 10 LTSC 1809, WSL1, Debian 9. Like imanabu I was also able to trace the issue down to termcap, but don't know enough to go further. Before finding this issue, I worked around the problem by setting TERM to one of the other available terminals from ls /lib/terminfo/*. TERM=vt220 works for me. Not an alt terminal though, but still better than blank.

Thank you! export TERM=xterm-color works for me!

foobird13 commented 2 years ago

I'm observing this issue on Server 2019 (multiple machines) running Ubuntu 18.04 under WSL1. In addition to the blank screen issue, as another person mentioned, if I had tried editing a file I would see portions of the file contents displayed in the console after exiting the app.

Exporting a different TERM does work to resolve the issue of a blank screen in vi, nano, etc., but I do still see some artifacts. For instance, after exiting nano, the console screen would be empty (i.e. none of my history showing) except the prompt at the bottom. So the issue seems to be affecting the buffer as well.

stemabo commented 2 years ago

+1 cannot use nano, vim or htop (on WindowsServer 2019) Terminal appears to be fine first, but starting nano or vim results in a blank black screen. Need to then hit CTRL+X or CTRL+C to get my prompt back, this also displays some zombie-nano content: the content of the file I was trying to open in nano + a grey "GNU nano 4.8" header - but at the bottom I am back at the command prompt (smiliar with htop). reboot does not resolve the issue - but export TERM=xterm-color is a workaround for now (might be related to KB5008602 and/or KB5006368 but I am unable to roll these back in order to find out)

ehm-wehb commented 2 years ago

Same issue here, just recently lost htop, vim, and nano in WSL. Recently installed KB5006368.

export TERM=xterm-color worked for me.

Server 2019 - 1809, Ubuntu 20.04

moodmosaic commented 2 years ago

FYI, export TERM=xterm-color seems to be working also for me. It'd be good to know why KB5x update broke it.

moodmosaic commented 2 years ago

FYI, export TERM=xterm-color seems to be working also for me. It'd be good to know why KB5x update broke it.

FWIW, this also fixes clear where otherwise you'd have to do clear -x.

AriESQ commented 2 years ago

export TERM=xterm-color is not a complete fix. That replaces the blank screen issue with improper display of color codes. such as with the ncurses interface of the package nmon and some functionality with arrow keys displaying control characters instead of being interpreted.

Microsoft, you have broken my ability to use terminal for weeks. Behind an opaque update process where I can't even identify the breaking patch.

Do you understand what a cardinal sin it is to break the terminal, for someone that wants to productively use their system?!

epistlewool commented 2 years ago

I am having the same problem since KB5007206 which I am unable to roll back. I noticed that the terminal in vscode works as expected when running in WSL mode. So that is an alternative workaround.

HontoNoRoger commented 2 years ago

I had the same issue, and as already pointed out by @AriESQ xterm-color causes problems with arrow keys.

For me it was issues when using a previous command and changing it. The terminal replaced the input with the pre-existing output rather than inserting it. When executing the command however, the input seemed to have been inserted properly, so it was just a display issue.

Setting export TERM=Eterm-color (Thanks @Primozz for showing the other available terminals) actually helped me with this issue as well, so I'm issue free now, as far as I can tell.

craigloewen-msft commented 2 years ago

Hi folks, we're taking a look into this issue. I'll post updates here as soon as they're available.

AriESQ commented 2 years ago

Thank you @craigloewen-msft There is a lot of information in my initial post. I'd like to point out that this is reproducible in the WSL default shell. As well as when using a CMD window and SSH.exe into a remote host. To me that says there is a shared operating-system level component involved.

Also, from all the people in this thread, we have a pretty good idea which KB updates were involved. I was personally able to narrow it down by uninstalling/re-installing. However, as other's have mentioned the KB update is no longer available to be uninstalled. I think this has something to do with how MS rolls out "preview" updates, but it's complicated and I don't personally understand it. I would have otherwise worked to diff my system before/after the update to narrow down what changes were made.

TY

JackMerlin commented 2 years ago

Hello, @craigloewen-msft

I can confirm this problem. We are a small business with 10 computers running LTSC 2019. We received a report on this problem after the November update. an employee reported that files cannot be edited after logging into our unix-like servers using the SSH client in CMD, and both vim and nano commands will return a blank screen. Our IT contacted Microsoft Enterprise Support and they did not provide any useful solutions. we are currently using third-party software (PuTTY) to replace the affected system components.

At present, our system has been updated to the December update (KB5008218), and it can be confirmed that the current problem still exists.

Please solve this problem as soon as possible, because it has caused many problems for us, thank you.

Update:

The export TERM=xterm-color command does't work for our unix-like servers, and the blank screen is still returned under vim. but nano will return Error opening terminal: xterm-color.

However, using export TERM=linux [1] allows us to use nano, and vim still cannot work in the built-in SSH client on Windows 10.

[1] https://superuser.com/questions/1172222/issues-editing-files-with-nano-in-bash-windows-10

AriESQ commented 2 years ago

@JackMerlin Thank you for providing additional detail, it is noteworthy that you are experiencing this in CMD.exe and SSH.exe; as I have. Maybe people are experiencing this with the WSL terminal, so it seems reasonable that Microsoft updated a component that is shared between those terminals.

My sense is tha vim and nano seize control of the terminal window. And perhaps operate differently than an ncurses program that draws special characters to the screen. Would you be able to confirm you are experiencing the problem with an ncurses program such as htop?

I think we have tee'ed this up with plenty of info for Microsoft to resolve, many thanks to everyone supplying info.

JackMerlin commented 2 years ago

@JackMerlin Thank you for providing additional detail, it is noteworthy that you are experiencing this in CMD.exe and SSH.exe; as I have. Maybe people are experiencing this with the WSL terminal, so it seems reasonable that Microsoft updated a component that is shared between those terminals.

Yes, we did not use WSL, we did not even use Powershell, just cmd.exe and the built-in openssh client (OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5). We have already ruled out the cause of the problem caused by our servers in contact with Microsoft technical experts. We can reproduce the problem after installing LTSC 2019 on a virtual machine and updating to the latest cumulative update. So we insist that it is the Windows update that caused the problem.

This is a screenshot of the blank screen returned by vim and nano, hope it helps. blank_screen_returned_by_vim_and_nano

My sense is tha vim and nano seize control of the terminal window. And perhaps operate differently than an ncurses program that draws special characters to the screen. Would you be able to confirm you are experiencing the problem with an ncurses program such as htop?

Sorry, our system uses top instead of htop. In my case, top can work normally.

I think we have tee'ed this up with plenty of info for Microsoft to resolve, many thanks to everyone supplying info.

Thank you, I think this issues is more useful than some technical experts, because they did a lot of diagnosis, but did not provide any useful solutions.

I am here to reply to the hope that Microsoft will pay attention to this issue and hope that it can be fixed in the next update.

Thank you.

mihalycsaba commented 2 years ago

I think I have this issue, every TUI program stopped working on one server I administer, like 2 months ago, but I have the same issue using putty too. Also I can't connect with winscp to that server. ctrl+c and ctrl+z doesn't do anything,

mihalycsaba commented 2 years ago

Forgot to mention, that other people use that server too, but they don't have issues. So this is why I suspect, that it was a windows update that broke something on my machine.

colhountech commented 2 years ago

+1 Windows Server 2019 Standard - 10.0.17763 Build 17763

AriESQ commented 2 years ago

Here is some documentation on Windows Console: https://docs.microsoft.com/en-us/windows/console/

I'm realizing that looking at process trees, conhost.exe is the common link between this issue presenting under cmd.exe and wsl.exe.

@craigloewen-msft any movement on this? We have identified in-thread the updates that introduced breaking changes. I wish I could give a diff of what files changed, but I don't understand how "preview" updates work, and KB5006744 is no longer available for me to uninstall from my system.

My terminal, a fundamental, basic thing to how I work, has been broken for months now. This is leaving a really bad taste in my mouth about using WSL going forward.

craigloewen-msft commented 2 years ago

Yes we have movement on this! We've debugged this internally and have identified a fix. Currently we're working on servicing this fix and I'll be sure to post updates to this thread when they're available.

@AriESQ Does the work around of using export TERM=linux help you at all to unblock you?

AriESQ commented 2 years ago

@craigloewen-msft TERM not working: xterm-256color (ubuntu default) linux vt100

TERM sort of working: xterm-color eterm-color (best) vt220 (black and white, to be expected)

TERM: Linux and vt100, introduce a character display issue, but fix the screen blackout issue. 2022-01-06_Ubuntu_18 04_LTS815

Thank you for your responses. I respect that getting such a patch out to MS software globally is a huge undertaking.

lfr commented 2 years ago

Same here on Windows Core, which I don't even know how to update or manage 😟

JackMerlin commented 2 years ago

@craigloewen-msft We need some updates and progress, is there a release schedule out there, will we see a fix in the February cumulative update?

craigloewen-msft commented 2 years ago

We've found the cause of this bug, identified a fix, and are working on servicing that fix out to Windows! I will be sure to post updates to this thread when they're available.

datvv commented 2 years ago

I just installed Windows 11, Now I can't use HTOP and edit files on my server via VIM editor. What should I do now?

mihalycsaba commented 2 years ago

Hey guys, I wrote here earlier, I had a problem with a server a while ago, last week some of the client win10 machines started to have problems too. The TUI app stopped loading.

Had another symptom, winscp couldn't connect. With this I managed to figure out it was a problem with the MTU size, the ISP changed something on their end(couldn't ping the server ip), that broke the mtu size setting.

Fixed it with reducing the mtu size of the ethernet interface on the server.

datvv commented 2 years ago

this issue is fixed after I upgraded to use wsl2. https://pureinfotech.com/install-windows-subsystem-linux-2-windows-10/

greenaussie commented 2 years ago

I have this issue with neither of the hot fixes mentioned above installed. Server 2019 Datacentre / 1809 / 17763.2565 (on AWS WorkSpaces).

Looking forward to the fix!

sietse commented 2 years ago

@AriESQ , thank you for your overview of how well different $TERM values work for you. What did eterm-color do better than xterm-color for you? I ask because I had the opposite experience, described below.

For me, xterm-color worked better:

(This was with WSL 2, Ubuntu-20.04, WSL last updated on 2021-11-12, Kernel version: 5.10.60.1.)

AriESQ commented 2 years ago

@sietse Are you having the blank screen issue that is the topic of this ticket? xterm-color which is the default does not seem to be working for people and eterm-color is a workaround.

sietse commented 2 years ago

@AriESQ apologies, I don't have the blank screen issue. Your overview comment was helpful to me, and I wanted to offer further information as thanks; but I was so intent on replying to your comment that I forgot to consider this is an issue thread. I'm sorry for the noise.

AriESQ commented 2 years ago

127 days (4 months and 7 days) since this was reported.

Microsoft broke the Console in an opaque way and still hasn't fixed it. This is embarrassing.

moodmosaic commented 2 years ago

It really is.

mikepietruszka commented 2 years ago

Count me in as another person having this issue :(

craigloewen-msft commented 2 years ago

Hi all, we have released a fix for this as part of

KB5011551. Right now this is available for seekers, but next month it will be part of the patch Tuesday patch.

Thank you for filing this! I'll close this issue once it's pushed as part of a regular update.

AriESQ commented 2 years ago

I don't see the fix listed in the release notes, so it looks like we may never know the root cause or the solution. Very disappointing compared to open source tools where there is transparency.

I haven't tested this yet, but looking at the changed files, my guess is that the fix is contained in :

Windows 10 version 1809 LCU x64-based               
File name   File version    Date    Time    File size
conhost.exe 10.0.17763.2746 7-Mar-22    23:39   826,880
sietse commented 2 years ago

@craigloewen-msft , good on you! I look forward to trying it out, the patch will probably reach me next month, if I remember I'll post a little update here.

Out of curiosity, what sort of problem caused this behaviour? Was it indeed some terminal control sequences getting misinterpreted by the WSL console / terminal emulator?

Cheers!

AriESQ commented 2 years ago

Via the Microsoft Terminal project; it looks like MS recently open-sourced the traditional ConHost terminal. 😲https://github.com/microsoft/terminal/tree/main/src/host

I did a quick look there (time bound by the appearance and patch of the bug.) And I didn't see anything obvious as a root cause of this present bug, in my casual search.

JackMerlin commented 2 years ago

@craigloewen-msft thank you, I can confirm that this does fix the issue and we will be deploying to all affected PCs within the next month.

Thanks also to this issue and to everyone who reported the issue to Microsoft, your efforts made this fix available.

Gyross commented 2 years ago

@AriESQ @sietse Have a look at this: https://github.com/microsoft/terminal/issues/12329

My hunch is that it's the alternate screen buffer sequence not being handled properly: the console view changes correctly, but not the write destination, but I'm 100% sure it's a conhost issue. I didn't check that classic API screen buffers still worked, but if they didn't it would probably break a lot of native windows tools as well.

AriESQ commented 2 years ago

@Gyross Thank you, I always suspected this was an issue in microsoft/terminal But because their minimum requirement is Windows 10 version 18362.0 or higher, I expected a ticket there for my problem would be closed immediately. I suspected MS Terminal, because who else would be modifying this ancient feature of Windows?

AriESQ commented 2 years ago

I have installed Cumulative Update Preview for Windows 10 Version 1809 for x64-based Systems (KB5011551) on the system listed in this original ticket. The screen blanking issue appears to be solved. This is with the default $TERM = xterm-256color. I have tested the WSL console, as well as CMD.exe and openssh.exe.

However, @craigloewen-msft this issue may not be fixed in it's entirely. I run a 2560x1440 monitor. And I use the "window snapping" feature (i.e. pressing Win+arrow key, or dragging window to the edge of monitor.) I can use the WSL terminal window in two configurations, "half-screen snapped" and "full-screen maximized".

Executing a full-screen (ncurses?) command in half-screen mode, and then moving to maximized, the bottom line of text stops displaying (possibly only when the Properties > Layout >Wrap text output on resize checkbox is un-checked. Unchecked appears to be a non-default option.)

This behavior is inconsistent between when $TERM is set to xterm-256color, eterm-256-color, and eterm. Eterm which was the previous temporary fix does not appear to be affected by this issue of the bottom line not being displayed

This solution is working for my purposes, I suppose. I don't have the time to do any more free troubleshooting for Microsoft. I am unsure at the correctness of the solution. I just think of all the effort that people have taken to ensure the integrity of POSIX terminals and I am disappointed. Microsoft in their own codebases are known for rigorous backwards compatability.

Anyways Craig, thank you for (mostly) running this to ground and being responsive to the bug. Whether you want to close this bug is outside of my discretion. Cheers. 👍