Windows 10 ssh Unicode characters display as U+25A1 'WHITE SQUARE' □

[IarwainBen-adar]

Troubleshooting steps https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps

Terminal issue? please go through wiki https://github.com/PowerShell/Win32-OpenSSH/wiki/TTY-PTY-support-in-Windows-OpenSSH I was optimistic that this would fix my issue, but alas it did not.

Please answer the following

"OpenSSH for Windows" version ((Get-Item (Get-Command ssh).Source).VersionInfo.FileVersion) I have only added the ssh client to my Windows install, I did not add the ssh server, so I am getting the version of ssh instead of sshd: 7.7.2.1

Client OperatingSystem Windows 10 Pro, here is my $PSVersionTable:

Name                           Value
----                           -----
PSVersion                      7.0.2
PSEdition                      Core
GitCommitId                    7.0.2
OS                             Microsoft Windows 10.0.18363
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

What is failing I am connecting via ssh to a Linux server, and am unable to draw Unicode characters. For example, the tree command just draws Unicode error boxes, instead of tree diagram characters. If I connect to this same server using PuTTy, I see all characters rendered correctly, so this does not seem to be a configuration issue on the ssh server end.

Expected output

user@host:/var/lib/nginx $ tree
.
├── body
├── fastcgi
├── proxy
│   ├── 1
│   │   └── 00
│   ├── 2
│   │   └── 00
│   └── 3
│       └── 00
├── scgi
└── uwsgi

Actual output

user@host:/var/lib/nginx $ tree
.
□□□ body
□□□ fastcgi
□□□ proxy
□□□ □□□ 1
□□□ □□□ □□□ 00
□□□ □□□ 2
□□□ □□□ □□□ 00
□□□ □□□ 3
□□□     □□□ 00
□□□ scgi
□□□ uwsgi

Amusingly, the box-rendered characters are in fact the correct Unicode characters. When I copied the incorrect tree rendering out of PowerShell in order to paste it into this bug report, the entire tree rendering pasted in correctly! So the characters are definitely being sent from the server to the ssh terminal correctly, it seems that ssh just doesn't want to render them correctly.

I am running ssh within PowerShell v7.0.2, where my $PROFILE includes the following:

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding
$env:TERM = 'xterm'

(And I can successfully render all manner of Unicode characters in PowerShell itself.)

I did reproduce this erroneous ssh Unicode character rendering in cmd.exe just in case this was an issue specific to PowerShell, and I get the same behavior with the same Unicode error boxes in a CMD-hosted ssh session just as I do with PowerShell.

To my slight dismay, I get almost-correct Unicode character rendering in a Windows Terminal tab of either PowerShell 7 or CMD. It is still not 100% correct, but it is not Unicode error boxes either. Here is the same output from an ssh session within a Windows Terminal-hosted PowerShell 7 tab:

user@host:/var/lib/nginx $ tree
.
╺━━ body
╺━━ fastcgi
╺━━ proxy
    ╺━━ 1
        ╺━━ 00
    ╺━━ 2
        ╺━━ 00
    ╺━━ 3
        ╺━━ 00
╺━━ scgi
╺━━ uwsgi

...which calls into question whether this is truly an ssh issue, or some other Windows console-rendering bug. If you have any opinion on how best to determine that, I would be happy to re-file this issue in a more appropriate GitHub if it turns out not to be a problem with Windows ssh itself.

Also, I am using the exact same font in each of PowerShell, CMD, Windows Terminal, and PuTTy.

[NoMoreFood]

Try experimenting with your console font. It's very possible the current font doesn't have a glyph for that character.

[IarwainBen-adar]

But, as mentioned above:

I am using the exact same font in each of PowerShell, CMD, Windows Terminal, and PuTTy

The font clearly has these glyphs, which PuTTy is able to select and render correctly. Obviously I could work around this problem altogether by just sticking to PuTTy for all my ssh needs. I opened this issue because (after controlling for console font and terminal application) it seemed possible that it was a defect in the Windows default ssh, and if so, could I provide any information that might help resolve it?

[NoMoreFood]

Can you verify you have the same issue with the latest version of the ssh client?

[IarwainBen-adar]

Yes, I get the same behavior with release v8.1.0.0p1-Beta.

[bagajjal]

@IarwainBen-adar - It works for me using V8.1. I am using Microsoft Windows [Version 10.0.19041.329]

[NoMoreFood]

Odd. I cannot repro. I put some characters in a file and outputted them to screen; worked fine. The 'tree' command also worked fine.

Can you run ssh -V and snapshot the output. Anything unique about your system that you can think of?

[bagajjal]

@IarwainBen-adar - I have the ssh -V as part of my previous reply. We use console support to parse the incoming VT sequences. Conhost.exe version matters here.

@DHowett is from console team. @DHowett is there any version of conhost (which parses VTSeq) has this issue? Does the "Windows terminal" app downloaded from windows store uses the latest conhost.exe? If yes, that's an option for end users to test these kind of issues.

[IarwainBen-adar]

This is my conhost.exe version: I'm on build 1909 of Windows 10 Pro, so I'm not sure if there is a newer conhost available for me?

Here is a screenshot showing ssh -V output (from a fresh download of v8.1.0.0p1-Beta), as well as showing what I see when I receive Unicode characters from the host: (And also as you can see, PowerShell itself has no issue rendering all the Unicode characters in my prompt setup, et cetera.)

For reference, here is a screenshot of a PuTTy session connected to the same host server, using the same console font, executing the same tree command:

@NoMoreFood re:

Anything unique about your system that you can think of?

I'm afraid not, this is a pretty standard Windows 10 installation. I guess the closest thing to non-standard would be the fact that I'm running PowerShell (core) v7.0.2 rather than the Windows default Windows PowerShell v5.1, but I think that's probably pretty far from unique at this point. I suppose the font itself is probably a bit outside the norm — I'm using the "Nerd Font" patched version of Cascadia Code — but I'm using that exact same font in PuTTy also, and PuTTy seems to be able to draw the tree diagram characters correctly.

[WSLUser]

This isn't an ssh issue. I tested this using PS7, Windows Powershell, CMD, and a WSL distro using 2 different fonts (Cascadia Code and Consolas). If you paste the □ it will display but getting it to display in tree or similar doesn't appear to be there. It looks like it's missing the unicode point for the translation. I will note the square looked slightly different between fonts. This issue should be moved to the Windows Terminal repo or the Cascadia Code repo.

[DHowett]

Sorry to take so long to reply to this.

Yeah, if ssh is just forwarding along the output from the remote server (as I believe it will do), this should work just fine and if it's not I'd love to take it up on the console repository (don't be alarmed: the entire windows console subsystem is in that repo, even though it's called "terminal".)

To remove the greatest number of variables, can you share the output of the following commands? Screenshots, preferably, since copy/pasting the broken glyphs doesn't recapitulate the issue.

$ locale
$ printf "\xe2\x94\x94\xe2\x94\x80\n"

It's fascinating that it's getting into the buffer correctly (as evidenced by the fact that copying them and pasting them somewhere smarter seems to show the right result.)

In addition, can you share a screenshot of the Options tab of the settings window for the console hosting ssh.exe?

[IarwainBen-adar]

@DHowett thank you for taking a look. Please let me know if these screenshots do not give you the information you're looking for.

PuTTY output:

pwsh-hosted ssh output:

pwsh options tab:

[DHowett]

Thanks! I've got one more thing for you to try, and this one's a bit crazy. I've been working on a console host debugging tool that will allow users to generate binary logs we can use to reproduce weird cases like this locally.

Would you be interested in giving it a try? If so, read on.

Conhost - dump and replay

[Here](https://terminalnuget.blob.core.windows.net/packages/OpenConsoleDumpAndReplay.exe) is a version of conhost built from our [open-source project](https://github.com/microsoft/terminal) with the ability to produce a binary log file of all the API traffic coming out of an application. _I know that running some random binary from some random guy is probably remarkably worrying. All I can really do is offer you a link to [the branch I'm using to develop the replay feature](https://github.com/microsoft/terminal/commits/dev/duhowett/hax/conhost_dump_replay)!_ Anyway. I'm interested in using this to see the data coming out of `ssh.exe`. > _⚠️ Please note, this dump may include identifying information. I don't recommend uploading it directly to GitHub. If you have to enter any passwords: user input is not captured as part of the replay file, but I understand if you're circumspect. My Microsoft e-mail address is in my profile!_ To use: 1. Download OpenConsoleDumpAndReplay 2. Double-click it. You might get a SmartScreen warning. 3. It will launch _cmd_. * This is actually the same thing that happens if you double-click `C:\Windows\system32\conhost.exe` :smile: * You may need to configure your font settings if you're not seeing the one you expect. Please let me know which font and font size you're using, as it could be important for the replay. 4. From cmd, launch ssh and reproduce the bug. 5. Exit or close the window. This will produce a file called `conhost_log.bin`. If you want to replay it (to make sure you got the right behavior), drag/drop it onto OpenConsoleDumpAndReplay. It should replay. Feel free to send it my way!

[he852100]

Its support for double byte Fonts is terrible,Please don't cover your ears

The server does not need any settings

[zoubenjia]

Exactly the same issue here.