Console changes automatically changes font

[Dormanil]

I am running Windows 10 Pro Insider Preview Build 17711.

Opening a Powershell session with Open Powershell here starts a session with codepage 65001 (utf8). When using any commandline tool like robocopy, the codepage seems to switch to cp932, my computer's locale for non-Unicode applications, for the duration of the tool's task. After the task has finished, the codepage reverts to 65001. In case of these two codepages, where one codepage does not support Lucida Console, this codepage switching is easily noticeable by the switches to the CJK console font and back to Lucida Console.

Expected performance is that the codepage is not changed, even though it is preferable in cases where the tool actually outputs cp932.

[miniksa]

@DarthSpock, It's got nothing to do with the new API because there isn't a tool on the planet yet that calls the new API.

@Dormanil, what language is your computer in by default? What did you set your regional and locale settings to? (a.k.a. how did you end up in 65001 to begin with and are you perhaps using a Japanese machine so it is trying to default to 932?)

[Dormanil]

My machine is a very weird mixture of regions and locales. My Country or region as specified in the Settings app is Germany, my regional formatting is English (United Kingdom), as is my Windows display language and Default app language. My language for non-Unicode programs/locale is Japanese (Japan), as I am frequently using apps that require that specific locale.

I honestly have no clue why Powershell now defaults to 65001 when opening via Open Powershell here, as cmd starts up with 932.

[miniksa]

Click the "Change system locale..." button on the "Region" page where you captured the last screenshot there.

There's a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox there. Is it checked?

[Dormanil]

It is not.

[miniksa]

Registry Editor. HKCU\Console node.

Check all the folders and subfolders. Report any of them that have the key name "CodePage" set and what it is set to.

As an example, you might say HKCU\Console > No CodePage Key HKCU\Console\%SystemRoot%_System32_WindowsPowerShell_v1.0_powershell.exe\ > CodePage key says 65001.

[Dormanil]

Node	CodePage Key Value
HKCU\Console	N/A
HKCU\Console\%SystemRoot%_System32_WindowsPowerShell_v1.0_powershell.exe	932
HKCU\Console\%SystemRoot%_SysWOW64_WindowsPowerShell_v1.0_powershell.exe	N/A
HKCU\Console\C:_Program Files_WindowsApps_CanonicalGroupLimited.UbuntuonWindows_1604.2017.922.0_x64__79rhkp1fndgsc_ubuntu.exe	N/A
HKCU\Console\C:_Program Files_WindowsApps_KaliLinux.54290C8133FEE_1.1.1.0_x64__ey8k8hqnwqnmg_kali.exe	N/A
HKCU\Console\Git Bash	N/A
HKCU\Console\Git CMD	N/A

[miniksa]

OK, then that is mega wacky how you even got to 65001 in the first place.

I'm going to go with your weird locale mixing dance is tripping up someone's code somewhere in this entire stack. I'll file a bug, but it's honestly priority 3 as most people don't mix locales as hard as you do.

This is now MSFT:18239932.

[Blealtan]

Same thing happens to me. Insider 17713.1000 is used. No CodePage is set in HKCU\Console and any of its subsets, locale set to China (Simplified Chinese) and the 'Beta UTF-8 Support' isn't checked; launching PowerShell goes direct to 65001 and running any command (including PowerShell built-in ones) will cause codepage changed to 936 until the command finishes.

Also, something even worse is that in WSL, I ssh onto some machine (it's fine till here) and open a tmux/vim, font is changed to "新宋体" which is not the setting I'm using anywhere (I've changed font in HKCU\Console to some other font, and also ubuntu.exe). Here I've checked the codepage after tmux/vim opened, it's still 65001, not changed. No clue why this would happen.

[miniksa]

OK, so some background on why this happens.

GDI, the really old rendering stack we use, converts your codepage to a "charset" in order to figure out what font to use. This is easy and makes sense for things like 437 US English Codepage = Western Charset, 932 Japanese Codepage = Japanese Charset, 949 Korean Codepage = Hangeul Charset.

However, when you have 65001 for whatever reason, that represents UTF-8 Unicode which is technically supposed to be capable of representing any character. But it doesn't have a well-defined charset.

When the codepage changes from 65001 to 936 (which is probably your "non-Unicode locale for system applications" in the Regional control panel page), GDI goes from "I have no idea what the charset is so I'm not going to try to match a font" to "Ooooh, ooooh, I know what this is, I'm going to choose a font that primarily supports simplified Chinese!" and the font switches on you.

If you were in 936 the whole time, the font wouldn't have changed.

This isn't an easy fix on the console window side because some people like the font changing when things go from 437 (English) to 936 (Simplified Chinese) because most English fonts don't have Simplified Chinese characters in them, and that old rendering engine of GDI we use doesn't have a very good way of doing font-fallback (or stitching characters from other fonts into your current one when a glyph is missing).

I'm not sure why Powershell is launching in 65001. That's the biggest mystery and what has changed in the system to start showing you this problem.

[bitcrazed]

Update: I am discussing this issue with the PowerShell team. Will post an update soon.

[Blealtan]

It seems to have been fixed here in build 17763.1; the terminal now launches in 936 which is the correct codepage on my PC.

[miniksa]

OK I tried on 17763.1 and on 18309 and I cannot get the Powershell window to launch in 65001 with Launch Powershell Here. I'm going to close this for now and I closed the internal bug. If it comes back, let me know.