eryksun
commented
4 years ago A seemingly related issue is that when a non-BMP character is manually pasted into the console, ReadConsoleW echoes and returns only the first code in the UTF-16 encoded surrogate pair. For example, with "😞" only 0xD83D is returned. Here's a simple example in Python:
>>> from win32console import *
>>> h = GetStdHandle(STD_INPUT_HANDLE)
>>> h.ReadConsole(4)
�
'\ud83d\r\n'(As before, I've replaced the echoed 0xD83D surrogate code with U+FFFD in order to avoid problems with pasting an invalid code point.)
This prevents pasting supplementary-plane characters on the command line in the CMD shell, which relies on the console's cooked read for its command-line interface. For example:
Legacy console:
C:\>echo 😞
😞Windows Terminal:
C:\>echo ��
😞V2 console:
C:\>echo �
�PowerShell uses low-level ReadConsoleInputW instead of ReadConsoleW, so it's not subject to this bug, though external programs that inherit PowerShell's console may be.
DHowett-MSFT
BDisp
DHowett
tig
o-sdn-o
Environment
Microsoft Windows [Version 10.0.18363.657] conhost.exe builtin console, V2 wt.exe terminal, V0.9.433.0
Steps to reproduce
readsp.zip
Extract, compile and run the attached readsp.c program under the V2 console. This programs exercises directly writing a non-BMP character to the input buffer via
WriteConsoleInputWand reading it back viaReadConsoleW, first with echo enabled and then with it disabled. Run the program with -v (e.g.readsp -v) to show the input key-event records that each step tries to read. It tries a normal key down/up event pair as well as the Alt+Numpad sequence that the console uses for pasted text. The latter uses 6 key events per wide-character and thus 12 key events for a surrogate pair. I included the paste sequence to try to clarify a related issue in which manually pasting a non-BMP character produces a different incorrect result, but it didn't help. I'll discuss that related issue in a comment, in case it's all due to the same underlying issue.Expected behavior
ReadConsoleWshould be able to correctly read supplementary-plane (i.e. non-BMP) characters such as "😞" (U+1F61E), regardless of whether they are typed or pasted into the terminal window, or written directly to the input buffer, or whether echo is enabled. Since the wide-character API uses 16-bit characters, the non-BMP character should be read as a UTF-16 surrogate pair, e.g. U+1F61E should be encoded as {0xD83D, 0xDE1E}.ReadConsoleWworks as expected with the legacy (V1) console. For example:It almost works correctly with Windows Terminal version 0.9.433.0:
Apparently a cooked read under Windows Terminal has a bug in which a non-BMP character gets echoed as two replacement characters, U+FFFD. But at least the
ReadConsoleWresult is correct.Actual behavior
In the output below, not only does the cooked read fail with
ERROR_INVALID_PARAMETER(87) when echo is enabled, but the echoed text contains only the first surrogate code of the surrogate pair, 0xD83D.Since it's not a valid Unicode character, I've replaced this lone surrogate code in the pasted text with the Unicode replacement character, U+FFFD, but the "screen" text, which gets read directly from the screen buffer, shows that the code displayed on the console is 0xD83D.