windows console doesn't print or input Unicode

[a09e2537-b6b9-4978-9d1d-78b1db358cbc]

BPO	1602
Nosy	@malemburg, @mhammond, @terryjreedy, @pfmoore, @amauryfa, @ncoghlan, @pitrou, @giampaolo, @tjguk, @mark-summerfield, @ned-deily, @ezio-melotti, @florentx, @4kir4, @lilydjwg, @berkerpeksag, @vadmium, @eryksun, @zooba, @davispuh
Superseder	bpo-28217: Add interactive console tests
Files	sys_write_stdout.patch unicode2.py doc-patch.diff: Proposed changes to user-visible documentation unicode3.py win_console.patch test_win_console.py streams.py wincontest.py: Example io.TextIOWrapper sublcass using WideCharToMultiByte winconsoleio.diff 1602_2.patch 1602_3.patch 1602_4.patch 1602_5.patch 1602_6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/zooba' closed_at = created_at = labels = ['type-bug', 'expert-unicode', 'OS-windows'] title = "windows console doesn't print or input Unicode" updated_at = user = 'https://github.com/mark-summerfield' ``` bugs.python.org fields: ```python activity = actor = 'THRlWiTi' assignee = 'steve.dower' closed = True closed_date = closer = 'steve.dower' components = ['Unicode', 'Windows'] creation = creator = 'mark' dependencies = [] files = ['19493', '20320', '20363', '23461', '23470', '23471', '36120', '40990', '44094', '44290', '44379', '44409', '44449', '44452'] hgrepos = [] issue_num = 1602 keywords = ['patch'] message_count = 148.0 messages = ['58487', '58621', '58651', '87086', '88059', '88077', '92854', '94445', '94480', '94483', '94496', '108173', '108228', '116801', '120414', '120415', '120416', '120700', '125823', '125824', '125826', '125833', '125852', '125877', '125889', '125890', '125898', '125899', '125938', '125942', '125947', '125956', '126286', '126288', '126303', '126304', '126308', '126319', '127782', '131657', '131854', '132060', '132061', '132062', '132064', '132065', '132067', '132184', '132191', '132208', '132266', '132268', '145898', '145899', '145963', '145964', '146471', '148990', '157569', '160812', '160813', '160897', '161151', '161153', '161308', '161651', '164572', '164578', '164580', '164618', '164619', '170899', '170915', '170999', '185135', '197700', '197751', '197752', '197773', '221175', '221178', '223403', '223404', '223507', '223509', '223945', '223946', '223947', '223948', '223949', '223951', '223952', '224019', '224086', '224095', '224596', '224605', '224690', '227329', '227330', '227332', '227333', '227337', '227338', '227347', '227354', '227373', '227374', '227441', '227450', '228191', '228208', '228210', '233347', '233350', '233916', '233937', '234019', '234020', '234096', '234371', '242884', '254405', '254407', '272596', '272605', '272645', '272662', '272675', '272716', '272718', '272720', '273999', '274449', '274673', '274884', '274906', '274912', '274939', '275003', '275004', '275005', '275157', '275362', '275510', '277047', '277048', '277050'] nosy_count = 38.0 nosy_names = ['lemburg', 'mhammond', 'terry.reedy', 'paul.moore', 'tzot', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'giampaolo.rodola', 'tim.golden', 'mark', 'ned.deily', 'christoph', 'ezio.melotti', 'v+python', 'hippietrail', 'flox', 'THRlWiTi', 'davidsarah', 'santoso.wijaya', 'akira', 'David.Sankel', 'python-dev', 'smerlin', 'lilydjwg', 'berker.peksag', 'martin.panter', 'piotr.dobrogost', 'eryksun', 'Drekin', 'steve.dower', 'wiz21', 'stijn', 'Jonitis', 'gurnec', 'escapewindow', 'dead1ne', 'davispuh'] pr_nums = [] priority = 'high' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = '28217' type = 'behavior' url = 'https://bugs.python.org/issue1602' versions = ['Python 3.6'] ```

[64b73bf9-3de7-49cd-8c73-5a4198eb3429]

Glenn wrote:

So if flush checks that bit, maybe TextIOWriter could just call buffer.flush, and it would be fast if clean and slow if dirty?

Yes. I'll benchmark how much overhead is added by the calls to flush; there's no point in breaking the abstraction boundary of BufferedWriter if it doesn't give a significant performance benefit. (I suspect that it might not, because Windows is very slow at scrolling a console, which might make the cost of flushing insignificant in comparison.)

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

David-Sarah wrote: Windows is very slow at scrolling a console, which might make the cost of flushing insignificant in comparison.)

Just for the record, I noticed a huge speedup in Windows console scrolling when I switched from WinXP to Win7 on a faster computer :) How much is due to the XP->7 switch and how much to the faster computer, I cannot say, but it seemed much more significant than other speedups in other software. The point? Benchmark it on Win7, not XP.

[vstinner]

I done more tests on the Windows console. I focused my tests on output.

To sum up, if we implement sys.stdout using WriteConsoleW() and sys.stdout.buffer.raw using WriteConsoleA():

• print() will not fail anymore on unencodable characters, because the string is no longer encoded to the console code page
• if you set the console font to a TrueType font, most characters will be displayed correctly
• you don't need to change the (console) code page to CP_UTF8 (65001) anymore if you just use print()
• you still need cp65001 if the output (stdout and/or stderr) is redirected or if you use directly sys.stdout.buffer or sys.stderr.buffer

Other facts:

• locale.getpreferredencoding() returns the ANSI code page
• sys.stdin.encoding is the console encoding (GetConsoleCP())
• sys.stdout.encoding and sys.stderr.encoding are the console output code page (GetConsoleOutputCP())
• sys.stdout is not a TTY if the output is redirect, e.g. "python script.py|more"
• sys.stderr is not a TTY if the output is redirect, e.g. "python script.py 2>&1|more" (this example redirects stdout and stderr, I don't know how to redirect only stderr)
• WriteConsoleW() is not affected by the console output code page (GetConsoleOutputCP)
• WriteConsoleA() is indirectly affected by the console output code page: if a string cannot be encoded to the console output code page (e.g. sys.stdout.encoding), you cannot call WriteConsoleA with the result...
• If the console font is a raster font and and the font doesn't contain a character, the console tries to find a similar glyph, or it falls back to the character '?'
• If the console font is a TrueType font, it is able to display most Unicode characters

[vstinner]

unicode3.py replaces sys.stdout, sys.stdout.buffer, sys.stderr and sys.stderr.buffer to use WriteConsoleW() and WriteConsoleA(). It displays also a lot of information about encodings and displays some characters (I wrote my tests for cp850, cp1252 and cp65001).

[vstinner]

win_console.patch: a more complete prototype

• patch the site module to replace sys.stdout and sys.stderr by UnicodeConsole and BytesConsole classes which use WriteConsoleW and WriteConsoleA
• UnicodeConsole inherits from io.TextIOBase and BytesConsole inherits from io.RawIOBase
• Revert the workaround for WriteConsoleA bug from io.FileIO

sys.stdout and/or sys.stderr are only replaced if there are not redirected.

[vstinner]

test_win_console.py: Small script to test win_console.patch. Write some characters into sys.stdout.buffer (WriteConsoleA) and sys.stdout (WriteConsoleW). The test is written for cp850, cp1252 and cp65001 code pages.

[vstinner]

I added a cp65001 codec to Python 3.3: see issue bpo-13216.

[f16fd4a9-cdd1-4f2c-9715-4d115d26a82e]

The underlying cause of Python's write exceptions with cp65001 is:

The ANSI C write() function as implemented by the Windows console returns the number of _characters written rather than the number of _bytes, which Python reasonably interprets as a "short write error". It then consults errno, which gives the effectively random error message seen.

This can be bypassed by using os.write(sys.stdout.fileno(), utf8str), which will a) succeed and b) return a count \<= len(utf8str).

With os.write() and an appropriate font, the Windows console will correctly display a large number of characters.

Possible workaround: clear errno before calling write, check for non-zero errno after. The vast majority of (non-Python) applications never check the return value of write, so don't encounter this problem.

[vstinner]

The issue bpo-14227 has been marked as a duplicate of this issue. Copy of msg155149:

This is on Windows 7 SP1. Run 'chcp 65001' then Python from a console. Note the extra characters when non-ASCII characters are in the string. At a guess it appears to be using the UTF-8 byte length of the internal representation instead of the character count.

Python 3.3.0a1 (default, Mar  4 2012, 17:27:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print('hello')
hello
>>> print('p\u012bny\u012bn')
pīnyīn
n
>>> print('\u012b'*10)
īīīīīīīīīī
�īīīī
�ī

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

Has something incompatible changed between 3.2.2 and 3.2.3 with respect to this bug?

I have a program that had an earlier version of the workaround (Michael's original, I think), and it worked fine, then I upgraded from 3.2.2 to 3.2.3 due to testing for bpo-14811 and then the old workaround started complaining about no attribute 'errors'.

So I grabbed unicode3.py, but it does the same thing:

AttributeError: 'UnicodeConsole' object has no attribute 'errors'

I have no clue how to fix this, other than going back to Python 3.2.2...

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

Oh, and is this issues going to be fixed for 3.3, so we don't have to use the workaround in the future?

[terryjreedy]

Glenn, I do not know what you are using the interactive interpreter for, but for the unicode BMP, the Idle shell generally works better. I only use CommandPrompt for cross-checking behavior.

[giampaolo]

Not sure whether a solution has already been proposed because the issue is very long, but I just bumped into this on Windows and come up with this:

from __future__ import print_function
import sys

def safe_print(s):
    try:
        print(s)
    except UnicodeEncodeError:
        if sys.version_info >= (3,):
            print(s.encode('utf8').decode(sys.stdout.encoding))
        else:
            print(s.encode('utf8'))

safe_print(u"\N{EM DASH}")

Couldn't python do the same thing internally?

[64b73bf9-3de7-49cd-8c73-5a4198eb3429]

Giampaolo: See #msg120700 for why that won't work, and the subsequent comments for what will work instead (basically, using WriteConsoleW and a workaround for a Windows API bug). Also see the prototype win_console.patch from Victor Stinner: #msg145963

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

I actually had to go back to 3.1.2 to get it to run, I guess I had never run with Unicode output after installing 3.2. So it isn't an incompatibility between 3.2.2 and 3.2.3, but more likely a change between 3.1 and 3.2 that invalidates this patch and workaround. At least it is easier to keep 3.1.x and 3.2.x on the same system!

Terry, applications for non-programmers that want to emit Unicode on the console... so the IDLE shell isn't appropriate.

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

A little more empirical info: the missing "errors" attribute doesn't show up except for input. print works fine.

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

For the win_console.patch, it seems like adding the line

self.errors='strict'

inside UnicodeOutput.__init__ resolves the problem with input causing exceptions.

Not sure if the sys_write_stdout.patch has the same sort of problem. Sure home this issue makes it into 3.3.

[terryjreedy]

3.3b0, Win7, 64 bit. Original test script stops at File "C:\Programs\Python33\lib\encodings\cp437.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_map)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\x80' in position 6:

I am slightly puzzled because cp437 is an extended ascii codepage and there *is* a character for 0x80 https://en.wikipedia.org/wiki/Code_page_437

If I add .encode('latin1'), it does not print the pentagon for 0x7e, but does print \x7e to \xff.

Someone wrote elsewhere that 3.3 could use cp65001. True?

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

My fix for this "errors" error, might be similar to what is needed for bpo-12967, although I don't know if my fix is really correct... just that it gets past the error, and 'strict' is the default for TextIOWrapper.

I'm not at all sure why there is now (since 3.2) an interaction between input on stdin and the particulars of the output class for stdout. But I'm not at all an expert in Python internals or Python IO.

I'm not sure whether or not you applied the patch to your b0, if not, that is what I'm running, too... but using the win_console.patch as supporting code. The original test script didn't use the supporting code.

If you did patch your b0 bwith unicode3.py, then you shouldn't need to do a chcp to write any Unicode characters; someone reported that doing a chcp caused problems, but I don't know how to apply the patch or build a Python with it, so can't really test all the cases. Victor did add a cp65001 codec using a different issue, not sure how that is relevant here, other than for the tests he wrote.

[terryjreedy]

I was reporting stock, as distributed 3.3b0.

Is unicode3.py something to run once or import in each app that wants unicode output? Either way, if it is possible to fix the console, why is it not distribute it with the fix?

Terry, applications for non-programmers that want to emit Unicode on the console... so the IDLE shell isn't appropriate.

Someone just posted on python-list about a problem with that.

Hmm. Maybe IDLE should gain a batch-mode console window -- basically a stripped down version of the current shell -- a minimal auto-gui for apps.

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

Terry said: Is unicode3.py something to run once or import in each app that wants unicode output?

I say: The latter... import it.

Terry said: Either way, if it is possible to fix the console, why is it not distribute it with the fix?

I say: Not sure what you are asking here. Yes it is possible to fix the console, but this fix depends on the version-specific internals of the Python IO system... so unicode3.py works with Python 3.1, but not Python 3.2 or 3.3. I haven't tested to see if my patched unicode3.py still works on Python 3.1 (I imagine it would, due to the nature of the fix just adding something that Python 3.1 probably would ignore.

So my opinion is the fix is better done inside Python than inside the application.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

Hello, I'm trying to handle Unicode input and output in Windows console and found this issue. Will this be solved in 3.3 final? I tried to write a solution (file attached) based on solution here – rewriting sys.stdin and sys.stdout so it uses ReadConsoleW and WriteConsoleW.

Output works well, but there are few problems with input. First, the Python interactive interpreter actually doesn't use sys.stdin but standard C stdin. It's implemented over file pointer (PyRun_InteractiveLoopFlags, PyRunInteractiveOneFlags in pythonrun). But still the interpreter uses sys.stdin.encoding (assigning sys.stdin something, that doesn't have encoding==None freezes the interpreter). Wouldn't it make more sense if it used sys.\_stdin__.encoding?

However, input() (which uses stdin.readline) works as expected. There's a small problem with KeyboardInterrupt. Since signals are processed asynchronously, it's raised at random place and it behaves wierdly. time.sleep(0.01) after the C call works well, but it's an ugly solution.

When code.interact() is used instead of standard interpreter, it works as expected. Is there a way of changing the intepreter loop? Some hook which calls code.interact() at the right place? The patch can be applied in site or sitecustomized, but calling code.iteract() there obviously doesn't work.

Some other remarks:

• When sys.stdin or sys.stdout doesn't define encoding and errors, input() raises TypeError: bad argument type for built-in operation.
• input() raises KeyboardInterrupt on Ctrl-C in Python 3.2 but not in Python 3.3rc2.

[vstinner]

Will this [issue] be solved in 3.3 final?

No. It would be an huge change and the RC2 was already released. No new feature are accepted after the version 3.3.0 beta 1: http://www.python.org/dev/peps/pep-0398/

I'm not really motivated to work on this issue, because it is really hard to get something working in all cases. Using ReadConsoleW/WriteConsoleW helps, but it doesn't solve all issues as you said.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

I have finished a solution working for me. It bypasses standard Python interactive interpreter and uses its own repl based on code.interact(). This repl is activated by an ugly hack since PYTHONSTARTUP doesn't apply when some file is run (python -i somefile.py). Why it works like that? Startup script could find out if a file is run or not. If anybody knows how to get rid of time.sleep() used for wait for KeyboardInterrupt or how to get rid of PromptHack, please let me know. The "patch" can be activated by win_unicode_console_2.enable(change_console=True, use_hack=True) in site or sitecustomize or usercustomize.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

Hello. I have made a small upgrade of the workaround. • win_unicode_console.enable_streams() sets sys.stdin, stdout and stderr to custom filelike objects which use Windows functions ReadConcoleW and WriteConsoleW to handle unicode data properly. This can be done in sitecustomize.py to take effect automatically.

• Since Python interactive console doesn't use sys.stdin for getting input (still don't know reason for this), there is an alternative repl based on code.interact(). win_unicode_console.IntertactiveConsole.enable() sets it up. To set it up automatically, put the enabling code into a startup file and set PYTHONSTARTUP environment variable. This works for interactive session (just running python with no script).

• Since there is no hook to run InteractiveConsole.enable() when a script is run interactively (-i flag), that is after the script and before the interactive session, I have written a helper script i.py. It just runs given script and then enters an interactive mode using InteractiveConsole. Just put i.py into site-packages and run "py -m i script.py arguments" instead of "py -i script.py arguments".

It's a shame that in the year 2013 one cannot simply run Python console on Windows and enter Unicode characters. I'm not saying it's just Python fault, but there is a workaround on Python side.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

Hello again. I have rewritten the custom stdio objects and implemented them as raw io reading and writing bytes in UTF-16-LE encoding. They are then wrapped in standard BufferedReader/Writer and TextIOWrapper objects. This approach also solves a bug of wrong string length given to WriteConsoleW when the string contained supplementary character. Since we are waiting for Ctrl-C signal to arrive, this implmentation doesn't suffer from http://bugs.python.org/issue18597 . It seems to work when main script is executed however it doesn't work in Python interactive REPL since the REPL doesn't use sys.stdin for input. However it uses its encoding which results in mess when sys.stdin is changed to object with different encoding like UTF-16-LE. See http://bugs.python.org/issue17620 .

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

Hi Drekin. Thanks for your work in progressing this issue. There have been a variety of techniques proposed for this issue, but it sounds like yours has built on what the others learned, and is close to complete, together with bpo-17620.

Is this in a form that can be used with Python 3.3? or 3.4 alpha? Can it be loaded externally from a script, or must it be compiled into Python, or both?

I've been using a variant of davidsarah's patch since 2 years now, but would like to take yours out for a spin. Is there a Complete Idiot's guide to using your patch? :)

[terryjreedy]

From reading the module, import stream; stream.enable() replaces sys.stdin/out/err with new classes.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

Glenn Linderman: Yes I have built on what the others learned. For your question, I made it and tested it in Python 3.3, it should also work in 3.4 and what I've tried, it actually works. As Terry J. Reedy says you can just load the module and enable the streams. I do this automatically on startup using sitecustomize. However as I said currently this meeses up the interactive session because of http://bugs.python.org/issue17620 . I have made some workaround – custom REPL built on stdlib module code. And also a helper script which runs the main script and then runs the custom REPL (I couldn't find any stadard hook which would run the custom REPL). I'm uploding full code. I will delete it if this isn't appropriate place.

Things like this could be fixed more easily if more core interpreter logic took place in stdlib. E. g. the code for interactive REPL. Few days ago I started some discussion on python ideas: https://mail.python.org/pipermail/python-ideas/2013-August/023000.html .

[ncoghlan]

The fact Unicode doesn't work at the command prompt makes it look like Unicode on Windows just plain doesn't work, even in Python 3. Steve, if you (or a colleague) could provide some insight on getting this to work properly, that would be greatly appreciated.

[zooba]

My understanding is that the best way to write Unicode to the console is through WriteConsoleW(), which seems to be where this discussion ended up. The only apparent sticking point is that this would cause an ordering incompatibility with stdout.write(); stdout.buffer.write(); stdout.write().

Last I heard, the official "advice" was to use PowerShell. Clearly everyone's keen to jump on that... (I'm not even sure it's an instant fix either - PS is a much better shell for file manipulation and certainly handles encoding better than type/echo/etc., but I think it will still go back to the OEM CP for executables.)

One other point that came up was UTF-8 handling after redirecting output to a file. I don't see an issue there - UTF-8 is going to be one of the first guesses (with or without a BOM) for text that is not UTF-16, and apps that assume something else are no worse off than with any other codepage.

So I don't have any great answers, sorry. I'd love to see the defaults handle it properly, but opt-in scripts like Drekin's may be the best way to enable it broadly.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

I have made some updates in the streams code. Better error handling (getting errno by GetLastError() and raising exception when zero bytes are written on non-zero input). This prevents the infinite loop in BufferedIOWriter.flush() when there is odd number of bytes (WriteConsoleW accepts UTF-16-LE so only even number of bytes is written). It also prevents the same infinite loop when the buffer is too big to write at once (see http://bugs.python.org/issue11395 ). The limit of 32767 bytes was added to raw write.

[vstinner]

@Drekin: Please don't send ZIP files to the bug tracker. It would be much better to have a project on github, Mercurial or something else, to have the history of the source code. You may try tp list all people who contributed to this code.

You may also create a project on pypi.python.org to share your code. This bug tracker is not the best place for that.

When the code will be consider mature (well tested, widely used), we can try to integrate it into Python.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

@Victor Stinner: You are right. So I did it. Here are the links to GitHub and PyPI: https://github.com/Drekin/win-unicode-console, https://pypi.python.org/pypi/win_unicode_console.

I also tried to delete the files, but it seems that it is only possible to unlink a file from the issue, but the file itself remains. Is it possible to manage the files?

[ncoghlan]

Thanks Drekin - I'll point folks to your project as a good place to provide initial feedback, and if that seems promising we can look at potentially integrating the various fixes into Python 3.5

[a09e2537-b6b9-4978-9d1d-78b1db358cbc]

I used pip to install the win_unicode_console package on windows 7 python 3.3.

It works but wouldn't freeze with cxfreeze because there's no \_init__.py file in the win_unicode_console directory.

[ncoghlan]

Hmm, I'm not sure if that would be a bug in cxFreeze or CPython - I don't think we've tried freezing or zipimporting namespace packages... (either way, adding the __init__.py to win_unicode_console would likely be the quickest fix)

[vstinner]

Since there is now an external project fixing the support of Windows console, I suggest to close this issue as "wontfix". In a few months, if we get enough feedback on this project, we may reconsider integrating it into Python. What do you think?

https://pypi.python.org/pypi/win_unicode_console.

I used pip to install the win_unicode_console package ...

Please don't use Python bug tracker to report bugs to the package.

[ncoghlan]

The poor interaction with the Windows command line is still a bug in CPython - we could mark it closed/later but I don't see any value in doing so.

I see Drekin's win_unicode_console module as similar to my own contextlib2 - used to prove the concept, and perhaps iterate on some of the details, but the ultimate long term solution is to fix CPython itself.

[vstinner]

The poor interaction with the Windows command line is still a bug in CPython - we could mark it closed/later but I don't see any value in doing so.

I don't see any value in keeping the issue open since nobody worked on it last 7 years. I just want to make it clear that we will *not* fix this issue.

Well, in fact I spent a lot of hours trying to find a way to fix the issue, and my conclusion is that it's not possible to handle correctly Unicode (input and output) in a Windows console. Please read the whole issue for the detail.

The win_unicode_console project may improve the Unicode support, but I'm convinced that it still has various issues because it is just not possible to handle all cases.

A workaround is to not use the Windows console, but use IDLE or another shell... Try maybe PowerShell. But PowerShell has at least an issue with the code page 65001 (Microsoft UTF-8): see the issue bpo-21927.

[ncoghlan]

Based on Steve's last post, the main challenge is that the IO model assumes a bytes-based streaming API - it isn't really set up to cope with a UTF-16 buffering layer.

However, that's not substantially different from the situation when the standard streams are replaced with StringIO objects, and they don't have an underlying buffer object at all. That may be a suitable model for Windows console IO as well - present it to the user in a way that doesn't expose an underlying bytes-based API at all.

Now, it may not be feasible to implement this until we get the startup code cleaned up, but I'm not going to squash interest in improving the situation when it's one of the major culprits behind the "Unicode is even more broken in Python 3 than it is in Python 2" meme.

[ncoghlan]

Changing targets to Python 3.5, since this is almost certainly going to be too invasive for a maintenance release.

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

This bug deserves to stay open with its high priority (for whatever good that does these last seven years, although I appreciate all the efforts put forth, and have been making heavy use of the workarounds in the patches), because when working with Unicode data in programs, even exception messages are not properly displayed... instead, they cause a secondary exception of not being able to display the data of the original exception to the console.

And writing Unicode data to the console as part of an interactive or command line program has to either be done with the hopes that the data only includes characters in the console, to avoid the failures, or with lots of special encoding calls and character substitutions for code points not in the console repertoire. Remember that the console is supposed to be human readable, not encoded numerically as ascii() would do.

ascii() is sort of OK for for exception messages, but since that doesn't happen by default, the initial message to the console with Unicode data often doesn't appear, and an extra repetition after a failed message and a rework of the message parameters is required, which impedes productivity.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

I have deleted all my old files and added only my current implementation of the stream objects as the only relevant part to this issue.

@Mark Summerfield: I have added __init__.py to the new version of win_unicode_console. If there is any problem, you can start an issue on project GitHub site or contact me.

@Victor Stinner, @Nick Coghlan: What's wrong with looking on Windows wide strings as on UTF-16-LE encoded bytes and building the raw stream objects around this?

[ncoghlan]

Drekin, you're right, that's a much better way to go, I just didn't think it through :)

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

To ensure that we're all talking about the same thing, is everybody using the /u unicode output option or /a ansi (which I'm assuming is the default) when running cmd?

[b5a9ce10-d67f-478f-ab78-b08d099eb753]

Mark, the /U and /A switches to CMD only affect (as the help messages say) the output of internal CMD commands. So they would only affect interoperability between internal command output piped to a Python program. The biggest issue in this bug, however, is the output of Python programs not being properly displayed by the console window (often thought of or described as the CMD shell window).

While my biggest concerns have been with output, I suppose input can be an issue also, and running the output of echo, or other internal commands, into Python could be an issue as well. I have pasted a variety of data into Python programs beyond ASCII, but I'm not sure I've gone beyond ANSI or beyond Unicode BMP. Obviously, once output is working properly, input should also be tested and fixed, although I think output is more critical.

With the impetus of your question... I just took some text supplied in another context that has a bunch of characters from different repertoires, including non-BMP, and tried to paste it into the console window. Here is the text:

こんにちは世界 - fine on Linux, all boxes on Windows (all boxes in Chrome on Linux too) مرحبا، العالم! - fine on Linux and Windows 안녕하세요, 세계! - fine on Linux, just boxes and punctuation on Windows (likewise in Chrome) Привет, мир! - fine on Linux and Windows Αυτή είναι μια δοκιμή - fine on both, but Google Translate has a problem with this! It returned "Hello, world!" as the Greek for "Hello, world!"... so I tried again with "This is a test". 𝓗𝓮𝓵𝓵𝓸, 𝔀𝓸𝓻𝓵𝓭! - not actually a language, but this is astral In the console window, which I have configured using the Consolas font, the glyphs for the non-ASCII characters in the first two and last lines were boxes... likely Consolas doesn't support those characters. I had written a Python equivalent of "echo", including some workarounds originally posted in this issue, and got exactly the same output as input, with no errors produced. So it is a bit difficult to test characters outside the repertoire of whatever font is configured for the console window. Perhaps someone that has Chinese or Korean fonts configured for their console window could report on further testing of the above or similar strings.

[90ff3b65-5c08-4f89-bb59-3f03c3cccbb5]

I think that boxes are ok, it's just missing font. Without active workaroud there is just UnicodeEncodeError (with cp852 for me). There is problem with astral characters – I'm getting each box twice. It is possible that Windows console doesn't handle astral characters at all – it doesn't interpret surrogate pairs.

[9b76dcf1-c6e5-4017-9e13-b31dcebd7cc6]

I don't know if this is 100% related, but here I go. Here's a session in a windows console (cmd.exe) :

Microsoft Windows [Version 6.1.7601] Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\stc>chcp 65001 Active code page: 65001

C:\Users\stc>\PORT-STCA2\opt\python3\python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print '€'

C:\Users\stc>

So basically, the python interpreters just quits without any message. Windows doesn't comply about python crashing though...

Best regards,

Stefan

[9b76dcf1-c6e5-4017-9e13-b31dcebd7cc6]

In my previous comment, I've shown :

print '€'

which is not valid python 3.4.1 (don't why the interpreter didn't complaing though). So I tested again with missing parenthesis added :

C:\PORT-STCA2\pl-PRIVATE\horse>chcp 65001 Active code page: 65001

C:\PORT-STCA2\pl-PRIVATE\horse>python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print("€")

C:\PORT-STCA2\pl-PRIVATE\horse>echo %PROCESSOR_IDENTIFIER% Intel64 Family 6 Model 42 Stepping 7, GenuineIntel

Exactly the same behaviour.