IOError: [Errno 0] Error on Windows

[heitorsilva]

Hello!

I'm using jrnl, on which I had some problems to create records with accents. After solving that problem, I'm having a strange problem with colorama, which produces IOError: [Errno 0] Error on the method write_plain_text of ansitowin32.py file, as you can see below:

Also, take a look at my original Issue, on jrnl: https://github.com/maebert/jrnl/issues/391#issuecomment-228336524

This error only happens when I use chcp 65001 command, to use UTF-8 on the cmd.exe terminal, and the file that was read, is in utf-8encoding.

I'm using Windows 7 (64-bits), with Brazilian Portuguese as main language (which usually uses Latin-1 as encoding). The characters on cmd.exe terminal are being treated as ascii, always.

Any thoughts?

[SteveDoyle2]

I think I figured this out. I'm using powershell on windows 10 and can consistently reproduce this when I'm running a long process and scroll up (and my guess is introducing an unexpected character). When I let go, it crashes.

Traceback (most recent call last):
  File "c:\my_code.py", line 189, in run_lots_of_files
    sys.stderr.write('%i  %s' % (npass, abs_filename))
  File "C:\Anaconda2\lib\site-packages\colorama\ansitowin32.py", line 40, in write
    self.__convertor.write(text)
  File "C:\Anaconda2\lib\site-packages\colorama\ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "C:\Anaconda2\lib\site-packages\colorama\ansitowin32.py", line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "C:\Anaconda2\lib\site-packages\colorama\ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
IOError: [Errno 0] Error

[robhagemans]

These weird errors are a known issue in Python 2 on Windows with CP65001, see e.g. here: https://bugs.python.org/issue1602#msg148990 :

The ANSI C write() function as implemented by the Windows console returns the number of characters written rather than the number of bytes, which Python reasonably interprets as a "short write error". It then consults errno, which gives the effectively random error message seen.

To summarise the little I understand of it, codepage 65001 in Windows is basically broken in many subtle ways and there is no proper UTF-8 support on the Windows console. The real solution is to use Windows's UTF-16 system API calls for all Unicode strings, which I think is what Python 3 does when writing str to the console. Unfortunately, on Python 2 this can't be implemented for compatibility reasons.

For my own project I'm working around this with an extension module that uses WriteConsoleW in the UTF-16 Windows API instead of writing to sys.stdout - see here: https://github.com/robhagemans/pcbasic/blob/master/pcbasic/compat/win32_console.c. Perhaps this can also be done using ctypes. I think the issue will not occur in Python 3, though.