Replacing the streams breaks IPython

[pfmoore]

In the IPython interactive console, if I type

import win_unicode_console.streams
win_unicode_console.streams.enable()

I get an infinite stream of errors, all of which appear to be as follows:

ArgumentError                             Traceback (most recent call last)
C:\Users\uk03306\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\terminal\interactiveshell.py in raw_input(self, prompt)
    588
    589         try:
--> 590             line = py3compat.str_to_unicode(self.raw_input_original(prompt))
    591         except ValueError:
    592             warn("\n********\nYou or a %run:ed script called sys.stdin.close()"

C:\Users\uk03306\AppData\Local\Continuum\Anaconda3\lib\site-packages\win_unicode_console\streams.py in write(self, b)
     84                 code_units_written = c_int()
     85
---> 86                 retval = WriteConsoleW(self.handle, buffer, code_units_to_be_written, byref(code_units_written), None)
     87                 bytes_written = 2 * code_units_written.value
     88

ArgumentError: argument 4: <class 'TypeError'>: expected LP_c_ulong instance instead of pointer to c_long

I'm sure this counts as just another case of "the module doesn't play nicely with the interactive REPL" (that same enable call fails in a normal Python console, or one with just pyreadline installed, as well, but not quite as badly...), but I thought I'd report it here just in case there was something else going on. I have pyreadline installed with IPython, as recommended, and that does some fairly complicated low-level console hacking which might interact badly with this module.

[Drekin]

Thank you for report. It seems it's problem in interaction with pyreadline. Just by installing pyreadline, my REPL stops to work. I'll investigate.

[Drekin]

The errors you are seeing are a small incompatibility between win_unicode_console and pyreadline. Both use WriteConsoleW function and pyreadline also sets its ctypes signature. It declares the fourth parameter as pointer to c_ulong, but win_unicode_console passes pointer to c_long. This can be easily fixed.

However it is still true that replacing the streams breaks IPython, which might be IPython/readline bug. (streams.enable() fails in normal Python console, because standard REPR incorrectly uses sys.stdin.encoding) It is not clear how IPython console interacts with sys.std* streams. I would think it doesn't use them at all, since both interactive input and ouput is handled directly to console to provide colors and autocompletition. But this isn't the case: IPython console handles Unicode output ("\N{greek small letter alpha}" shows "α" as repr) but not Unicode input (typing "α" runs help since it is transformed to "?" at low-level – this is same for normal Python). And changing streams messes IPython console in wierd way: changing only stderr works, chaning only stdout makes prompt disappear but otherwise works, changing only stdin prints color instructions for prompt as text and Unicode input works but auto-completition doesn't.

[pfmoore]

Ah, yes. I've seen that issue before. MSDN says that parameter has type LPDWORD, so it looks like pyreadline is right here.

It looks like IPython (actually, I think the code is in PyReacLine, see https://github.com/pyreadline/pyreadline/blob/master/pyreadline/console/ansi.py) wraps sys.stdout to get ANSI support (there are other packages, notably colorama, that do this, too). I think those packages are always going to have issues, because they are trying to do basically the same thing. This is why this sort of support really belongs in the core, so that we don't get clashes like this. (Although colorama seems quite happy to work alongside win_unicode_console).

[Drekin]

The c_ulong part is fixed in the new release. I'll ask on IPython tracker about the interaction with sys.std*. At least changing sys.stdout = 2 produces infinite stream of errors, which may be bug. I'm closing this for now.

[Drekin]

Aside: I found out that Unicode input in IPython console can be fixed by following trick:

>>> pyreadline.unicode_helper.pyreadline_codepage = "utf-8"

>>> class A:
...    encoding = "utf-8"
...    errors = "strict"
...    def fileno(self): return 0

>>> sys.stdin = A()

The point is that pyreadline is fully capable of handling Unicode in Windows terminal, but it has to encode input using sys.stdin.encoding which may not handle Unicode. I'll add options to use win_unicode_console with pyreadline using its capabilities.

[pfmoore]

Cool. Given that pyreadline hooks into the special case in the Python interpreter to load readline if present, I wonder if this means that it's possible to get full unicode support in the standard interactive interpreter by using win_unicode_console and pyreadline?

[pfmoore]

Hmm, sadly not...

>>> import pyreadline.unicode_helper
>>> pyreadline.unicode_helper.pyreadline_codepage = "utf-8"
>>> import win_unicode_console.streams
>>> win_unicode_console.streams.enable()
>>> print("\N{Snowman}")
  File "<stdin>", line 0

    ^
SyntaxError: 'utf-16-le' codec can't decode byte 0x0a in position 20: truncated data
>>>

[Drekin]

Yes, it would be possible and I plan to do it. But there is another problem: the fact that standard interactive intepreter cannot handle UTF-16-LE encoded stdin. See http://bugs.python.org/issue17620#msg226098. I plan to add another wrapper around my stdin object which transcodes between UTF-16-LE and UTF-8. The plan is that you would be able to get Unicode support just by using win_unicode_console – it will install its own readline hook, or if you have pyreadline, win_unicode_console will use pyreadline's hook so you'll get both Unicode and features like tab-completion.

[pfmoore]

Ah, I see. That's fantastic news.

[Drekin]

The features are implemented in the new release.

[pfmoore]

Nice :-) I'll give it a try in the next few days.