Open d-01 opened 5 years ago
Thank you for the detailed report! My uneducated guess would be that our python repl_process abstraction expects utf-8 but powershell by default uses utf-16, or perhaps that we incorrectly do increment decoding in the kernel. The kernel itself is relative small, I think you should have no troubles debugging it with changing the kernel code. I would not have time to do it any time soon myself, but I'm happy to help you navigate the code and code review any changes.
Problem solved:
--- a/subprocess_repl.py.orig
+++ b/subprocess_repl.py
@@ -9,10 +9,16 @@ import os
import sys
import re
import signal
+import locale
from subprocess import Popen
from codecs import getencoder, getincrementaldecoder
PY3 = sys.version_info[0] == 3
+# On Windows encoding expected to be something like 'cp1252' (en) or 'cp1251' (ru)
+# depending on system-wide "System locale" setting.
+# Path to setting: Region and Language -> Administrative (tab) ->
+# -> Language for non-Unicode programs -> Change system locale...
+ENCODING = locale.getpreferredencoding()
if os.name == 'posix':
POSIX = True
@@ -23,8 +29,8 @@ else:
class SubprocessRepl(object):
def __init__(self, cmd):
- self.encoder = getencoder('utf8')
- self.decoder = getincrementaldecoder('utf8')()
+ self.encoder = getencoder(ENCODING)
+ self.decoder = getincrementaldecoder(ENCODING)()
self.popen = Popen(cmd, bufsize=1,
stderr=subprocess.STDOUT, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
if POSIX:
@@ -60,7 +66,7 @@ class SubprocessRepl(object):
si.flush()
def reset_decoder(self):
- self.decoder = getincrementaldecoder('utf8')()
+ self.decoder = getincrementaldecoder(ENCODING)()
def read(self):
"""Reads at least one decoded char of output"""
Nice! @d-01 would you mind to send a pull request?
I have encountered a problem with cyrillic text encoding. From Windows Explorer:
From powershell console:
From Jupyter Notebook:
I have found a workaround, but not sure how to apply this to fix the problem:
Environment information:
The version of the notebook server is: 5.6.0 The server is running on this version of Python: Python 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] Kernel info:
What else I've tried so far:
$OutputEncoding
global variable[console]::OutputEncoding
[console]::InputEncoding
chcp 866
– doing nothing tocmd /cdir
andGet-ChildItem
/ls
outputchcp 65001
– fixescmd /cdir
but notGet-ChildItem
/ls
outputStandard kernel (IPython 6.5.0) works fine: In:
Out:
From powershell console:
From Jupyter Notebook: