win32: subprocess encoding

[GoogleCodeExporter]

What steps will reproduce the problem?
1. create a filename in non-utf8, say 'GBK'
2. launch hotwire-shell to browse the file
3. issue a command with filepath, if the encoding for the filesystem is
non-utf8, the filepath is either not found or garbage text

What is the expected output? What do you see instead?
Display the non-utf8 filename correctly.
Convert the encoding between the UI (UTF8) and the native filesystem.

What version of the product are you using? On what operating system?
0.700 on Windows with Python 2.5

Please provide any additional information below.

The encoding for the filename/directory might be not utf8, while the
encoding for GTK is utf8. So the conversion must be done when filenames are
involved.
Browse: native filesystem encoding (sys.getfilesystemencoding()) -> GTK
(utf8), e.g. the output of 'ls'
User command: GTK(utf8) -> native filesystem encoding, e.g. mkdir "你好"

Original issue reported on code.google.com by Zeng.Shi...@gmail.com on 25 Jan 2008 at 7:45

[GoogleCodeExporter]

Ok, hm.  I wonder if there is a way to convince Python to do this conversion 
for us?
 From looking at the docs, it says if you pass a "unicode" object into Python's
functions like "os.listdir", it will return Unicode objects.

I'll try to take a look at the code here and see if I can figure out how this 
should
work.

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 12:59

• Changed state: Accepted

[GoogleCodeExporter]

Ideally we can figure out a way to get this into the testsuite.

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 1:09

[GoogleCodeExporter]

To solve this kind of problems completely, I think it needs to be first worked 
out
what encoding should be used for different components.
Here are the encodings:
1. GTK+ input/output: UTF-8
2. Filenames: sys.getfilesystemencoding()
3. command arguments, sys.stdin.encoding
4. commands: sys.getfilesystemencoding() or sys.stdin.encoding??
5. outputs from commands: sys.stdout.encoding
6. Internal encoding in this program: UTF-8 or unicode always

Assume 'UTF-8' is used for the internal strings:
1. No need to do conversion on string reading/write to GTK+
2. convert to sys.getfilesystemencoding() whenever a path is operated on. i.e. 
in
cases of listdir, move, link, etc. Convert the paths returned from listdir(),
getcwd(), etc. from sys.getfilesystemencoding().
3. all the command arguments are in sys.stdin.encoding, it's the command's
responsibility to decide whether the further encoding conversion is needed or 
not.
4. all outputs are decoded as they are in sys.stdout.encoding
5. The encoding for the command itself to be determined. I have never seen a 
system
has a sys.stdin.encoding different from sys.getfilesystemencoding()

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 7:13

[GoogleCodeExporter]

Can you try running this program for me, and tell me what it prints?

Also could you attach some sample stack traces from things you were trying in 
Hotwire
above?

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 7:52

Attachments:

• testencoding1.py

[GoogleCodeExporter]

If the above program seems to work OK, then what I am guessing is at fault here 
is
that we're passing "str" objects (UTF-8) into the OS functions like listdir 
instead
of unicode objects.  

In hotwire/fs.py, the FilePath object subclasses 'str', not 'unicode'.  This 
will
probably need to change.  

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 7:59

[GoogleCodeExporter]

The output of above is:
C:\tools>testencoding1.py
str listdir: ['\xc4\xe3\xba\xc3']
unicode listdir: [u'\u4f60\u597d']

it seems OK.
Is it working on other systems that passing unicode path to OS functions? I 
haven't
tested that. even listdir returns unicode objects when its argument is unicode,
getcwd() doesn't. it returns the directory encoded in 
sys.getfilesystemencoding().

I still think that a consistent strategy needs to be worked out for encodings.
Otherwise, random hacks will make the code messy.

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 8:20

[GoogleCodeExporter]

For rendering, this patch is needed.
--- ../hotwire-origin/hotwire_ui/renderers/file.py  2008-01-18 10:45:44 -0600
+++ ./hotwire_ui/renderers/file.py  2008-01-24 23:33:08 -0600
@@ -21,6 +21,8 @@

 import gtk, gobject, pango

+import sys
+
 import hotwire
 import hotwire_ui.widgets as hotwidgets
 from hotwire.command import Pipeline
@@ -141,7 +143,7 @@
             text = path[len(self.__basedir)+offset:]
         else:
             text = path
-        cell.set_text(text)
+        cell.set_text(text.decode(sys.getfilesystemencoding()).encode('UTF-8'))

     def _render_size(self, col, cell, model, iter):
         obj = self._file_for_iter(model, iter)

But, for commands involving paths are much more difficult to deal with. I tried 
to
work out a patch, but I failed at the first attempt.

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 8:24

[GoogleCodeExporter]

Ok, so the Python library does seem to work right on your system, which makes
sense...it was far more likely to be a Hotwire bug =)

For os.getcwd(), note that there is a os.getcwdU() function.   However, that's 
mostly
irrelevant because Hotwire never uses os.getcwd() internally; it's not 
threadsafe. 
In the startup script we do os.chdir('/') and don't change it.  Instead there 
is a
"HotwireContext" object that is per-tab/window that holds the cwd.

I totally agree with you about the consistent strategy.  I think the right 
approach
is to internally, use the Python "unicode" object for unicode text as much as
possible, and at the lowest level possible.

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 8:33

[GoogleCodeExporter]

Hm, in that patch you're doing the conversion there at the very last minute 
when we
try to do the display.  But what we should really do instead is make sure that 
the
File object itself has the right data (internally, that has to be UTF-8).

And actually, there was a patch included to fix issue 122 which went in after 
0.700
which should help with File objects, hopefully.

I just committed a patch which is an example of the kind of thing I think we 
need to do:

Committed r870
    M   hotwire/command.py
r870 = 74d56d6805e210cf195942e6e2eec8cc6cb49e52 (git-svn)

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 8:40

[GoogleCodeExporter]

Could you try updating to the latest Subversion?  I made another FilePath change
which may help.

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 8:54

[GoogleCodeExporter]

As of Version 871, I have:

C:\tools\hotwire-shell-read-only>c:\Python25\python.exe ui\hotwire
Running uninstalled, extending PYTHONPATH with: C:\tools\hotwire-shell-read-only

15:07:07 [3228] hotwire.Main WARNING No IPC subsystem available for this platfor
m
C:\tools\hotwire-shell-read-only\hotwire_ui\pixbufcache.py:71: GtkWarning: 
鎵句
笉鍒板浘鏍団€減ython.ico鈥濄€傗€渉icolor鈥濅富棰樹篃娌��
�湁鎵惧埌锛屽彲鑳芥偍闇€
瑕佸厛瀹夎瀹冦€?
鎮ㄥ彲浠ヤ粠涓嬮潰鐨勪綅缃幏寰椾竴涓壇鏈細
        http://icon-theme.freedesktop.org/releases
  return theme.load_icon(name, size, 0)
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 287, in <lam
bda>
    self.__input.connect("notify::text", lambda *args: self.__on_input_changed()
)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 956, in __on
_input_changed
    self.__completions.set_history_search(self.get_active_lang().uuid, curvalue
or None)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\completion.py", line 465, in
 set_history_search
    histitems = map(lambda result: (lang_uuid,result), self.__context.history.se
arch_commands(lang_uuid, histsearch))
  File "C:\tools\hotwire-shell-read-only\hotwire\state.py", line 114, in search_
commands
    for v in cursor.execute(sql, args):
sqlite3.OperationalError: Could not decode to UTF-8 column 'dirpath' with text '
C:/Documents and Settings/Jialai Wang.JIALAIWANG/哈哈'
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 287, in <lam
bda>
    self.__input.connect("notify::text", lambda *args: self.__on_input_changed()
)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 956, in __on
_input_changed
    self.__completions.set_history_search(self.get_active_lang().uuid, curvalue
or None)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\completion.py", line 465, in
 set_history_search
    histitems = map(lambda result: (lang_uuid,result), self.__context.history.se
arch_commands(lang_uuid, histsearch))
  File "C:\tools\hotwire-shell-read-only\hotwire\state.py", line 114, in search_
commands
    for v in cursor.execute(sql, args):
sqlite3.OperationalError: Could not decode to UTF-8 column 'dirpath' with text '
C:/Documents and Settings/Jialai Wang.JIALAIWANG/哈哈'
15:07:12 [3228] hotwire.ui.Shell ERROR Exception in callback
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\logutil.py", line 28, in _exec_
cb
    return func(*args, **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 782, in __on
_input_keypress
    self.__execute()
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 636, in __ex
ecute
    self.execute_pipeline(self.__parsed_pipeline, origtext=text)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 590, in exec
ute_pipeline
    self.context.history.append_command(curlang.uuid, text, self.context.get_cwd
())
  File "C:\tools\hotwire-shell-read-only\hotwire\state.py", line 84, in append_c
ommand
    cursor.execute('''COMMIT''')
OperationalError: cannot commit transaction - SQL statements in progress

The filename "哈哈" is encoded in "CP936".

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 9:10

[GoogleCodeExporter]

Hmm.  I think we may have some badly encoded data now in your sqlite history.  
The
sqlite DB too should be UTF-8.  Can you try removing the database file?  On 
Windows
that's in ~/Application Data/hotwire.

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 9:14

[GoogleCodeExporter]

Also try to save any stack traces you can get.  They help a lot in debugging.

I'll try to reboot to Windows tomorrow probably when I get some spare time and 
see if
I can reproduce any of this.

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 9:21

[GoogleCodeExporter]

Thanks for pointing out the problem. I also guessed it might be caused by 
database
corruption. But I failed to find where the database is. ;)

Now it works in some cases: show/mkdir/cp/mv Chinese filenames/directories. But 
it
doesn't work with external executable files. Say, If I want to run a script 
named
"abc测试.py", it sits in the current directory, and I type in: 
"abc测试.py", the
stack traces are:

C:\tools\hotwire-shell-read-only\hotwire_ui\pixbufcache.py:71: GtkWarning: 
鎵句
笉鍒板浘鏍団€減ython.ico鈥濄€傗€渉icolor鈥濅富棰樹篃娌��
�湁鎵惧埌锛屽彲鑳芥偍闇€
瑕佸厛瀹夎瀹冦€?
鎮ㄥ彲浠ヤ粠涓嬮潰鐨勪綅缃幏寰椾竴涓壇鏈細
        http://icon-theme.freedesktop.org/releases
  return theme.load_icon(name, size, 0)
15:33:04 [1164] hotwire.Completion ERROR Exception in callback
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\logutil.py", line 28, in _exec_
cb
    return func(*args, **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 210, in __
do_async_complete
    result = self.__get_completions(completer, text, cwd)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 205, in __
get_completions
    return CompletionResults(list(completer.completions(text, cwd)))
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 155, in co
mpletions
    if not fname.startswith(text_prefix):
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid dat
a
15:33:34 [1164] hotwire.Completion ERROR Exception in callback
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\logutil.py", line 28, in _exec_
cb
    return func(*args, **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 210, in __
do_async_complete
    result = self.__get_completions(completer, text, cwd)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 205, in __
get_completions
    return CompletionResults(list(completer.completions(text, cwd)))
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 155, in co
mpletions
    if not fname.startswith(text_prefix):
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid dat
a
15:33:37 [1164] hotwire.Completion ERROR Exception in callback
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\logutil.py", line 28, in _exec_
cb
    return func(*args, **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 210, in __
do_async_complete
    result = self.__get_completions(completer, text, cwd)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 205, in __
get_completions
    return CompletionResults(list(completer.completions(text, cwd)))
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 155, in co
mpletions
    if not fname.startswith(text_prefix):
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid dat
a
15:33:39 [4204] hotwire.ui.Shell ERROR Exception in callback
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\logutil.py", line 28, in _exec_
cb
    return func(*args, **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 782, in __on
_input_keypress
    self.__execute()
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 621, in __ex
ecute
    self.__do_parse(throw=True, resolve=True)
  File "C:\tools\hotwire-shell-read-only\hotwire_ui\shell.py", line 932, in __do
_parse
    resolve=resolve)
  File "C:\tools\hotwire-shell-read-only\hotwire\command.py", line 1084, in pars
e
    return Pipeline.parse(text, context=self.__context, resolver=(resolve and se
lf.__resolver or None), **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire\command.py", line 918, in parse

    return Pipeline.create(context, resolver, *tokens)
  File "C:\tools\hotwire-shell-read-only\hotwire\command.py", line 786, in creat
e
    (b, cmdargs) = resolver.resolve(builtin_token.text, context)
  File "C:\tools\hotwire-shell-read-only\hotwire\command.py", line 413, in resol
ve
    for completion in vc.completions(text, context.get_cwd()):
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 155, in co
mpletions
    if not fname.startswith(text_prefix):
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid dat
a
15:33:39 [1164] hotwire.Completion ERROR Exception in callback
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\logutil.py", line 28, in _exec_
cb
    return func(*args, **kwargs)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 210, in __
do_async_complete
    result = self.__get_completions(completer, text, cwd)
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 205, in __
get_completions
    return CompletionResults(list(completer.completions(text, cwd)))
  File "C:\tools\hotwire-shell-read-only\hotwire\completion.py", line 155, in co
mpletions
    if not fname.startswith(text_prefix):
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid dat
a

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 9:34

[GoogleCodeExporter]

Awesome!  Glad to hear we're getting closer.  Ok, I see the issue with this 
one. 
Ultimately the prefix we're using for those completions is coming from
Filesystem.get_path_generator().  The win32 implementation is in
hotwire/sysdep/fs_impl/fs_win32.py:Win32Filesystem, get_path_generator.

We're splitting the PATH variable into byte strings, and we need to be using 
Unicode.
 Can you tell me what the encoding of those strings is?  Should we be using
sys.getfilesystemencoding()?  I'm guessing so, here's a patch to fix it:

Committed r872
    M   hotwire/sysdep/fs_impl/fs_win32.py
    M   hotwire/sysdep/fs_impl/fs_unix.py
r872 = f76f6de7b751fdcaacf47444b37960815d4e6ba8 (git-svn)

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 9:56

[GoogleCodeExporter]

Actually this patch is a bit saner (we should be doing the decoding before the 
split,
not after):

Committed r874
    M   hotwire/sysdep/fs_impl/fs_win32.py
    M   hotwire/sysdep/fs_impl/fs_unix.py
r874 = b0cd6cf993222ac413ecb8a90b436ad9cbc998c9 (git-svn)

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 10:00

[GoogleCodeExporter]

This time there is no exception any more, Wow!
But it complains:
Failed to parse pipeline: No matches for abc中文.py

Actually, I can see it right there with "ls".

Yes, the encoding is sys.getfilesystemencoding()

I'm talking about r874

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 10:06

[GoogleCodeExporter]

So it completes the file name?  Sounds much better if so.

One thing to remember is that Hotwire has more of a Unix-like syntax and 
semantics
than Windows CMD; for example you need to type "./filename.py" to execute 
something
in the current directory.  If that doesn't work, then we may be getting the 
encoding
wrong and so the string comparison is failing?

Does typing "python abc中文.py" work?

How about "py-eval -f abc中文.py"?

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 10:15

[GoogleCodeExporter]

Actually, I tried "./abc中文.py" before I commented.

It doesn't complete the filename either, there might still be some encoding 
errors
somewhere.

By trying "c:/python25/python.exe abc中文.py", it returns:
c:/python25/python.exe: can't open file 'abc中文.py': [Errno 2] No such file 
or directory

by "py-eval -f abc中文.py", it works, but no output is shown. and this time, 
it does
complete the filename.

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 10:24

[GoogleCodeExporter]

Oh, actually...this may not be an encoding issue.  It may be that Hotwire just
doesn't have the right concept of what makes a file executable on Windows, and 
so
it's ignoring it as a possible verb.

The way this works is like this:

in hotwire/sysdep/fs.py:File, function _do_get_xaccess we use the builtin Python
os.access(path, os.X_OK) to see if we can execute it.  But in fs_win32.py, we
override it:

    def _do_get_xaccess(self):
        super(Win32File, self)._do_get_xaccess()
        self.xaccess = self.xaccess and win_exec_re.search(self.path)

I think the reason I added that regular expression was because otherwise we were
picking up lots of random .DLL files as possible verbs, which was ugly.  Maybe 
we
need to give up on that and just show all files that os.access says are 
executable?

So that's one bug.

Now though, if "c:/python25/python.exe abc中文.py" doesn't work, that 
probably means
we're not passing arguments to subprocesses correctly, which probably is
encoding-specific.  If you just name the file abc.py, does it work?

Original comment by cgwalt...@gmail.com on 25 Jan 2008 at 10:40

[GoogleCodeExporter]

yes, it does work by renaming it to abc.py and invoking as 
"c:/python25/python.exe
abc.py"

Original comment by Zeng.Shi...@gmail.com on 25 Jan 2008 at 11:29

[GoogleCodeExporter]

If I comment out the line 
self.xaccess = self.xaccess and win_exec_re.search(self.path)

and run "./abc中文.py", it returns:
C:\tools\hotwire-shell-read-only>c:\Python25\python.exe ui\hotwire
Running uninstalled, extending PYTHONPATH with: C:\tools\hotwire-shell-read-only

17:28:48 [920] hotwire.Main WARNING No IPC subsystem available for this platform

C:\tools\hotwire-shell-read-only\hotwire_ui\pixbufcache.py:71: GtkWarning: 
鎵句
笉鍒板浘鏍団€減ython.ico鈥濄€傗€渉icolor鈥濅富棰樹篃娌��
�湁鎵惧埌锛屽彲鑳芥偍闇€
瑕佸厛瀹夎瀹冦€?
鎮ㄥ彲浠ヤ粠涓嬮潰鐨勪綅缃幏寰椾竴涓壇鏈細
        http://icon-theme.freedesktop.org/releases
  return theme.load_icon(name, size, 0)
17:28:55 [5596] hotwire.Command ERROR Caught exception: [Error 2] The system can
not find the file specified
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\command.py", line 312, in __run

    for result in self.builtin.execute(self.context, *target_args, **kwargs):
  File "C:\tools\hotwire-shell-read-only\hotwire\builtins\sys_builtin.py", line
216, in execute
    subproc = subprocess.Popen(args, **subproc_args)
  File "C:\Python25\lib\subprocess.py", line 593, in __init__
    errread, errwrite)
  File "C:\Python25\lib\subprocess.py", line 793, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
ui\hotwire:202: PangoWarning: pango_layout_get_context: assertion `layout != NUL
L' failed
ui\hotwire:202: PangoWarning: pango_context_get_language: assertion `context !=
NULL' failed
ui\hotwire:202: PangoWarning: pango_context_get_metrics: assertion `PANGO_IS_CON
TEXT (context)' failed
ui\hotwire:202: PangoWarning: pango_font_metrics_get_approximate_char_width: ass
ertion `metrics != NULL' failed
ui\hotwire:202: PangoWarning: pango_font_metrics_get_approximate_digit_width: as
sertion `metrics != NULL' failed
ui\hotwire:202: PangoWarning: pango_layout_set_width: assertion `layout != NULL'
 failed
ui\hotwire:202: PangoWarning: pango_layout_get_extents: assertion `layout != NUL
L' failed

if I rename the file to "abc.py", it shows:
17:31:07 [4716] hotwire.Command ERROR Caught exception: [Error 8] %1 is not a va
lid Win32 application
Traceback (most recent call last):
  File "C:\tools\hotwire-shell-read-only\hotwire\command.py", line 312, in __run

    for result in self.builtin.execute(self.context, *target_args, **kwargs):
  File "C:\tools\hotwire-shell-read-only\hotwire\builtins\sys_builtin.py", line
216, in execute
    subproc = subprocess.Popen(args, **subproc_args)
  File "C:\Python25\lib\subprocess.py", line 593, in __init__
    errread, errwrite)
  File "C:\Python25\lib\subprocess.py", line 793, in _execute_child
    startupinfo)
WindowsError: [Error 8] %1 is not a valid Win32 application

So, this raises another problem: even if the encoding is handled correctly, it 
can't
directly launched via subprocess, since it's a python script that needs to be
interpreted by Python. The proper way may be going through the registry and 
find out
the associated program to open the file. I googled for it, and found that
ShellExecute can invoke the associated program to open the file, but it cann't
redirect the stdout of the subprocess.

Original comment by Zeng.Shi...@gmail.com on 26 Jan 2008 at 12:06

[GoogleCodeExporter]

Ok, there are two bugs here mainly left - testing for file executability, and 
the
encoding.  Let's keep this one for the subprocess encoding one.

I've moved the executability test out into a new issue:
http://code.google.com/p/hotwire-shell/issues/detail?id=128

Original comment by cgwalt...@gmail.com on 26 Jan 2008 at 1:22

• Changed title: win32: subprocess encoding

[GoogleCodeExporter]

While working on another bug, I fixed something which may help with the argument
encoding issue:

Committed r877
    M   hotwire/command.py
r877 = 513f2852bdec32b27a2a37c18f462069f9649c0b (git-svn)

Original comment by cgwalt...@gmail.com on 26 Jan 2008 at 2:45

[GoogleCodeExporter]

hmmm, it seems another package is needed which is not installed on my system.
However, I grab from internet and install. It runs, but nothing changes.

I was testing with r885.

Original comment by Zeng.Shi...@gmail.com on 26 Jan 2008 at 3:25

[GoogleCodeExporter]

See also issue 133 for Linux.  These may be closely related.

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 6:32

[GoogleCodeExporter]

I committed some work on issue 133 for Linux, this may help Windows too:

Committed r891
    M   hotwire/builtins/sys_builtin.py
    M   hotwire_ui/renderers/unicode.py
    M   hotwire_ui/odisp.py
r891 = f0cd00edbad4ae61529e55e7cb5fdd0d653ba89a (git-svn)

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 10:45

[GoogleCodeExporter]

No, it still doesn't work correctly.
c:/Python25/python.exe abc中文.py
Traceback (most recent call last):
  File "C:\tools\hotwire-bzr\hotwire\async.py", line 106, in __do_idle
  File "c:\tools\hotwire-read-only\hotwire_ui\odisp.py", line 368, in __idle_han
dle_output
    odisp.append_object(item, **append_kwargs)
  File "c:\tools\hotwire-read-only\hotwire_ui\odisp.py", line 184, in append_obj
ect
    self.__display.append_obj(obj, **kwargs)
  File "c:\tools\hotwire-read-only\hotwire_ui\renderers\unicode.py", line 295, i
n append_obj
    self.__append_locale_chunk(obj)
  File "c:\tools\hotwire-read-only\hotwire_ui\renderers\unicode.py", line 281, i
n __append_locale_chunk
    decoded = self.__locale_decoder.decode(obj, flush)
UnicodeDecodeError: 'gbk' codec can't decode bytes in position 46-47: illegal mu
ltibyte sequence

Original comment by Zeng.Shi...@gmail.com on 28 Jan 2008 at 2:00

[GoogleCodeExporter]

Doing "import codecs; codecs.lookup('CP936')" gives me the "gbk" encoding, so 
these
must be the same thing.

So is the Python subprocess outputting something different, or are we decoding
incorrectly?  Let's try to get a handle on exactly what data we're getting back 
from
subprocess.  Can you try this test program?

Original comment by cgwalt...@gmail.com on 28 Jan 2008 at 3:56

Attachments:

• testsubproc.py

[GoogleCodeExporter]

With/without locale.setlocale(locale.LC_ALL, '') commented produces the same 
result:
$ /c/Python25/python.exe ./testsubproc.py
Traceback (most recent call last):
  File "./testsubproc.py", line 9, in <module>
    proc = subprocess.Popen([ur'c:\Python25\python.exe', u'abc涓枃.py'], stdou
t=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "C:\Python25\lib\subprocess.py", line 593, in __init__
    errread, errwrite)
  File "C:\Python25\lib\subprocess.py", line 793, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-27: ord
inal not in range(128)

Original comment by Zeng.Shi...@gmail.com on 28 Jan 2008 at 5:04

[GoogleCodeExporter]

That's strange.  Why is it trying to use "ascii"?  <minute or two of Google
searching>  Ah hah:

http://kofoto.rosdahl.net/wiki/UnicodeInPython
"On Windows 2000, the subprocess module does not handle Unicode strings, so 
properly
encoded byte strings must be created for all arguments. XXX: Bug? "

Ok, that's not too hard.  Can you try this one?

Original comment by cgwalt...@gmail.com on 29 Jan 2008 at 12:59

Attachments:

• testsubproc1.py

[GoogleCodeExporter]

No, it doesn't work. it complains:
Traceback (most recent call last):
  File "C:\Documents and Settings\Jialai Wang.JIALAIWANG\testsubproc1.py", line
13, in <module>
    proc = subprocess.Popen(fmt_subproc_args([ur'C:\Python25\python.exe', u'abc
涓枃.py']), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "C:\Documents and Settings\Jialai Wang.JIALAIWANG\testsubproc1.py", line
11, in fmt_subproc_args
    return map(lambda x: x.encode(enc), args)
  File "C:\Documents and Settings\Jialai Wang.JIALAIWANG\testsubproc1.py", line
11, in <lambda>
    return map(lambda x: x.encode(enc), args)
  File "C:\Python25\lib\encodings\cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 3-4: cha
racter maps to <undefined>

Here's what I got:
Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, '')
'English_United States.1252'
>>> 

Maybe I have a strange locale. I live in US, but need to read Chinese. ;)

But, this one works:
#!/usr/bin/python
# -*- coding: utf-8 -*-

import os,sys,subprocess,locale

locale.setlocale(locale.LC_ALL, '')

def fmt_subproc_args(args):
  enc = sys.getfilesystemencoding()
  return map(lambda x: x.encode(enc), args)

proc = subprocess.Popen(fmt_subproc_args([ur'C:\Python25\python.exe', u'abc中
文.py']), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
result = proc.communicate()[0]
print "%r" % (result,)

And the result is:
"Hello, this program is invoked by ['abc\\xd6\\xd0\\xce\\xc4.py']\r\n"

abc中文.py is like:
#!/usr/bin/python
import sys
print "Hello, this program is invoked by %s" % sys.argv

'\\xd6\\xd0\\xce\\xc4' is '中文''s code in 'gbk' encoding.

In my opinion, argv[0](the command itself, a path actually) should be in
sys.getfilesystemencoding(), and other arguments should be in 
sys.stdin.encoding, as
I pointed out in comment #3

Original comment by Zeng.Shi...@gmail.com on 29 Jan 2008 at 2:00

[GoogleCodeExporter]

Ok, this should help then:

Committed r919
    M   hotwire/builtins/sys_builtin.py
    M   ui/hotwire
r919 = 225c4a9a433666c4e91a6c74528287e73806c57c (git-svn)

But I am still confused about the stack trace in comment 28, which is about 
program
output.  The decoding seems to work when I test on Linux with 
LANG=fr_FR.ISO-8859-1.

Hm - random guess - perhaps originally Windows was throwing an error back at us 
which
was in the same encoding we gave to it.  If we pass it the correctly encoded
arguments it might work.

Original comment by cgwalt...@gmail.com on 29 Jan 2008 at 3:36

[GoogleCodeExporter]

hmmmmmm,
I can't test it right now, because:

23:27:40 [3844] hotwire.ui.Shell ERROR Exception in callback
Traceback (most recent call last):
  File "c:\tools\hotwire-win32-trash\hotwire\logutil.py", line 28, in _exec_cb
    return func(*args, **kwargs)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\shell.py", line 784, in __on_inp
ut_keypress
    self.__execute()
  File "c:\tools\hotwire-win32-trash\hotwire_ui\shell.py", line 638, in __execut
e
    self.execute_pipeline(self.__parsed_pipeline, origtext=text)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\shell.py", line 608, in execute_
pipeline
    self.__outputs.add_pipeline(pipeline)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\command.py", line 580, in add_pi
peline
    odisp = MultiObjectsDisplay(self.__context, pipeline)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\odisp.py", line 253, in __init__

    self.append_ostream(pipeline.get_output_type(), None, pipeline.get_output(),
 False)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\odisp.py", line 295, in append_o
stream
    odisp = ObjectsDisplay(otype, self.__context)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\odisp.py", line 50, in __init__
    self.__add_display(output_spec)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\odisp.py", line 58, in __add_dis
play
    self.__display = ClassRendererMapping.getInstance().lookup(output_spec, self
.__context)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\render.py", line 42, in lookup
    return self.__map[cls](context=context)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\renderers\file.py", line 49, in
__init__
    **kwargs)
  File "c:\tools\hotwire-win32-trash\hotwire_ui\render.py", line 115, in __init_
_
    self._setup_view_columns()
  File "c:\tools\hotwire-win32-trash\hotwire_ui\renderers\file.py", line 148, in
 _setup_view_columns
    self.__sync_visible_columns()
  File "c:\tools\hotwire-win32-trash\hotwire_ui\renderers\file.py", line 353, in
 __sync_visible_columns
    self._table.get_column(self._column_info['permissions'][3]).set_visible(pref
s.get_pref('hotwire.ui.render.File.columns.permissions', default=True))
AttributeError: 'NoneType' object has no attribute 'set_visible'

Original comment by Zeng.Shi...@gmail.com on 29 Jan 2008 at 5:31

[GoogleCodeExporter]

This may be fixed in r926.

Original comment by cgwalt...@gmail.com on 29 Jan 2008 at 6:23

[GoogleCodeExporter]

it works now. Cool!

Original comment by Zeng.Shi...@gmail.com on 29 Jan 2008 at 6:55

[GoogleCodeExporter]

The whole thing?  You can execute "c:/python25/python.exe abc中文.py"?  
That's great
news if so, we can finally mark this issue as fixed!

Also, I meant to ask - did you want commit access to the Hotwire Subversion, so 
you
can directly add improvements for win32?

Original comment by cgwalt...@gmail.com on 29 Jan 2008 at 7:21

[GoogleCodeExporter]

Yes, It can be marked as fixed.

I'd like to have the commit access, but I don't think I have enough knowledge 
about
hotwire at the moment. I might ask you for the permission, when I'm confident 
that I
can improve hotwire. Thanks. 

Original comment by Zeng.Shi...@gmail.com on 29 Jan 2008 at 9:30

[GoogleCodeExporter]

Woo!  Thanks a ton for all of your patience and patches helping an american
programmer try to get his software to meet the outside world =)

Original comment by cgwalt...@gmail.com on 29 Jan 2008 at 9:40

• Changed state: Fixed