doesn't support output in non-UTF8 locale

[GoogleCodeExporter]

program output in non-UTF8 locale (for instance, when system is still using
a non-UTF8 locale) is discarded instead of being displayed :

try :
LC_ALL=fr_FR.ISO-8859-15 /sbin/ifconfig

you won't see any output in hotwire

try :
LC_ALL=fr_FR.UTF-8 /sbin/ifconfig

output is visible

Original issue reported on code.google.com by fcro...@gmail.com on 27 Jan 2008 at 4:18

[GoogleCodeExporter]

This issue seems related to issue 125, although that one is for Windows.

Does gnome-terminal/bash work in this locale?  I'm wondering if there is some 
magic
we can pass to the Python subprocess module to make it recode output back into 
UTF-8
for us, or if we have to do that manually.

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 6:03

• Changed state: Accepted

[GoogleCodeExporter]

Yes, vte/gnome-terminal doesn't have any issue with this locale (we've been 
using
this for years, way before UTF-8 became the norm).

I think python is expecting UTF-8 output (somehow).

I'm getting this output when trying to launch ifconfig :

usr/lib/python2.5/site-packages/hotwire_ui/renderers/unicode.py:285: GtkWarning:
gtk_text_buffer_emit_insert: assertion `g_utf8_validate (text, len, NULL)' 
failed
  buf.insert(buf.get_end_iter(), start and obj[start:] or obj)

I guess you should call g_locale_to_utf8 (of its equivalent in python)

Original comment by fcro...@gmail.com on 27 Jan 2008 at 6:11

[GoogleCodeExporter]

So to correctly handle this locale, what we would have to do I/O to subprocesses
using the locale enocding, as well as ensure that files we create are in that
encoding as well?

For example, if you do "cat foo.txt > foo2.txt", foo2.txt should also be in
ISO-8859-15?  Remember that while this looks like Unix shell syntax, Hotwire is
actually parsing every line of foo.txt into UTF-8 unicode objects internally, 
because
that also lets you do: 
"cat foo.txt | py-map 'it.upper()' > foo-uppercase.txt".

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 6:37

[GoogleCodeExporter]

I'd say yes, hotwire should be "locale" agnostic, since most users don't know 
how to
handle such things. It is becoming less and less an issue but non-UTF8 locales 
are
still in use (in China and Russia).

At minimal, incoming text from external apps should be converted from local 
locale to
UTF-8. For output text (ie '>') it is opened to question (which text encoding 
would
you send to gedit or gvim ?)

Nice problem, isn't it ? ;)

Original comment by fcro...@gmail.com on 27 Jan 2008 at 6:45

[GoogleCodeExporter]

Ok, from a quick look at the GEdit source code, it does appear to try to handle
legacy locale-encoded files.  

We can do this too - need to use the Python codecs.open function for files 
instead of
plain open().  There aren't too many places in Hotwire where we directly open 
files.
Similarly, we wrap subprocess I/O streams using codecs. 

I can reproduce the problem by using "LC_ALL=fr_FR.ISO-8859-1 python 
ui/hotwire" on
my system (though for some reason -15 doesn't work).

I'll take a look at solving this.

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 7:04

[GoogleCodeExporter]

This patch appears to work for LC_ALL=fr_FR.ISO-8859-1 python ui/hotwire, then
running "ifconfig".  Still TODO is recoding for file operations.

Committed r891
    M   hotwire/builtins/sys_builtin.py
    M   hotwire_ui/renderers/unicode.py
    M   hotwire_ui/odisp.py
r891 = f0cd00edbad4ae61529e55e7cb5fdd0d653ba89a (git-svn)

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 10:44

[GoogleCodeExporter]

Ok, the second part, file I/O:

Committed r892
    M   hotwire/builtins/pyeval.py
    M   hotwire/builtins/fsearch.py
    M   hotwire/builtins/write.py
    M   hotwire/builtins/cat.py
    M   hotwire/fs.py
    M   hotwire/command.py
    M   hotwire_ui/editor.py
r892 = 30a04634ac90c8674aaec02e8184625de1b70e4e (git-svn)

Original comment by cgwalt...@gmail.com on 27 Jan 2008 at 11:16

• Changed state: Fixed