If a file name has non-ascii characters then code completion in the editor stops working

[spyder-bot]

From ccordoba12 on 2011-09-25T12:40:14Z

This also happens when a directory has non-ascii characters and you open a file inside it.

Original issue: http://code.google.com/p/spyderlib/issues/detail?id=770

[spyder-bot]

From ccordoba12 on 2011-09-26T08:19:01Z

More on this issue: after activating DEBUG in spyderlib/widgets/sourcecode/codeeditor.py, I've got this Traceback in a file named rope.log:

Logging time: Mon Sep 26 10:11:03 2011

Context

get_completion_list

Traceback

Traceback (most recent call last): File "/home/carlos/Projects/spyder/trunk/spyderlib/widgets/sourcecode/codeeditor.py", line 135, in get_completion_list source_code, offset, resource) File "/home/carlos/Projects/spyder/trunk/rope/contrib/codeassist.py", line 33, in code_assist return assist() File "/home/carlos/Projects/spyder/trunk/rope/contrib/codeassist.py", line 324, in call completions = list(self._code_completions().values()) File "/home/carlos/Projects/spyder/trunk/rope/contrib/codeassist.py", line 392, in _code_completions pymodule = fixer.get_pymodule() File "/home/carlos/Projects/spyder/trunk/rope/base/utils.py", line 10, in _wrapper setattr(self, name, func(self, _args, *_kwds)) File "/home/carlos/Projects/spyder/trunk/rope/contrib/fixsyntax.py", line 42, in get_pymodule new_message) ModuleSyntaxError: Syntax error in file </tmp/dinamicos/t\xe9st_1.py> line <1>: Syntax errors in file /tmp/dinamicos/t\xe9st_1.py:

• line 1: 'ascii' codec can't encode character u'\xe9' in position 16: ordinal not in range(128) ... fixed
• line 1: 'ascii' codec can't encode character u'\xe9' in position 16: ordinal not in range(128) ... raised!

So this seems clearly a rope problem, which can't handle non-ascii characters when trying to read files.

[spyder-bot]

From pierre.raybaut on 2011-10-01T05:24:35Z

A file with non-ascii characters in its name?! You are looking for trouble, Carlos :)

More seriously, you'd be surprised of how many respectable editors will fail editing those files... (I've worked a lot to solve related issues in Spyder)

Judging by the traceback above, is it only a problem of non-ascii chars in file name: it seems to complain about non-ascii chars at line #1. (maybe it's a side effect of the non-ascii chars in the file name)

[spyder-bot]

From ccordoba12 on 2011-10-01T19:17:36Z

Well, then this would be another selling point of Spyder, because I've managed to solve it ;-). It took me several hours of tinkering, from Spyder to rope to ast to compile and then to Spyder again, and PyQt/Pyside at last!

I'm more interested in directories with non-ascii characters rather than files (because people who speak languages other than English sometimes put their code in these dirs), but I think the problem is the same for both.

It has to do with the default encoding from which unicode strings are created. Python sets it to ascii, so when you do something like:

unicode('año')

it assumes that all the characters in 'año' (year in Spanish) are ascii characters, which is obviously false. To solve it you have to do:

unicode(‘año’, ‘utf-8’)

The problem is that modern Linux distros (and OSX also) use UTF8 as their default file system encoding, so file and dir strings are already unicode, but PyQt/PySide transform them again to unicode before opening the file. So the file opens just fine (thanks to Qt I guess), but when another python program tries to read/write the file, it tries to decode the string first and then complains about non-ascii characters in it (which is what rope is trying to do)

The solution I found is to redefine the default encoding before opening the file, and for that I added:

reload(sys) sys.setdefaultencoding('UTF8')

in spyderlib/qt/compat.py, after the

import os, sys

statement.

This could lead to problems with other libraries since I think this changes the encoding for the entire application. So I don't want to make the change before consulting you. I would like to isolate the change only to the moment when you open a file, but I haven't had time to do more tests.

[spyder-bot]

From pierre.raybaut on 2011-10-02T05:28:27Z

Well, Carlos, I know how annoying this kind of bug can be: I had to deal with this kind of bugs in the past in Spyder or in my work projects.

However, I think that the solution to use 'sys.setdefaultencoding' can't be implemented as is. It is really a risky solution as this function is not intended to be used outside a sitecustomize script (see spyderlib/widgets/externalshell/sitecustomize.py for example).

My (little) experience of Python/string-encoding issues tells me that there is always a clean solution to those problems: you have to find a way to encode strings in the right codec at the right time and everything will work just fine. If it means patching rope, then so be it, there is already a rope monkey-patch script in Spyder, as you know.

The first step may be to post a message on rope's discussion group. They react usually quite fast.

[spyder-bot]

From ccordoba12 on 2011-10-02T09:38:25Z

Sorry for the fuss. I thought I've got it right, but string-encoding in Python seems quite tricky to understand. I hope these things have become easier in Python 3!!

What I've got right though is that changing 'sys.setdefaultencoding' is risky and that's why I didn't push it to the tree. But, as you requested, I think I've developed a cleaner solution and that is "encode [the filename] in the right codec at the right time" :)

Could you please test this patch and tell me if it's working ok for you?

diff -r 8fa2dd1c17ee spyderlib/widgets/sourcecode/codeeditor.py --- a/spyderlib/widgets/sourcecode/codeeditor.py sáb sep 24 15:16:50 2011 +0200 +++ b/spyderlib/widgets/sourcecode/codeeditor.py dom oct 02 11:32:57 2011 -0500 @@ -123,7 +123,7 @@ class RopeProject(object): return []

     try:

         resource = rope.base.libutils.path_to_resource(self.project,

•                                                     filename)

•                                                filename.encode('utf-8'))
  
  except Exception, _error:
  
     if DEBUG:
  
         log_last_error(LOG_FILENAME, "path_to_resource: %r" % filename)

[spyder-bot]

From pierre.raybaut on 2011-10-02T13:59:48Z

I think that you have found the right solution indeed. It seems right even before testing it, and even more after testing it as it works perfectly!

Of course, this means that all other calls to rope have to be changed as well.

Regarding Python 3, this should help indeed but string encoding will still be there... (BTW, I'm thinking of switching to Py3k in a year or so, but not before.)

[spyder-bot]

From pierre.raybaut on 2011-10-16T10:20:27Z

This issue was closed by revision 72b0cf66b20d .

Status: Fixed