Support Unicode based character sets

[GoogleCodeExporter]

What steps will reproduce the problem?
1.import a unicode based filename (Ringel-S or something)
2.UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 33: 
ordinal not in range(128)

What is the expected output? What do you see instead?
Should be able to handle unicode compliant file systems

What version of the product are you using? On what operating system?
Windows, linux

Original issue reported on code.google.com by j.bak...@atmind.nl on 28 Oct 2009 at 8:06

[GoogleCodeExporter]

Original comment by j.bak...@atmind.nl on 28 Oct 2009 at 8:06

• Added labels: Type-Defect
• Removed labels: Type-Enhancement

[GoogleCodeExporter]

It is not the filename, but the directory name.Perhaps the same as issue 24 
(spaces)

Original comment by j.bak...@atmind.nl on 3 Nov 2009 at 7:25

[GoogleCodeExporter]

Exception happened during processing of request from ('127.0.0.1', 46998)
Traceback (most recent call last):
  File "/usr/lib64/python2.6/SocketServer.py", line 281, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib64/python2.6/SocketServer.py", line 307, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.6/SocketServer.py", line 320, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib64/python2.6/SocketServer.py", line 615, in __init__
    self.handle()
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
    method()
  File "./server.py", line 97, in do_POST
  File "./server.py", line 174, in doHandleService
  File "./serviceproduction.py", line 59, in handleGetProductionView
  File "./indexer.py", line 75, in updateIndex
  File "/usr/lib64/python2.6/os.py", line 284, in walk
    if isdir(join(top, name)):
  File "/usr/lib64/python2.6/posixpath.py", line 70, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 30: 
ordinal not
in range(128)
----------------------------------------
localhost - - [04/Nov/2009 21:49:40] "GET /images/unknownfile.png HTTP/1.1" 200 
-
localhost - - [04/Nov/2009 21:49:40] "GET /images/filter.png HTTP/1.1" 200 -
localhost - - [04/Nov/2009 21:49:40] "GET /images/missing.png HTTP/1.1" 200 -
localhost - - [04/Nov/2009 21:49:40] "GET /favicon.ico HTTP/1.1" 200 -

Original comment by j.bak...@atmind.nl on 4 Nov 2009 at 8:57

• Changed state: Accepted

[GoogleCodeExporter]

jbakker: I have some problems reproducing your UnicodeDecodeError
jbakker: Have you any clues or example what can help me.

- 21:39 -
mib2berlin: okay, i have a look to a folder that not work and tell you

- 21:54 -
mib2berlin: so, i have a folder with many files and folders
mib2berlin: in the folder is one file Gesch�tz.tga
mib2berlin: and Entlüftung.jpg
mib2berlin: http://www.pasteall.org/8830/text
mib2berlin: complete message
jbakker: ah.. and these files are used within blend files....
mib2berlin: yes, for textures
mib2berlin: i'll try to rename, but there are many and mabe i dont find all

- 21:59 -
jbakker: No, I think that I know the issue. It is that blender does not really
support unicode. I have created strange names on my US language machine, to 
test your
bug 

Original comment by j.bak...@atmind.nl on 4 Nov 2009 at 9:00

[GoogleCodeExporter]

Also the OS complaints about the unicode name... Is it really a unicode name?

Original comment by j.bak...@atmind.nl on 5 Nov 2009 at 5:34

Attachments:

• unicode-test.png

[GoogleCodeExporter]

http://www.gossamer-threads.com/lists/python/python/700968

Python has problems supporting this.

kaito: jbakker: yuo mean the way how file names are stored in strings
kaito: that is depending on the line of code it happens 
kaito: stay away from it!
kaito: files with umlauts <shudder>
stiv: first spaces in file names and now ulauts!  is there no end to the 
madness?
jbakker: Kaito: yes, it about linked images. Modern filesystems are unicode
complaint, blender can handle these, but they are stored and read in a very 
strange
unreadable way 
kaito: apparently a working way 
kaito: i didnt do anything special; filenames are retrieved via statfs
kaito: and then fed to open()

http://library.gnome.org/devel/glib/unstable/glib-Character-Set-Conversion.html

Original comment by j.bak...@atmind.nl on 5 Nov 2009 at 6:33