non-latin characters in filenames results in failed uploads

[gbezyuk]

It seems that this problem is on django side.

[sehmaschine]

I´m not sure what this is about – please reopen the ticket with additional details if necessary.

[gbezyuk]

I'm not permitted to reopen this ticket. Anyway, required details are: I try to upload a file. File is named with non-latin characters. Let's say, filename is "РусскоеИмяФайла.jpg" Result of this upload is "upload failed", and no uploaded file copy in filebrowser. That's sad.

[sehmaschine]

what version of the filebrowser? is your server setup to deal with non-latin characters? did you set FILE_CHARSET?

[gbezyuk]

filebrowser version is 3.4.1 server deals with non-latin characters, russian locale is set FILE_CHARSET, as far as I know (https://docs.djangoproject.com/en/dev/ref/unicode/#templates) has nothing to do with file uploads. Am I wrong?

[sehmaschine]

you´re right about FILE_CHARSET. however, why did you say "the problem is on django side"?

another test you could try: when you upload the file (via FTP, for example) and then browse the upload-directory with the filebrowser ... does that work and do you see the correct filename?

[gbezyuk]

Unfortunately, I don't see even a browsing page, but UnicodeDecodeError exception. Full exception text here: http://pastebin.com/xLRNfy0j Well, I wasn't totally correct saying that, as I see now. But django is somehow related to this issue, at least with a misconfiguration. Look, if I try to upload file with russian-lettered filename on my development server (./manage.py runserver, latest Kubuntu, russian locale, python2.7 from standart package) that works just fine. I can upload file, I can browse through filebrowser, I can see russian-lettered files and use select them. Versions also works perfectly well. Then, I switch to my staging server. It has Debian 6.04 with russian locale and python2.6, standart for this version of Debian, installed. And here thins go berserk - I can't upload non-ascii-named files, and when I try to browse I see UnicodeDecodeErrors, like this: UnicodeDecodeError at /admin/filebrowser/browse/ 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128) Exception Location: /usr/lib/python2.6/posixpath.py in join, line 70 (again, full text is available at http://pastebin.com/xLRNfy0j) One point more: if I set up russian-lettered imagefiles on development server, and then just clone the repo (together with sqlite database), fb_versions for this images also becoming broken.

Can you help me just with anything? My customers really wants to see this damn evil russian letters in filenames.

Some details to see your own eyes:

1. Look at last two (right-bottom) squares at http://prana.co-de.org/ - there are some russian-named imagefiles, and fb_versions for them gives just en empty src attribute. I believe, that's because of UnicodeDecodeException inside templateTag or deeper.
2. http://prana.co-de.org/admin/filebrowser/browse/?&dir=Lady%20furshet - filebrowser browsing page for directory with russian-named files. You can use root / password to log in.

[sehmaschine]

import sys sys.getfilesystemencoding() sys.getdefaultencoding()

... please tell me what you get.

LANG and LC_ALL is set correctly, right? see https://docs.djangoproject.com/en/dev/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror

[gbezyuk]

Same UnicodeDecodeError occurs if I try to create a new folder with russian name via filebrowser: UnicodeEncodeError at /admin/filebrowser/createdir/ 'ascii' codec can't encode characters in position 63-66: ordinal not in range(128) <...> Unicode error hint The string that could not be encoded/decoded was: test/русь

[sehmaschine]

can you please send me the jpeg you´r using to sehmaschine@gmail.com?

[gbezyuk]

First, on a development machine, where everything seems fine: In [1]: import sys In [2]: sys.getfilesystemencoding() Out[2]: 'UTF-8' In [3]: sys.getdefaultencoding() Out[3]: 'ascii'

Then, on staging server:

import sys sys.getfilesystemencoding() 'UTF-8' sys.getdefaultencoding() 'ascii' so, same-same. Looks like it's not a python problem.

LANG and LC_ALL looks more promising, I believe now that they are causing this problem. Would you please tell me how to check or to set them properely for russian locale? I'm not a big Guru of Apache configuration =(

[sehmaschine]

sorry, I don´t know how to set/change LANG and LC_ALL ... when we had this problem, we talked to the company responsible for our servers.

[gbezyuk]

so same will I do =) I'll write here about results. Anyway, thank you a lot for help!

[sehmaschine]

alright – please keep me updated (maybe we need to add something to the docs about this issue).

[gbezyuk]

Problem is solved, thanks to you again! basing on: https://docs.djangoproject.com/en/dev/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror we need to edit file /etc/apache2/envvars and add there this two lines (for russian locale) export LANG='ru_RU.UTF-8' export LC_ALL='ru_RU.UTF-8' another (and possibly better) option is including /etc/default/locale in that file