Closed jgosmann closed 10 years ago
Setting LC_CTYPE
as suggested in the link above does not work for me. However, the problem wasn't to hard to fix by making the encoding of the progress output failsafe.
Should be fixed now.
Just tested isso 0.9.2 and it seems to work. :)
I get the following error message when trying to import comments from Disqus:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3282: ordinal not in range(128)
isso --version
tells me I'm using v0.9.8.
Full stack trace:
Traceback (most recent call last):
File "/usr/local/bin/isso", line 9, in <module>
load_entry_point('isso==0.9.8', 'console_scripts', 'isso')()
File "/usr/local/lib/isso/local/lib/python2.7/site-packages/isso/__init__.py", line 231, in main
migrate.dispatch(args.type, mydb, args.dump)
File "/usr/local/lib/isso/local/lib/python2.7/site-packages/isso/migrate.py", line 261, in dispatch
peek = fp.read(io.DEFAULT_BUFFER_SIZE)
File "/usr/local/lib/isso/lib/python2.7/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3282: ordinal not in range(128)
It is definitely a bug in Isso. Can you give me details about your environment, specifically: LANG
, LANGUAGE
, LC_ALL
, LC_CTYPE
?
Here's the result from locale
. Should I reset any of these?
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
It should work with LANG=C.UTF-8
I think. The main issue is that Python uses the mentioned variables to determine the file encoding when readinga file as non-binary (aka text).
I tried it and it worked. Thank you.
The UnicodeDecodeError during the import is now fixed, but the issue remains for the configuration file. If your configuration file contains non-ascii characters Isso will fail and I am not attempt to fix this, because
locale.getpreferredlocale(True)
)locale.getpreferredlocale(False)
) is not working for Python2 (returns ANSI_X3.4-1968 instead of UTF-8).In the end, all you have to do is: configure your locale correctly (or at least the encoding, C.UTF-8 is sufficient).
This will be fine, I guess. Probably it's worth mentioning in the documentation somewhere.
Trying to import comments an Wordpress-XML containing Unicode characters I get the following exception:
The file is UTF-8 encoded. The problematic character is a German umlaut (ä).
I was using the current master branch.