search

errbotio / errbot

Errbot is a chatbot, a daemon that connects to your favorite chat service and bring your tools and some fun into the conversation.
http://errbot.io
GNU General Public License v3.0
3.13k stars 614 forks source link

UnicodeDecodeError in process_reply() #263

Closed hanks closed 10 years ago

hanks commented 10 years ago

When to execute fab --list to the bot to show help of each fab command, with some commands contained Japanese characters. It throws UnicodeDecodeError below: 

File "/hanks/.pyvirtualenvs/kemono/lib/python2.7/site-packages/errbot/backends/base.py", line 510, in process_reply
    return unicode(reply)
File "/hanks/.pyvirtualenvs/kemono/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 289-290: invalid continuation byte

I use temp work around return unicode(reply, errors='replace') that can show help with some garbages. And I also check all the fab command files, they all contain #coding: utf8 at the begininng. Does here need something like django's smart_text library that can handle any inputs?

zoni commented 10 years ago

What is the code you are using to shell out to fab --list and return it's output exactly? I suspect you aren't feeding utf-8 encoded data to Err itself.

hanks commented 10 years ago

@zoni Yes, I use output = subprocess.check_output(cmd, shell=True), and I found the output is a byte string not encoded in UTF-8...I use chardet to detect it and result shows 30% possibility to be iso8859-2 , It is a little weird. I will handle this in my plugin level. Thank you for your advice.

zoni commented 10 years ago

Great, glad you got it sorted out!