praekeltfoundation / vumi-wikipedia

Vumi Wikipedia Application
BSD 3-Clause "New" or "Revised" License
8 stars 8 forks source link

Wikipedia articles with unicode chars in the titles crash the worker #3

Closed smn closed 12 years ago

smn commented 12 years ago

I searched for "Zundapp" which goes to Zündapp

2012-02-24 11:01:53+0000 [WorkerAMQClient,client] Unhandled Error
    Traceback (most recent call last):
      File "/vol/var/praekelt/vumi/develop/releases/20110623_130621/vumi/ve/lib/python2.6/site-packages/twisted/internet/defer.py", line 1014, in unwindGenerator
        return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
      File "/vol/var/praekelt/vumi/develop/releases/20110623_130621/vumi/ve/lib/python2.6/site-packages/twisted/internet/defer.py", line 893, in _inlineCallbacks
        result = g.send(result)
      File "/vol/var/praekelt/vumi/develop/releases/20110623_130621/vumi/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia.py", line 245, in process_message_content
        session['page'], int(msg['content'].strip()) - 1)
      File "/vol/var/praekelt/vumi/develop/releases/20110623_130621/vumi/ve/lib/python2.6/site-packages/twisted/internet/defer.py", line 1014, in unwindGenerator
        return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
    --- <exception caught here> ---
      File "/vol/var/praekelt/vumi/develop/releases/20110623_130621/vumi/ve/lib/python2.6/site-packages/twisted/internet/defer.py", line 893, in _inlineCallbacks
        result = g.send(result)
      File "/vol/var/praekelt/vumi/develop/releases/20110623_130621/vumi/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia.py", line 111, in get_content
        'page': page_name.encode('utf-8'),
    exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
hodgestar commented 12 years ago

Otherwise looks good.