search

iclab / centinel

http://iclab.org/
MIT License
34 stars 17 forks source link

handle BeautifulSoup parsing exceptions #279

Closed rpanah closed 7 years ago

rpanah commented 7 years ago 
Exception in thread Thread-2238:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/xxx/centinel/centinel/primitives/http.py", line 201, in get_request
    meta_redirect_url = meta_redirect(redirect_http_result["response"]["body"])
  File "/home/xxx/centinel/centinel/primitives/http.py", line 25, in meta_redirect
    soup = BeautifulSoup.BeautifulSoup(decoded)
  File "/usr/local/lib/python2.7/dist-packages/BeautifulSoup.py", line 1522, in __init__
    BeautifulStoneSoup.__init__(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/BeautifulSoup.py", line 1147, in __init__
    self._feed(isHTML=isHTML)
  File "/usr/local/lib/python2.7/dist-packages/BeautifulSoup.py", line 1189, in _feed
    SGMLParser.feed(self, markup)
  File "/usr/lib/python2.7/sgmllib.py", line 104, in feed
    self.goahead(0)
  File "/usr/lib/python2.7/sgmllib.py", line 143, in goahead
    k = self.parse_endtag(i)
  File "/usr/lib/python2.7/sgmllib.py", line 320, in parse_endtag
    self.finish_endtag(tag)
  File "/usr/lib/python2.7/sgmllib.py", line 358, in finish_endtag
    method = getattr(self, 'end_' + tag)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-8: ordinal not in range(128)