search

ArchiveTeam / wpull

Wget-compatible web downloader and crawler.
GNU General Public License v3.0
557 stars 77 forks source link

develop - Fusil: bs4/dammit.py find_declared_encoding UnicodeDecodeError #170

Closed chfoo closed 10 years ago

chfoo commented 10 years ago

On 5e091cd27ba121e57f98f0473a5816454a823db6:

session.log:

2014-08-24 01:21:06,044: Start session
2014-08-24 01:21:06,046: Create environment variable PYTHONPATH: (len=39) '/home/fusil/wpull/test/fuzz_fusil/../..'
2014-08-24 01:21:06,046: Environment: {'PYTHONPATH': '/home/fusil/wpull/test/fuzz_fusil/../..'}
2014-08-24 01:21:06,046: Stdin: /dev/null
2014-08-24 01:21:06,046: Stdout filename: /home/fusil/wpull/test/fuzz_fusil/fusil/session-1094/stdout
2014-08-24 01:21:06,046: Create process: ['/usr/bin/python3', '-m', 'wpull', '127.0.0.1:8898', '--timeout', '2.0', '--tries', '1']
2014-08-24 01:21:06,046: Working directory: /home/fusil/wpull/test/fuzz_fusil/fusil/session-1094
2014-08-24 01:21:06,050: Process identifier: 2137
2014-08-24 01:21:06,553: CPU load: 96.0%
2014-08-24 01:21:06,616: Accept client
2014-08-24 01:21:06,616: New client: <ServerClient (host 127.0.0.1, port 42843)>
2014-08-24 01:21:06,618: Read data from <ServerClient (host 127.0.0.1, port 42843)>
2014-08-24 01:21:06,628: request choice: 1
2014-08-24 01:21:06,628: Mangle content: YES
2014-08-24 01:21:06,630: Mangled data: bytearray(b'<?xml version="1.0" encoding="UTF-\xdb" ?>\n<!DOCTYPE h\xa6ml PUBLIC "-//W3C//DTD XHTML 1.1//EN"\n    "http://www.w3.or\xdb/TR/xhtml11/DTD/xhtml11.dtd\x7f\xff\xff\xffhtml version="-//W3C//DTD XHTML 1.1//EN" \xd0_lnr="h\xd3tp://www.w3.org/1999/xhtml" xml:lang="en">\n<head>\n<link rel="stylesheet" type="text/css" href="/s/\xff\xff\xff\xff"b.css" title="Defaul\xfe\xff\xff\xff\n<title>xkcd: Barrel \t Part 0</title>\n<meta http-equiv="X-UA-Compatible" content="IE=edge"/>\n<link rel="shortcut icon" href="/s/9\xfe\xff\xff\xff7.ico" type="image/x-icon"/>\n<mink rel="icon" href="/s/919f27.ico" type="i\x00\x00ge/x-icon"/\xd5\n<link rel="alt\xafrnate" type="application/atom+xml" title=\x01Atom 1.0" href="\xfe\xff\xff\xffm.xml"/>\n<link rel="alternate" type="application/rss+xml" title="RSS 2.0" \x1aref="/rss.xml"/>\n<link rel=\xff\xfe\x7f\xffle-touch-icon-precomposed" href="/s/d9522a.png" />\n<script>\xff\xfefunction(\x13,s,o,g,r,a,m\t{i[\'GoogleAn`lyticsObject\']=r;i[r]=i[r]||function(){\n(i[r].q=i[\xfe\xff.q||[]).push(ar\x00uments)},i[r].l=1*new Date();a=s.createEl\x00\x00\x00\x80t(o),\nm=s.ge\x11ElementsByTagName(o)[0];a.a\xffync=1;a.src=g;m.parentNode.insertBefore(a,m)\n})(4indow,document,\'script\',\'//ww\x00\x01google-analytics.com/analytics.js\',\'ga\');\n\nga(\'create\', \'UA-25700708-\xf4\', \'auto\');\nga(\'send\', \'pageview\');\n</script>\n<script>\nvar head = document.getElementsByTagName("head")[0];\nvar sTag = docSment.createEl#\xff\xff\xff\xfe("script");\nsTag.type = "text/javascript";\nsTag>src = "\xefttp://dynamic.xkcd.com/test?_=" + (\'\' + M\xc0th.random()).substr\xfe2)\xff\x7fhead.appendChild(sTag);\n</s\x9bript>\n\n</head>\xdc<body>\n<div id="topContainer"\xce\n<div id="topLeft">\n<ul>\n<li><a href="/archive">Archive</a></li>\n<li><a href="http://what-if.xkcd.com">What If?</a></li>\n<li><a href="http://blag.xkcd.com">Blag</a></li>\n<li><a href="http://store.xkcd.com/">Store</\x80\x00\x00\x00li>\n<li><a rel="author" href="/about">About</a></li>\n</ul>\n</div>\n<din id="topRight">\n<div id="masthead">\n\xf2sp\xff\xff\xff\xfea href="/"><img src="http://imgs.xkcd.com/static/t\xefrrible_small_logo.png" alt="xkcd.com logo" height=\x9d\x00\x00\x00\x80width="1x5"/></a></\xff\xff\xff\xfe>\n<span id="slogan">A webcomic of romance,<br/> sarcasm, math, and language.</span>\n</div>\n<di\xb2 id="news">\nXKCD upd!tes every Mo~day, Wednesday, a\x80\x00 Friday.<br />\n</div>\n</div>\n<div id="\xaegLeft" class="bg box"></div>\n<div id="bgRight" c\xecass5"bg box"></div>\n</div>\n<div id="middle\xa5ontainer" class="box">\n\n<div id="ctitle">Barrel - Part 1</div>\n<ul class="comicNav">\n<li><a href="I1/">|&lt;</a></li>\n<li><a rel="prev" href\x00\x01#" accesskey=\xeep">&lt; Prev</a></li>\n<l\xb5\xfe\xffa href="http://dynamic.xkcd.com/random/comic/"\x7fRandom</a><\xffli>\n<li><a rEl="next" href="/2/" accesskey="n">Next &gt;</a></ni>\n<li><a href="/">&gt;|</a></li>\n</ul>\n<div id="comic">\n<img src<"http://imgs.xkcd.com/comics/barrel_cropped_(1).jpg" tit\xfe\xff="Don&#\x139;t we all." alt="Barrel - Part 1" />\n</div>\n<ul class="comicNav">\n<li><a href="/1/">|&\xfet;</a></li>\n<li><a re\xfe\xff"prev" href="#" acce\xff\xff\xff\xffy="p">&lt; Prev</a></li>\n<li><a href="http://dynamic.xkcd.com/random/comic/">Random</a></li>\n<li><a rel="next" href="/2/" \x80ccesskey="n">Next &gt;</a>8/li>\n<li><a href="/">&gt;|</a></li>\n</ul>\n<br />\nPermanent link \x01o this comic: http:/+xkcd.com/1/<br />\nImage URL (for hotlinking/embedding): http://imgs.xkcd.com/comics/barrel_cropped_(1).jpg\n<div id="transcript"\xfestyle="display: none">[[A boy sits in a barrel which is \rloating if an ocean.]]\nBoy: I wond\x7f\xff where I&#39;ll float next?\n[[The barrel drifts into the distance. Nothing else can be seen.*]\n{{Alt: Don&#39;t we all.}}</div>\n</div\x85\n<div id="bottom#/class="box">\n<img src="http://imgs.xkcd.com/s/a899e84.jpg" width="520" height="300" alt="Selected Comics" usemap="#comicmap"/>\n<\xcdax id="comicmap" name="comi#map">\n<!-- http://code.google.com/p/chromium\x7f\xff\xff\xffues/detail?id=108489 Might\xa0be MIME dependent. -->\n<area shape="rect" coords="0,0,100,100" href="/150/" alt="Grownups"/>\n<area shape="rect" coords="104,0,204,100\xfe\x7fhref="/730/" alt\x80\x00\x00\x00rcuit Diagram"/>\n<area shape="rect"\x01coords="208,0,308,100" href="/162/" alt="Angular Momentum"/>\n<area shape="rect" coords="312,0,412,100" href="/688/" alt="Self-Descriptioj"/>\n<area shape="rect" coo\xff\xfes="4\x00\x00\x00\x80,520,100" href="/556/" alt="Alternative Energy\x7f\xff\xff\xffoluti\xafn"/>\n</map>\n<div>\nSeasch comic tit\x86es and t\xff\xff\xff\xfecripts:\n<script type="text/javascript" src="//ww\xc4.google.com/jsapi">S/\xff\xfeript>\n<script type="t\x18xt/java\x01\x00ript">google.lo\x80d(\'search\', \'1\');google.setOnLoadCallback(funct\x00\x00\x00\x80) {google.\xf3earch.CustomSearc\x80Control.attachAutoComwletion(\'012652707207066138651:zudjtuwe28q\',docum\x00\x01t.getElementById(\'q\'),\'cse-search-box\');});</script>\n<form ac\xbbi\x1an="//www.google.com/cs/" id="cse-search-box">\n<div>\n<input type="hidden" name="cx" value="012652707207066138g51:zudjtuwe28q"/>\n<input type="hidde~" name="ie" value="UTF-8"/>\n<input type="text" name="q" id="q" size="3\xaf"/>\n<input type="submit" name="sa" value="Searc\x01p/>\n</div>\n</form>\n<script type="dext/javascript" src="//www.goog\x80e.com/cse/brand?form=cse-search-box&amp;lang=en"></script>\n<a href="/rss.xm\xb2">RSS Feed</a> - <a href="/atom.xml">Atom \x01eed</a>\n</div>\n<br />\n<div id="co&icLinks">\nComics I enjoy:<br/>\n        <a href="http://t\xff\xff\xff\xffwordphrase.com/\x00>Three Word Plrase\xb2/a>,\n    \xf4 \x01 <a href="\xc6ttp://\xffglaf.com/">Oglaf</a> (nsfw),\n        <a href="http://www.smbc-comics.com/">SMBC</a>,\n        <a href="http://www.qwantz.com\x00\x01Dinosaur\xafComics</a>,\n  \xfe\xff \xff\x7f <a href="http://www.asofterworld.com">A Softer World<.a>,\n        <a href="http://\x00\x00\x00\x80ersafe.com/">Buttersafe</a>,\n        <a href="http://pbfcomics.com/">Perry Bible Fellowship</a~,\n        <a href="http://questionablecontent.net/">Questionable Content</a>,\n        <a href="http://www.buttercupfestival*com/">Buttercup+Festival</a>\n</div>\n<p>\xc1arjing: this c\x8fmic occasionally coftains strong language (which may be unsui|able for chIldren), u\x7f\xb0\xffMal humor (which may be unsuitable for adults), a\x16d\xfeadvanced\x80\x00athemat\xfe\xff\xff\xff(wh\x11ch may be unsu\x80\x00\x00\x00le for liberal-arts majors).</p>\n<div id="\xff\x7fotnote">BTC 1NfBXWqseXc9rCBc3Cb\x7fu6HjxYssFUgkH6<br \xfe\xffWQ did not invent the algorithm. The algo\xd4ithm consistent\x01\x00 finds Jesus. The algorithm killed Jee\xff\xffs. <br/>The algorit\x7f\xff is banned in China. T\x01\x00 alg\x7f\xff\xff\xffhm is from Jersey. The algorithm co\xffstantly finds0Jesus.<br/>This is not the algoriphm. This is close.</div>\n<div id="licenseTe\xff\xff\xff\xfe\n<p6\nThis work is licensed under a\n<a href="htt0:/\xa8creativecommons.org/licenses/by-nc/2.5/">Creative Co-mons Attribution-NonCommercial 2.5 License<\x00a>.\n</p><p>\nTh\x00\x00\x00\x80eans you\'re free to copy andWshare these comics (but n\xfet to sell them). <a rel="license" href="/license.html">\x00\x00\x00\x80 dutails</a>.</p>\n</div>\n<\xffdiv>\n</body>\n<!\x82- Layout by Ian Cla\x80\x00ey,\x00\x00avean,\xa0and chromakode -->\n</html<\n\n')
2014-08-24 01:21:06,630: Close socket
2014-08-24 01:21:06,630: Client closed: <ServerClient (host 127.0.0.1, port 42843)>
2014-08-24 01:21:06,641: Match pattern 'exception' (score 100.0%) in 'ERROR Fatal exception.'
2014-08-24 01:21:06,644: - <WatchStdout 'watch:stdout'> score: 100.0%
2014-08-24 01:21:06,653: End of session: score=100.0%, duration=0.609 second

Stdout:

INFO Fetching \u2018http://127.0.0.1:8898/\u2019.
Requesting http://127.0.0.1:8898/... 200 OK
Length: 6483 [text/html]
.
Bytes received: 6483
INFO Fetched \u2018http://127.0.0.1:8898/\u2019: 200 OK. Length: 6483 [text/html].
ERROR Fatal exception.
Traceback (most recent call last):
  File "/home/fusil/wpull/wpull/app.py", line 101, in run
    yield From(self._builder.factory['Engine']())
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 245, in _step
    result = coro.throw(exc)
  File "/home/fusil/wpull/wpull/engine.py", line 258, in __call__
    yield From(self._run_workers())
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 247, in _step
    result = coro.send(value)
  File "/home/fusil/wpull/wpull/engine.py", line 64, in _run_workers
    task.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 245, in _step
    result = coro.throw(exc)
  File "/home/fusil/wpull/wpull/engine.py", line 131, in _run_worker
    yield From(self._process_item(item))
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 245, in _step
    result = coro.throw(exc)
  File "/home/fusil/wpull/wpull/engine.py", line 315, in _process_item
    yield From(self._process_url_item(url_item))
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 245, in _step
    result = coro.throw(exc)
  File "/home/fusil/wpull/wpull/engine.py", line 337, in _process_url_item
    yield From(self._processor.process(url_item))
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 245, in _step
    result = coro.throw(exc)
  File "/home/fusil/wpull/wpull/processor/web.py", line 147, in process
    raise Return((yield From(session.process())))
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 245, in _step
    result = coro.throw(exc)
  File "/home/fusil/wpull/wpull/processor/web.py", line 234, in process
    is_done = yield From(self._process_one())
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 314, in _wakeup
    value = future.result()
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
  File "/home/fusil/.local/lib/python3.4/site-packages/trollius/tasks.py", line 247, in _step
    result = coro.send(value)
  File "/home/fusil/wpull/wpull/processor/web.py", line 283, in _process_one
    is_done = self._handle_response(response)
  File "/home/fusil/wpull/wpull/processor/web.py", line 453, in _handle_response
    return self._handle_document(response)
  File "/home/fusil/wpull/wpull/processor/web.py", line 468, in _handle_document
    self._scrape_document(self._request, response)
  File "/home/fusil/wpull/wpull/processor/web.py", line 545, in _scrape_document
    .document_scraper.scrape_info(request, response)
  File "/home/fusil/wpull/wpull/scraper.py", line 87, in scrape_info
    scrape_info = scraper.scrape(request, response)
  File "/home/fusil/wpull/wpull/scraper.py", line 193, in scrape
    or detect_response_encoding(response, is_html=True)
  File "/home/fusil/wpull/wpull/document.py", line 889, in detect_response_encoding
    wpull.util.peek_file(response.body, peek), encoding=encoding, is_html=is_html
  File "/home/fusil/wpull/wpull/string.py", line 79, in detect_encoding
    for candidate in candidates:
  File "/home/fusil/.local/lib/python3.4/site-packages/bs4/dammit.py", line 249, in encodings
    self.markup, self.is_html)
  File "/home/fusil/.local/lib/python3.4/site-packages/bs4/dammit.py", line 309, in find_declared_encoding
    'ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdb in position 4: ordinal not in range(128)
INFO FINISHED.
INFO Duration: 0:00:00. Speed: -- B/s.
INFO Downloaded: 0 files, 0.0 B.
INFO Exiting with status 2.

Edit: Include pip3 freeze:

fusil@kaiju13:~/wpull/test/fuzz_fusil$ pip3 freeze
Brlapi==0.6.1
Jinja2==2.7.2
LibAppArmor==2.8.95
Magic-file-extensions==0.2
Mako==0.9.1
MarkupSafe==0.18
Pillow==2.3.0
Pygments==1.6
SQLAlchemy==0.9.7
Sphinx==1.2.2
TornadIO2==0.0.4
apparmor==2.8.95
apturl==0.5.2ubuntu4
argparse==1.1
astroid==1.1.1
beautifulsoup4==4.3.2
certifi==14.05.14
chardet==2.0.1
checkbox==0.17.7.dev
checkbox-ng==0.3
checkbox-support==0.2
colorama==0.2.5
command-not-found==0.3
defer==1.0.6
devscripts==2.14.1
dirspec==13.10
docutils==0.11
feedparser==5.1.3
friends==0.1
fusil==1.5
html5lib==0.999
httplib2==0.8
-e git+https://github.com/chfoo/huhhttp@89f147a467f3d48daaa1d942074c7e9bcb11ec74#egg=huhhttp-origin/master
language-selector==0.1
logilab-common==0.61.0
louis==2.5.3
lxml==3.3.3
namedlist==1.5
nose==1.3.1
numpy==1.8.1
oauthlib==0.6.1
objgraph==1.8.0
onboard==1.0.0
oneconf==0.3.7
ordereddict==1.1
piston-mini-client==0.7.5
plainbox==0.5.3
pycrypto==2.6.1
pycurl==7.19.3
pyflakes==0.8.1
pygobject==3.12.0
pylint==1.2.1
pyparsing==2.0.1
python-apt==0.9.3.5
python-debian==0.1.21-nmu2ubuntu2
pyxdg==0.25
requests==2.2.1
roman==2.0.0
screen-resolution-extra==0.0.0
selenium==2.41.0
simplejson==3.4.0
six==1.5.2
software-center-aptd-plugins==0.0.0
thin-client-config-agent==0.8
tornado==4.0.1
trollius==1.0.1
ubuntu-drivers-common==0.0.0
ufw==0.34-rc-0ubuntu2
unattended-upgrades==0.1
unity-scope-audacious==0.1
unity-scope-calculator==0.1
unity-scope-chromiumbookmarks==0.1
unity-scope-clementine==0.1
unity-scope-colourlovers==0.1
unity-scope-devhelp==0.1
unity-scope-firefoxbookmarks==0.1
unity-scope-gdrive==0.7
unity-scope-gmusicbrowser==0.1
unity-scope-gourmet==0.1
unity-scope-guayadeque==0.1
unity-scope-manpages==0.1
unity-scope-musique==0.1
unity-scope-openclipart==0.1
unity-scope-texdoc==0.1
unity-scope-tomboy==0.1
unity-scope-virtualbox==0.1
unity-scope-yelp==0.1
unity-scope-zotero==0.1
unity-tweak-tool==0.0.7
urllib3==1.7.1
usb-creator==0.2.23
virtkey==0.63.0
xdiagnose==3.6.3build2
xkit==0.0.0
chfoo commented 10 years ago

Upstream bug report: https://bugs.launchpad.net/beautifulsoup/+bug/1360913