search

ArchiveTeam / wpull

Wget-compatible web downloader and crawler.
GNU General Public License v3.0
551 stars 77 forks source link

EOFError: Compressed file ended before the end-of-stream marker was reached #301

Open ivan opened 8 years ago

ivan commented 8 years ago

This crashes after about a minute:

~/.local/bin/wpull3 --concurrent 10 -r --sitemaps http://2ch.en.utf8art.com/
INFO Fetching ‘http://2ch.en.utf8art.com/’.
INFO Fetched ‘http://2ch.en.utf8art.com/’: 200 OK. Length: 93708 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/robots.txt’.
INFO Fetching ‘http://2ch.en.utf8art.com/sitemap.xml’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2009/12’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/giko’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/syakin’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/a-happy-new-year’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/presentiment’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/buzzwords’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/snow’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/bicycle’.
INFO Fetched ‘http://2ch.en.utf8art.com/robots.txt’: 200 OK. Length: unspecified [text/plain; charset=utf-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/bird’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/giko’: 200 OK. Length: 94675 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/03’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2009/12’: 200 OK. Length: 77083 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/a-happy-new-year’: 200 OK. Length: 94864 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/syakin’: 200 OK. Length: 92138 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/01’.
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/09’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/presentiment’: 200 OK. Length: 93629 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character/1601-3200char’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/buzzwords’: 200 OK. Length: 95554 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/mona’.
INFO Fetched ‘http://2ch.en.utf8art.com/sitemap.xml’: 200 OK. Length: 847077 [application/xml].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/usugeman’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/snow’: 200 OK. Length: 85690 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/bicycle’: 200 OK. Length: 91437 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/03’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/dance_78.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/bird’: 200 OK. Length: 91468 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/pc’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/01’: 200 OK. Length: 95344 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/get_a_grip_136.html/comment-page-1’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line’: 200 OK. Length: 77775 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/kotatsu’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/03’: 200 OK. Length: 95588 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/12’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character/1601-3200char’: 200 OK. Length: 78207 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/syoboon’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/mona’: 200 OK. Length: 76928 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/oeedori’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/09’: 200 OK. Length: 96763 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/usugeman’: 200 OK. Length: 93285 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/megassa’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/03’: 200 OK. Length: 79643 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character/101-200char’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/dance_78.html’: 200 OK. Length: 73684 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/30’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/pc’: 200 OK. Length: 77485 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/06’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/get_a_grip_136.html/comment-page-1’: 200 OK. Length: 73127 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/12’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/kotatsu’: 200 OK. Length: 92931 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/08’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/12’: 200 OK. Length: 98179 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/jump’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/syoboon’: 200 OK. Length: 93665 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character’: 200 OK. Length: 77897 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/calendar’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/valentine-day’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/oeedori’: 200 OK. Length: 76256 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/shii’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/megassa’: 200 OK. Length: 77505 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/moonlight-party’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/30’: 200 OK. Length: 78035 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/10’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/jump’: 200 OK. Length: 76841 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/shimamurakun’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/06’: 200 OK. Length: 95745 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/10’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character/101-200char’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/11’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/shii’: 200 OK. Length: 78450 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/09’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/08’: 200 OK. Length: 95773 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/christmas’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/valentine-day’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/vega-festival’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/moonlight-party’: 200 OK. Length: 77445 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/04’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/10’: 200 OK. Length: 95130 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/ricecake’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/shimamurakun’: 200 OK. Length: 94951 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/09’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/calendar’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/06’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/10’: 200 OK. Length: 89412 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/iyahoo’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/11’: 200 OK. Length: 78678 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/09’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/09’: 200 OK. Length: 93795 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/01’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/12’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/sorette_okashikunee_11.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/christmas’: 200 OK. Length: 94690 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/bed’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/vega-festival’: 200 OK. Length: 93632 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/reindeer’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/ricecake’: 200 OK. Length: 76885 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/04’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/04’: 200 OK. Length: 96445 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/its-no’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/09’: 200 OK. Length: 78783 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/02’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/06’: 200 OK. Length: 96944 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/otu’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/iyahoo’: 200 OK. Length: 71807 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/11’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/09’: 200 OK. Length: 78971 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/flying’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/01’: 200 OK. Length: 80173 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/sorette_okashikunee_11.html’: 200 OK. Length: 73281 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/07’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/08’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/reindeer’: 200 OK. Length: 77962 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/2ch’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/bed’: 200 OK. Length: 92398 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/a_happy_new_year_58.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/04’: 200 OK. Length: 96960 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/its-no’: 200 OK. Length: 77367 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/sorette-okashikunee’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/aramaki-scaltinof’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/02’: 200 OK. Length: 79857 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/otu’: 200 OK. Length: 77742 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/popular’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/12’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/11’: 200 OK. Length: 96620 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/09’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/flying’: 200 OK. Length: 77927 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/05’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/08’: 200 OK. Length: 94437 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/09’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/a_happy_new_year_58.html’: 200 OK. Length: 74602 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/sorette-okashikunee’: 200 OK. Length: 77177 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/4’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/08’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/09’: 200 OK. Length: 93998 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/tea’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/07’: 200 OK. Length: 98518 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/02’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/12’: 200 OK. Length: 77348 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/05’.
INFO Fetched ‘http://2ch.en.utf8art.com/popular’: 200 OK. Length: 141779 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/aramaki-scaltinof’: 200 OK. Length: 94184 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/hotspring’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/05’: 200 OK. Length: 95183 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/02’.
INFO Fetching ‘http://2ch.en.utf8art.com/?size=1’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/2ch’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/syoboon_77.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/09’: 200 OK. Length: 96655 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character/401-800char’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/tea’: 200 OK. Length: 92693 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/03’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/02’: 200 OK. Length: 96556 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/06’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/08’: 200 OK. Length: 96246 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/unreadable’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/hotspring’: 200 OK. Length: 76000 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/comments/feed’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/4’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/end’.
INFO Fetched ‘http://2ch.en.utf8art.com/?size=1’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/cat’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/05’: 200 OK. Length: 94129 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/05’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/03’: 200 OK. Length: 82086 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/07’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/06’: 200 OK. Length: 78700 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character/401-800char’: 200 OK. Length: 77853 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/house’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/06’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/syoboon_77.html’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/scoop_3.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/unreadable’: 200 OK. Length: 77362 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/mamono’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/end’: 200 OK. Length: 91981 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/pc_28.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/cat’: 200 OK. Length: 92410 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/brothers’.
INFO Fetched ‘http://2ch.en.utf8art.com/comments/feed’: 200 OK. Length: unspecified [text/xml; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/03’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/05’: 200 OK. Length: 95731 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/234’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/07’: 200 OK. Length: 79594 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/01’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/02’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character/over3201char’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/house’: 200 OK. Length: 76838 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/02’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/06’: 200 OK. Length: 96421 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/scoop_3.html’: 200 OK. Length: 92527 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/11’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/kotatsu_39.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/mamono’: 200 OK. Length: 76928 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/04’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/brothers’: 200 OK. Length: 76803 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/sorette_okashikunee_17.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/03’: 200 OK. Length: 97094 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/01’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/234’: 200 OK. Length: 73655 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/kotatsu_30.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character/over3201char’: 200 OK. Length: 93663 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/02’: 200 OK. Length: 79667 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/01’: 200 OK. Length: 95246 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/06’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/pig’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/maro’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/11’: 200 OK. Length: 80309 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/friedrice’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/pc_28.html’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/04’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/kotatsu_39.html’: 200 OK. Length: 73878 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/08’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/sorette_okashikunee_17.html’: 200 OK. Length: 72348 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/04’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/06’: 200 OK. Length: 81235 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line/2line’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/pig’: 200 OK. Length: 91907 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/04’: 200 OK. Length: 80099 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/text’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/03’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/maro’: 200 OK. Length: 90413 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/02’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/friedrice’: 200 OK. Length: 77697 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/mameshibata’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/01’: 500 Internal Server Error. Length: 538 [text/html; charset=iso-8859-1].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/04’: 200 OK. Length: 81550 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line/over21line’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/08’: 200 OK. Length: 94026 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/10’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/04’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/04’: 200 OK. Length: 80949 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/kotatsu_30.html’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/shimamurakun_78.html/comment-page-1’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/12’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line/2line’: 200 OK. Length: 66767 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/sleep’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/text’: 200 OK. Length: 76227 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/get-a-grip’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/03’: 200 OK. Length: 94666 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/05’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line/over21line’: 200 OK. Length: 77357 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/mameshibata’: 200 OK. Length: 77332 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/05’.
INFO Fetching ‘http://2ch.en.utf8art.com/tagcloud’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/10’: 200 OK. Length: 92014 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/08’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/04’: 200 OK. Length: 94300 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/a_happy_new_year_60.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/shimamurakun_78.html/comment-page-1’: 200 OK. Length: 81291 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/?size=3’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/12’: 200 OK. Length: 95767 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/unreadable_52.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/sleep’: 200 OK. Length: 91765 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/carp-streamer’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/get-a-grip’: 200 OK. Length: 77952 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/3’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/05’: 200 OK. Length: 81014 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/hattoushin’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/02’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/05’: 200 OK. Length: 81968 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/02’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/presentiment_42.html/comment-page-1’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/a_happy_new_year_60.html’: 200 OK. Length: 72681 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/girl’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/08’: 200 OK. Length: 97572 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/06’.
INFO Fetched ‘http://2ch.en.utf8art.com/tagcloud’: 200 OK. Length: 233948 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/deliberation’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/carp-streamer’: 200 OK. Length: 79051 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/08’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/unreadable_52.html’: 200 OK. Length: 74350 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/aramaki_scaltinof_92.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/?size=3’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/01’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/hattoushin’: 200 OK. Length: 93812 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/shimamurakun_131.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/presentiment_42.html/comment-page-1’: 200 OK. Length: 83282 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/10’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/02’: 200 OK. Length: 96587 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/group’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/girl’: 200 OK. Length: 75699 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line/3-5line’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/06’: 200 OK. Length: 95374 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/03’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/deliberation’: 200 OK. Length: 93421 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/07’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/08’: 200 OK. Length: 79063 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/20’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/aramaki_scaltinof_92.html’: 200 OK. Length: 74221 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/xmlrpc.php’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/shimamurakun_131.html’: 200 OK. Length: 72286 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/mona_56.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/01’: 200 OK. Length: 80084 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/reindeer_1.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/10’: 200 OK. Length: 80336 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/ski_5.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/group’: 200 OK. Length: 92326 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/car’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/03’: 200 OK. Length: 96025 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line/3-5line’: 200 OK. Length: 77048 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/10’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/mona_56.html’: 200 OK. Length: 75907 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/kitaaa’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/07’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/3’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/presentiment_51.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/xmlrpc.php’: 200 OK. Length: unspecified [text/plain; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2ch_56.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/20’: 200 OK. Length: 78158 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/its_no_75.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/reindeer_1.html’: 200 OK. Length: 71555 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/template_15.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/07’: 200 OK. Length: 80823 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/ski_5.html’: 200 OK. Length: 72657 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/kancolle_134.html’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/12’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/car’: 200 OK. Length: 119344 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/destroy_1.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/10’: 200 OK. Length: 95204 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2014/11’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/kitaaa’: 200 OK. Length: 77457 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/5’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/07’: 200 OK. Length: 79746 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/heater’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/presentiment_51.html’: 200 OK. Length: 74375 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2011/07’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2ch_56.html’: 200 OK. Length: 73312 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/mona_69.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/its_no_75.html’: 200 OK. Length: 102304 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/12’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/12’: 200 OK. Length: 96047 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/page/2’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/template_15.html’: 200 OK. Length: 99305 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/childrens-day’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2014/11’: 200 OK. Length: 93970 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/destroy_1.html’: 200 OK. Length: 72309 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/fusagiko’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/everybody’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/heater’: 200 OK. Length: 68778 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/hotspring_6.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2011/07’: 200 OK. Length: 95839 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character/201-400char’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/kancolle_134.html’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2010/05’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/mona_69.html’: 200 OK. Length: 73662 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/dance’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/12’: 200 OK. Length: 97287 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/daddy-cool’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/childrens-day’: 200 OK. Length: 96109 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/11’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/fusagiko’: 200 OK. Length: 91101 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/nabe’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/everybody’: 200 OK. Length: 92849 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2013/10’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/hotspring_6.html’: 200 OK. Length: 72320 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/mouse_14.html/comment-page-1’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/5’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/cake’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character/201-400char’: 200 OK. Length: 78122 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/destroy’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2010/05’: 200 OK. Length: 79791 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/01’.
INFO Fetched ‘http://2ch.en.utf8art.com/page/2’: 200 OK. Length: unspecified [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line/11-20line’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/daddy-cool’: 200 OK. Length: 77587 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/feed’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/11’: 200 OK. Length: 98064 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2012/07’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/dance’: 200 OK. Length: 76990 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/tag/nabe’: 200 OK. Length: 67180 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/usugeman_20.html’.
INFO Fetching ‘http://2ch.en.utf8art.com/tag/savanna’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2013/10’: 200 OK. Length: 96351 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/2015/11’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/cake’: 200 OK. Length: 76816 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/mouse_14.html/comment-page-1’: 200 OK. Length: 72457 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/ski’.
INFO Fetching ‘http://2ch.en.utf8art.com/arc/heater_4.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/destroy’: 200 OK. Length: 74928 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/sleep_13.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/01’: 200 OK. Length: 80690 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/character/1-100char’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line/11-20line’: 200 OK. Length: 93574 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line/1line’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2012/07’: 200 OK. Length: 81128 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/cat/line/6-10line’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/usugeman_20.html’: 200 OK. Length: 87374 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/dash’.
INFO Fetched ‘http://2ch.en.utf8art.com/feed’: 200 OK. Length: unspecified [text/xml; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/about’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/savanna’: 200 OK. Length: 91924 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/2015/11’: 200 OK. Length: 77132 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/iyahoo_14.html’.
INFO Fetching ‘http://2ch.en.utf8art.com/page/10’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/ski’: 200 OK. Length: 75159 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/syoboon_34.html/comment-page-1’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/heater_4.html’: 200 OK. Length: 72024 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/hair-disadvantaged’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/sleep_13.html’: 200 OK. Length: 72002 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/character/1-100char’: 200 OK. Length: 77290 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/sitemap.xml.gz’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line/1line’: 200 OK. Length: 65622 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/giko_26.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/cat/line/6-10line’: 200 OK. Length: 77677 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/arc/musical_instrument_6.html’.
INFO Fetched ‘http://2ch.en.utf8art.com/tag/dash’: 200 OK. Length: 76007 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/armored-trooper-votoms’.
INFO Fetched ‘http://2ch.en.utf8art.com/about’: 200 OK. Length: 48078 [text/html; charset=UTF-8].
INFO Fetching ‘http://2ch.en.utf8art.com/tag/christmas-tree’.
INFO Fetched ‘http://2ch.en.utf8art.com/arc/iyahoo_14.html’: 200 OK. Length: 73044 [text/html; charset=UTF-8].
INFO Fetched ‘http://2ch.en.utf8art.com/sitemap.xml.gz’: 200 OK. Length: 16384 [application/x-gzip].
INFO Fetched ‘http://2ch.en.utf8art.com/arc/syoboon_34.html/comment-page-1’: 200 OK. Length: 76446 [text/html; charset=UTF-8].
ERROR Fatal exception.
Traceback (most recent call last):
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/app.py", line 128, in run
    yield From(self._builder.factory['Engine']())
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/engine.py", line 281, in __call__
    yield From(self._run_workers())
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 252, in _step
    result = coro.send(value)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/engine.py", line 70, in _run_workers
    task.result()
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/futures.py", line 286, in result
    raise self._exception
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/engine.py", line 149, in _run_worker
    yield From(self._process_item(item))
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/engine.py", line 330, in _process_item
    yield From(self._process_url_item(url_record))
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/engine.py", line 387, in _process_url_item
    yield From(self._processor.process(url_item))
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/delegate.py", line 27, in process
    raise Return((yield From(self.web_processor.process(url_item))))
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/web.py", line 123, in process
    raise Return((yield From(session.process())))
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/web.py", line 215, in process
    yield From(self._process_loop())
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 250, in _step
    result = coro.throw(exc)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/web.py", line 274, in _process_loop
    exit_early, wait_time = yield From(self._fetch_one(self._request))
  File "/home/grab/.local/lib/python3.4/site-packages/trollius/tasks.py", line 252, in _step
    result = coro.send(value)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/web.py", line 339, in _fetch_one
    action = self._handle_response(request, response)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/web.py", line 455, in _handle_response
    request, response, self._url_item
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/processor/rule.py", line 423, in scrape_document
    request, response, url_item.url_record.link_type
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/scraper/base.py", line 186, in scrape_info
    scrape_result = scraper.scrape(request, response, link_type)
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/scraper/sitemap.py", line 38, in scrape
    for link in link_iter:
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/scraper/base.py", line 150, in iter_processed_links
    for link in self.iter_links(file, encoding):
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/document/sitemap.py", line 69, in iter_links
    for html_obj in self._html_parser.parse(file, encoding):
  File "/home/grab/.local/lib/python3.4/site-packages/wpull/document/htmlparse/html5lib_.py", line 39, in parse
    for token in tokenizer:
  File "/home/grab/.local/lib/python3.4/site-packages/html5lib/tokenizer.py", line 67, in __iter__
    while self.state():
  File "/home/grab/.local/lib/python3.4/site-packages/html5lib/tokenizer.py", line 275, in dataState
    chars = self.stream.charsUntil(("&", "<", "\u0000"))
  File "/home/grab/.local/lib/python3.4/site-packages/html5lib/inputstream.py", line 366, in charsUntil
    if not self.readChunk():
  File "/home/grab/.local/lib/python3.4/site-packages/html5lib/inputstream.py", line 268, in readChunk
    data = self.dataStream.read(chunkSize)
  File "/usr/lib/python3.4/codecs.py", line 491, in read
    newdata = self.stream.read(size)
  File "/usr/lib/python3.4/gzip.py", line 365, in read
    if not self._read(readsize):
  File "/usr/lib/python3.4/gzip.py", line 449, in _read
    self._read_eof()
  File "/usr/lib/python3.4/gzip.py", line 482, in _read_eof
    crc32, isize = struct.unpack("<II", self._read_exact(8))
  File "/usr/lib/python3.4/gzip.py", line 286, in _read_exact
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
CRITICAL Sorry, Wpull unexpectedly crashed.
CRITICAL Please report this problem to the authors at Wpull's issue tracker so it may be fixed. If you know how to program, maybe help us fix it? Thank you for helping us help you help us all.
INFO FINISHED.
INFO Duration: 0:01:40. Speed: 185.6 KiB/s.
INFO Downloaded: 204 files, 17.3 MiB.
INFO Exiting with status 1.
JustAnotherArchivist commented 1 year ago

Happened again recently on AB job dpo3p04ihp1d0gf9wpbdeyrk3, although I couldn't figure out which URL caused it. Two sitemaps were being retrieved at the time of the crash; one seems fine, the other's domain didn't even resolve.

JustAnotherArchivist commented 9 months ago

This happened once again today on AB job 3iul86mvgzh1j3qxc3mzvn81l and was caused by a corrupted sitemap.xml.gz file. Here's a reproducer with wpull 2.0.3:

wpull --recursive --sitemaps --delete-after --accept-regex '^https://safetyfirst\.airbus\.com/(robots\.txt|sitemap\.xml\.gz)?$' https://safetyfirst.airbus.com/

grab-site with ludios_wpull 3.0.9 does not crash on this. The crash doesn't seem to depend on the HTML parser in wpull 2.0.3. I didn't test with bare ludios_wpull.