Closed blackblitz closed 4 years ago
Seems like the bug is here:
https://github.com/google/corpuscrawler/blob/master/Lib/corpuscrawler/util.py#L756
That function should be more lenient when there is a 404. In this case, it found a link to /page/2
in this category, but that page is actually a 404, and that should not be a fatal error.
I would probably change
assert pgdoc.status == 200, (pgdoc.status, pgurl)
to something more like
if pgdoc.status != 200:
print("Error %3d: %s" % (pgdoc.status, pgurl))
next
Can you open a PR?
Closing as fixed; let me know if you have any more issues.
I ran
./corpuscrawler --language=my-t-d0-zawgyi --output=./corpus
(with python 2.7 on Ubuntu 18.04) and the program crashed while downloading from some url. The output is shown below.