Open ivan opened 8 years ago
Happened again recently on AB job dpo3p04ihp1d0gf9wpbdeyrk3, although I couldn't figure out which URL caused it. Two sitemaps were being retrieved at the time of the crash; one seems fine, the other's domain didn't even resolve.
This happened once again today on AB job 3iul86mvgzh1j3qxc3mzvn81l and was caused by a corrupted sitemap.xml.gz
file. Here's a reproducer with wpull 2.0.3:
wpull --recursive --sitemaps --delete-after --accept-regex '^https://safetyfirst\.airbus\.com/(robots\.txt|sitemap\.xml\.gz)?$' https://safetyfirst.airbus.com/
grab-site with ludios_wpull 3.0.9 does not crash on this. The crash doesn't seem to depend on the HTML parser in wpull 2.0.3. I didn't test with bare ludios_wpull.
This crashes after about a minute: