Open JustAnotherArchivist opened 4 years ago
Real-world example of something that effectively can't be archived due to this without ridiculous manual effort: ShoutWiki. It refuses connections in a pretty much random manner, which then wrecks any recursive crawls.
ArchiveBot does not retry
[Errno 111] Connection refused
errors. I think it should and consider this a bug. This simply requires adding the wpull option--retry-connrefused
.