ArchiveTeam / ArchiveBot

ArchiveBot, an IRC bot for archiving websites
http://www.archiveteam.org/index.php?title=ArchiveBot
MIT License
356 stars 72 forks source link

Status code of URL list downloads isn't checked #510

Open JustAnotherArchivist opened 3 years ago

JustAnotherArchivist commented 3 years ago

DownloadUrlFile does not verify that the server responded with an HTTP 200. This morning, there was an issue, which lead to lots of errors and occasional 502s. The latter were saved to the -urls.txt file, which obviously then led to further errors. Should probably use r.raise_for_status to take care of that.