edgi-govdata-archiving / web-monitoring-processing

Tools for access, "diff"-ing, and analyzing archived web pages
https://edgi-govdata-archiving.github.io/web-monitoring-processing
GNU General Public License v3.0
20 stars 20 forks source link

Correct configured DB client URLs and backoff #843

Closed Mr0grog closed 1 year ago

Mr0grog commented 1 year ago

When configuring the DB client, you might specify a URL with a trailing slash (e.g. https://abc.org/) and wind up with it requesting URLs like https://abc.org//api/v0/pages (note the double slash). Our API handles this fine (yay), but it would be better if that didn't happen in the first place, so this fixes it.

While doing that, I also noticed we could do with some backoff when monitoring import statuses, and added that, too.