search

lmullen / cchc

America's Public Bible for Computing Cultural Heritage in the Cloud
Creative Commons Zero v1.0 Universal
8 stars 1 forks source link

Slow down if rate limited #2

Closed lmullen closed 3 years ago

lmullen commented 3 years ago

Sometimes the API gives an HTTP 429 Too Many Requests error even with the rate limiters in place. On receiving such an error, it would probably be best to take some kind of action for the crawler as a whole and delay. Otherwise, the IP address might get temporarily locked out.

lmullen commented 3 years ago

Quitting works for now. This could be handled with the Docker restart policy as much as anything and doesn't need to be an application level concern.