mloesch / sickle

Sickle: OAI-PMH for Humans
Other
106 stars 42 forks source link

Handle retries using urllib3 Retry #44

Open danmichaelo opened 4 years ago

danmichaelo commented 4 years ago

Per #43, this is my take at replacing the custom retry code with the tried and tested urllib3 Retry class. Benefits include that it also handles Timeout and ConnectionError, and that the sleep time increases for each retry. Tests pass, and I'm also testing the branch in a real world scenario now, where I need to harvest from a server which is quite unstable, and happens to produce both timeouts, connection errors and 500 errors during updates. It seems to work fine so far!

I'm not sure if this should be the responsibility of sickle or the user, though.