Implement retries for spurious connection issues

[bennybp]

Description

Requests that result in certain errors due to spurious connection/networking issues should now be automatically retried.

The types of exceptions handled should cover most cases, but I am open to expanding the list if people find ones that I missed.

The request will be tried again after waiting 0.5, 1.0, 2.0, and 4.0 seconds (since the previous request)

Seem reasonable @dotsdl? Closes #741

Changelog description

Requests now will be automatically retried in case of connection or networking issues

Status

• [X] Code base linted
• [X] Ready to go

[codecov[bot]]

Codecov Report

Merging #758 (3eef76e) into main (e8d9cba) will decrease coverage by 0.02%. Report is 6 commits behind head on main. The diff coverage is 96.87%.

Additional details and impacted files

[dotsdl]

Reviewing now! Thanks for this @bennybp!

[dotsdl]

retry looks kinda dead; looks like backoff may be a better bet for avoiding bitrot?

[bennybp]

If adding another dependency seems overkill for this, it's not too hard to implement your own solution

Let me think about just hardcoding a solution. We are only using it in one place, so it it's not be too bad to implement it myself (using the aforementioned class attributes)

[bennybp]

Ok implemented by hand now. It's pretty simple :)

There is only one other place I would consider adding this kind of logic (when dealing with certain database operations). But it will end up looking a lot different, so I'm not worried about duplicating code.

[dotsdl]

Giving this another review today!