Extend HTTP timeout

[joshtemple]

Change description

We were not backing off and retrying on manifest changes, which was by design since often the manifest is not there and we don't want to retry a 404. However, we can still retry timeouts, so I've added backoff for that endpoint for timeouts but not status errors.

We were not backing off and retrying on httpx.ReadError, so I've used the parent class NetworkError to catch this in our backoff logic.

We were only retrying most requests 3 times by default. With exponential backoff, this isn't waiting that long in total. I've increased the number of retries to 10 for any network error or 502 status. I've ensured we give up immediately and don't retry on status error that isn't a 502.

Type of change

• [x] Bug fix (fixes an issue)
• [ ] New feature (adds functionality)

Related issues

Checklists

Security

• [x] Security impact of change has been considered
• [x] Code follows security best practices and guidelines

Code review

• [x] Pull request has a descriptive title and context useful to a reviewer

[joshtemple]

Yes, and I'm still not sure why I'm seeing a 5 sec ReadTimeout in the logs. Thoughts on merging the backoff change for TimeoutException on the manifest call but removing the timeout changes?

[DylanBaker]

Yeah, I'm good with that.