search

inspirehep / inspire-next

The INSPIRE repo.
https://inspirehep.net
GNU General Public License v3.0
59 stars 69 forks source link

workflows: json_api_request should be fault-tolerant #2355

Closed jacquerie closed 7 years ago

jacquerie commented 7 years ago

Sentry: https://sentry.cern.ch/inspire-sentry/inspire-labs/group/822046

Unlike https://github.com/inspirehep/inspire-next/pull/2349, this can be implemented using requests only.

chris-asl commented 7 years ago

Could we do something like the suggestion here?

Mount an HTTPAdapter using the max_retries parameter (more) on the session object with a url pattern, which can be the url in the post method.

Regarding the max_retries parameter, we can have more control over how many retries or exponential backoff factor per request by using the Retry object (more).

How does this approach look? @jacquerie @david-caro

jacquerie commented 7 years ago

Yes, this is the right solution.

jacquerie commented 7 years ago

Moreover we found out that Magpie is hosted in the Wigner data center, which explains why this problem is localized to this particular machine.

chris-asl commented 7 years ago

Actually the solution in the linked StackOverflow question doesn't fully fit our needs.

It only provides a retry mechanism (optionally, with back-off) for when receiving certain HTTP error codes (e.g. 500).

In the case of an exception, e.g. ConnectionError, as happened on Sentry, we need retry logic, i.e. retrying on exception which isn't offered by requests (source).

@jacquerie Could this be a good fit for using backoff?
(and a combination of the retry logic of requests for certain HTTP error codes)