Orbit file script backoff/retry fix

collinss-jpl commented 5 months ago

Purpose

This PR fixes a bug in stage_orbit_file.py where the giveup function, which determines whether an automatic retry should occur or not, was not implemented correctly. This resulted in Orbit file downloads failing after only a single transient error was encountered, rather than once every ~15 seconds until success or 5 minutes have elapsed.

Issues

Fixes #796

Testing

A unit test has been added to test_stage_orbit_file.py which simulates all the transient error codes we check for during an orbit file download, and ensures that the download is retried for each failure code.

hhlee445 commented 5 months ago

Executed historical processing with 35 max ASG. Submitted 6237 slc_download jobs and 16 failed with the errors as requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://sentinel1.asf.alaska.edu/s3credentials Didn't get any orbit file download error.

collinss-jpl commented 5 months ago

Executed historical processing with 35 max ASG. Submitted 6237 slc_download jobs and 16 failed with the errors as requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://sentinel1.asf.alaska.edu/s3credentials Didn't get any orbit file download error.

I will add 502 to the list of error codes that we retry for so we hopefully see less of this type of failure going forward

nasa / opera-sds-pcm