Add retry logic to Open AQ fetch

[amehta-scottlogic]

Acceptance Criteria

• If response from OpenAQ times out, retry the request to a max of 3 times
• No need for any backoff or waits between tries
• The city and URL that the request failed for is logged only if it fails three times

Test Checklist:

• [ ] Test Analysis & discussion with developer
• [ ] Write Test Cases / Charters
• [x] Test Cases & Charters Reviewed
• [ ] Test Cases Executed
• [ ] Test Charters Executed
• [x] Regression tests executed
• [x] Automation Tests added to project

[mwalker-scottlogic]

forecast_api_seeded_data_tests.py, forecast_api_validation_tests.py, cams_known_grib_test.py, forecast_calibration_tests.py & open_aq_etl_tests.py tests all passed.

Executed manual test cases, had to make some changes to test assertions, but expected behaviour was observed

[mwalker-scottlogic]

Test Analysis was not written for this ticket in time.

Automation added by @rstrange-scottlogic to cover:

• 3 timeouts creates an error and a log
• 1 timeout followed by success doesn't error or log
• A non timeout error doesn't retry, errors and logs

I have reviewed and added a couple of questions:

• Do we know that OpenAQ responds with a 408 if the connection times out. This will be hard to test, so we may need to rely on the API being correctly designed and look out for unusual behaviour
• Testing that a 500 gets a log 'Response for London contained no results' after 1 call - the logic for this isn't actually described in the ticket, is this defined elsewhere?
• 'The city and URL that the request failed for is logged only if it fails three times' - We don't check the logs if it fails twice and passes once, but have tested if it fails once and then passes - would adding this test condition be excessive do you think?

Received responses on these:

• I included this to ensure we weren't retrying for every error case - the ticket specifies to do the retry on timeout, so I thought I should cover another case which could retry
• Yeah I thought about that and just figured it would be overkill - I figured checking there is a limit to the number of retries, and that it no longer retries after a success would do it.

I'm happy with this