Add retry mechanism to HTTP requests

[michaelwood]

We've seen a few cases where an HTTP request fail and then subsequently work. Retrying a request is fairly standard due to the nature of the internet so it would be a useful feature to add to the datagetter.

[michaelwood]

Add HTTP retry mechanism to the datagetter

[michaelwood]

example code:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

def get_data(url):
    retry_strategy = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        method_whitelist=["HEAD", "GET", "OPTIONS"]
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session = requests.Session()
    session.mount("http://", adapter)
    session.mount("https://", adapter)

    response = session.get(url)

    if response.status_code != 200:
        response.raise_for_status()

    return response.content

[michaelwood]

include all tries including those to the registry when we may get invalid json due to connection problem with SF