infrae / pyoai

The oaipmh module is a Python implementation of an "Open Archives$ Initiative Protocol for Metadata Harvesting"
http://pypi.python.org/pypi/pyoai
Other
83 stars 53 forks source link

allow customization of "retry" policy. #32

Closed adimascio closed 6 years ago

adimascio commented 6 years ago

When the harvested OAI server returns an HTTP 503, the default policy is to retry 5 times and wait 120 seconds between each try. Due to the variety of OAI server implementations, one might want to configure those parameters. This policy can be customized through the BaseClient.custom_retry_policy's parameter. For instance::

>>> client = Client('http://the-oai-base-url.org', custom_retry_policy={
        # retry on both 500 and 503 HTTP return codes
        'expected-errcodes': {500, 503},
        # wait for 30 seconds before retrying
        'wait-default': 30,
        # retry 10 times
        'retry': 10,
    })

)

adimascio commented 6 years ago

An alternative would be to use https://github.com/jd/tenacity but that would require and external dependency.

adimascio commented 6 years ago

Any chance this PR be reviewed any time soon ?

jascoul commented 6 years ago

Thanks, this looks really great, I've just merged and updated the HISTORY.txt