Closed dmolesUC closed 2 years ago
It looks like recent versions of Faraday support the Retry-After
header, and the following workaround seems to do it:
require 'oai'
require 'faraday_middleware'
http_client = Faraday.new do |conn|
conn.request(:retry, max: 5, retry_statuses: 503)
conn.response(:follow_redirects, limit: 5)
conn.adapter :net_http
end
client = OAI::Client.new(base_url, http: http_client)
opts = {from:'2012-03-01', until:'2012-04-01', metadata_prefix:'arXiv'}
puts client.list_records(opts).full.count
I haven't tested whether this behaves correctly in the event of a 503 without Retry-After, however.
The challenge here is that this gem is not very opinionated about faraday versions, which means it can't reliably spec a dependency on faraday-middleware or faraday-retry. Hmm.
Thank you for documenting an approach from a client application's perspective, by the way! this might be the best resolution.
@dmolesUC I've added your example to the README (thanks!); I don't think we can do more than that until/unless we specify faraday versions more narrowly.
I first noticed this when attempting to retrive several thousand records from arXiv (which imposes a 20-second delay) with
.full
and resumption tokens, but you can see the same thing if you just try to retrive one page of records twice in quick succession.Steps to reproduce:
Expected:
Actual:
I tried monkey-patching
OAI::Client#get
to print some debug info:and reran the second case above, confirming that we're getting a 503 with retry-after as per the OAI spec.
It looks like there's a related pull request #29, although it would be nice if both the number of retries and max retry wait time were configurable.