Provider needs a way to iterate over data that is provided in paginated web requests, or separate files by date.
There are too many variables to define a universal DSL for how to iterate:
How do you find the number of pages: when do you stop, maybe just when you run out of records?
What do you step by: is it a date, or a sequential number?
What part of the request needs to increment: is it a different path, a query param, or in the POST body?
So, we'll define a "repeat" block that lets the user yield each instance of the provider with changed parameters:
provider :soap do
# ... same
end
reader :xml do
# ... same
end
# NEW!
repeat do |soap|
total_pages = /<Data totalPages="(\d+)"/.match( soap.data )[1].to_i # e.g.
1.upto total_pages do |next_page|
yield soap
soap.message["getPage"] = next_page
end
end
The job can fetch these in turn and parse them into the list of records. It should automatically clear the data after each yield to force a re-fetch on each iteration.
avit
commented
10 years ago
Provider needs a way to iterate over data that is provided in paginated web requests, or separate files by date.
There are too many variables to define a universal DSL for how to iterate:
So, we'll define a "repeat" block that lets the user yield each instance of the provider with changed parameters:
The job can fetch these in turn and parse them into the list of records. It should automatically clear the data after each yield to force a re-fetch on each iteration.