search

r-lib / httr2

Make HTTP requests and process their responses. A modern reimagining of httr.
https://httr2.r-lib.org
Other
238 stars 59 forks source link

Fetch data from paginated API call with an equivalent of resp_body_json? #565

Closed deschen1 closed 1 week ago

deschen1 commented 2 weeks ago

In a single API call, I can fetch the relevant data, using httr2::resp_body_json(), see here: https://httr2.r-lib.org/reference/resp_body_raw.html.

However, when I collect data from an API that uses pagination and I'm using httr2's req_perform_iterative() function, I'm getting a long list of json objects.

Currently I'm doing some purrr::map magic to pull the relevant response bodies from the response object, but I'M wondering if there's a more straightforward way or resp_body_json() equivalent that also works with such pagination responses.

hadley commented 1 week ago

https://httr2.r-lib.org/reference/resps_successes.html ?

deschen1 commented 1 week ago

Awesome, I must have overlooked these helpers. With an additional flatten() as the first step, these functions also work nicely in case there is an additional list layer (e.g. when combining fetching from an paginated API that is wrapped in a loop/map after building a list of requests (e.g. one per country)).