upper bound too low for limit in fredr_category_series()

[shokru]

Hi! Thank you for this great package. I was wondering why the limit is capped at 1,000 in the fredr_category_series() function. I'm trying to access a category with much more items (https://fred.stlouisfed.org/categories/32291) => would it be possible to increase the limit to 100k? Thanks a lot in advance!

[DavisVaughan]

The limit is set by fred actually https://fred.stlouisfed.org/docs/api/fred/category_series.html#limit

The intention is that you also use offset to "paginate" your way through multiple pages of data so it doesn't have to send over a possibly massive amount of info in one call.

Ideally we'd handle this for you in fredr itself and make it an internal detail, but that would require some internal changes (i.e. some kind of loop where we request the maximum limit value and bump the offset until we don't get any more data back from the api)

[DavisVaughan]

It might be quite an undertaking, but switching to httr2 would allow fredr to use req_perform_iterative() to ease internal handling of this case https://github.com/r-lib/httr2/blob/824f142f048489d698673c4d2ada149b4e4c80c7/R/iterate.R#L6

[sboysel]

@DavisVaughan I agree with both sentiments: fredr ought to handle pagination internally but ideally by outsourcing the functionality from a dedicated dependency like httr2. Also a good opportunity to reevaluate some fredr internals.

@shokru In the meantime, you can take a look at something I mocked up for pagination a some time ago: https://github.com/sboysel/fredr/issues/68#issue-366539555