search

ropensci / openalexR

Getting bibliographic records from OpenAlex
https://docs.ropensci.org/openalexR/
Other
89 stars 19 forks source link

`oa_generate()` and coro request returning page instead of record? #219

Closed rkrug closed 3 months ago

rkrug commented 3 months ago

Hi

I am in the process of downloading 10 million records using opanalexR (grouped by year) and am I using my modified version of oa_request with the changes here https://github.com/IPBES-Data/IPBES.R/blob/ba08530b94e29e6e82bd1d9a000e47c67074d1ec/R/oa_request_IPBES.R#L241-L254.

This works nicely and reliable. It seems that I could have used Coro for that. but that fetches one record per time if I understand it correctly, which makes it really slow. So would it be possible, to include a parameter which specifies if Coro goes per record or per page? In this case I could have used that one instead.

trangdata commented 3 months ago

Hi @rkrug so I don't think it would be slow actually. coro does return one record at a time for ease of use but it still actually makes request for 200 records at a time, i.e., a new request won't be triggered until you reach record 201.

rkrug commented 3 months ago

OK - makes sense. Then I could use it (with a variable parameter for the length of the sets saved). Nice.