search

AgentschapPlantentuinMeise / TETTRIs-mapping-taxonomists

TETTRIs WP3, task 3.2: automatic mapping of taxonomic expertise
MIT License
0 stars 2 forks source link

paging #2

Closed qgroom closed 1 month ago

qgroom commented 1 year ago

When you pull data from OpenAlex you use a page length of 200, but you don't use paging. You get away with this because it only pulls back 161 lines. However, there is a big danger in future runs that you will pull back 200 hundred rows and you won't notice that you are loosing data. If you don't want to implement paging yet, it would be good to introduce a trap so that if you retrieve the same number of rows of the page length, the script should warm you that there might be a problem.

qgroom commented 1 year ago

I should have said. This refers to 1-2 Retrieving articles from taxonomic journals.ipynb

qgroom commented 2 months ago

@melanie-de-nolf can you give an update on this. Perhaps this issue can be closed

melanie-de-nolf commented 1 month ago

All requests from the OpenAlex API are now made with functions from the download.py (request_works and request_sources) and these functions include paging