Feature/crossref events doi prefix query

The-Academic-Observatory / oaebu-workflows

Telescopes, Workflows and Data Services for the 'Book Analytics Dashboard Project (2022-2025)', building upon the project 'Developing a Pilot Data Trust for Open Access eBook Usage (2020-2022)'

https://documentation.book-analytics.org/

Apache License 2.0

5 stars 0 forks source link

Feature/crossref events doi prefix query #226

Closed jdddog closed 6 months ago

jdddog commented 6 months ago

Re-work the ONIX Workflow so that it fetches Crossref Events in a more reliable way (hopefully):

Fetch events via DOI prefix rather than individual DOIs.
Remove threaded requests, since multiple workflows will be running anyway.
To make better use of memory: save events to disk, then when transforming, read each event, transform it and write in same loop.