What problem does this address?
ETL's Zenodo data, and grabs two more useful values from the archiving process.
What did you change in this PR?
Update archiver to get publication date and version for each version record
Add raw_ and core_ Zenodo outputs
Add a dataset_slug column that makes it easy to identify what dataset each record corresponds to
Write the core_ output to a DB
Add column descriptions to model for new table
NEW: reconfigure archiver to grab a JSON with metadata for all records in a version, then extract these. This enables us to save everything, and drop what we don't want later, rather than only archiving a few variables.
NEW: Move make_request to helpers.py
Testing
How did you make sure this worked? How can a reviewer verify this?
Generate raw and core assets.
To-do list
- [x] Review the PR yourself and call out any questions or issues you have
Overview
Closes #181.
What problem does this address? ETL's Zenodo data, and grabs two more useful values from the archiving process.
What did you change in this PR?
raw_andcore_Zenodo outputsdataset_slugcolumn that makes it easy to identify what dataset each record corresponds tocore_output to a DBmake_requesttohelpers.pyTesting
How did you make sure this worked? How can a reviewer verify this? Generate raw and core assets.
To-do list