Add the publish step - Githubissues

sul-dlss-labs / rialto-airflow

Airflow for harvesting data for research intelligence and open access analysis

Apache License 2.0

1 stars 0 forks source link

Add the publish step #92

Closed edsu closed 3 months ago

edsu commented 3 months ago

The harvest DAG has a publish step, but it isn't currently doing anything. I think we should probably copy (or symlink) the newly created pubs and contribs parquet files in the snapshots folder to a place where the Jupyter Notebook environment can always find them, perhaps:

/data/latest/publications.parquet
/data/latest/contributions.parquet

lwrubel commented 3 months ago

Requested getting /data mounted to the sul-notebook VM: https://github.com/sul-dlss/operations-tasks/issues/3835

lwrubel commented 3 months ago

Share is mounted now.