Design and write a curation + ingress workflow script

[Bankso]

Discussed with @aditigopalan 1/24/24 - we would like to put together a shell script (or github workflow) that strings our suite of curation and ingress scripts together.

This will help streamline work by introducing a reusable/reproducible pipeline that can be run on a regular basis.

Pubmed crawler will remain separate from this workflow

High-level workflow steps (starting with manifests containing metadata from multiple grants) are outlined below, but will need to be expanded on.

Updated 2/26/24 to reflect current progress

Status: shell script written for this part of the process

• MC2 preliminary curation and ingress: split manifests by grant number --> identify target folder Synapse IDs --> build upload manifest CSV --> clean manifests --> update table schemas --> generate and assign primary keys for datasets and tools --> validate manifests --> upload manifests to grant-based Synapse projects

Status: all scripts are written, but have not been written into a pipeline yet

• Database management steps (regular curation): update defining SQL for UNION tables --> run QC on metadata in UNION tables --> store QC results in Synapse --> sync metadata from admin tables to CCKP database

Status: subset of scripts are written. A workflow for this is lower priority, so we can address it later.

• • Database management steps (new grant onboarding): create Synapse project, team for new grants --> store grant and team info in GrantView manifest, alongside other grant info --> upload GrantView manifest to CCKP admin Synapse project --> update master fileview scope to include new projects** --> update all tables view scope to include new projects --> sync new grant information to CCKP database

*== script not yet written, step performed manually == script written, requires updates ^== step has not been implemented manually or programmatically

[aditigopalan]

MC2 Curation and Ingress steps workflow here

[Bankso]

PR #39 adds initial version of workflow shell script and associated readme. Let's plan on updating the shell script to reflect any missing steps and (if it makes sense) converting it to a github action config

[Bankso]

Updated description to reflect current status on this. Note that the database sync was implemented via PR https://github.com/mc2-center/mc2-center-dcc/pull/53

[aclayton555]

This got larger than expected, opportunity to break this down further and align on what still needs to be implemented. @aditigopalan will structure this work accordingly in tickets for our kick off later this week.