[July - Sep] Dataset Curation Workflow Tracking

[aditigopalan]

This ticket tracks curation workflow progression.

Note: It is possible for work to take place simultaneously in the three sections with overlapping periods, allowing curation workflows for different months to coincide.

1. Curation and Annotation

• [ ] Run Pubmed crawler to generate PublicationView manifest [205 publications generated, long sprint anticipated]

• [ ] Send Amber and Jineta a copy of the PublicationView manifest from latest crawl to review for MC2 Center Newsletter publication highlights

• [ ] Send Amber "News from CCKP" for MC2 Center Newsletter

• [ ] Annotate publications in PublicationView manifest [In progress]

• [x] Generate ToolView and DatasetView manifests based on PublicationView manifest

• [ ] Run the automated curation workflow to upload publications, datasets and tools [This includes splitting manifests, processing and validating manifests, generating target synapse IDs for upload, schema updates, upload to synapse and (in progress) a validation check for uploads)

• [ ] Generate UNION tables

• [ ] QC of staging tables

• [ ] Performing automate portal sync to CCKP

• [ ] Validate data on the CCKP

  Status check [Plan to report numbers for each category following pubmed crawl]:

• [ ] Publication upload [ ]

• [ ] Tool upload [ ]

• [ ] Data set upload [ ]

2. Data model

• [ ] Update valid values in the data model and build
• [ ] Generate templates from new model
• [ ] Release new model version [no changes, no release]
• [ ] Update DCA config [no changes, no update]

3. Contributor Engagement

• [ ] Emails to contributors (by grant) with info on newly added manifests, link to project, DCA, and instructions on review, annotation, validation, and submission, as applicable
• [ ] Jira Help Desk ticket tracking, review, triage, with data model updates, as needed (TBD)
• [ ] Annotation gap-filling of DatasetView and ToolView manifests

[aclayton555]

24-11/12 Initiate after:

• [ ] https://github.com/mc2-center/mc2-center-dcc/issues/74
• [ ] #145

Details within existing docs. Overall, process is the same, but metadata are different.