search

HumanCellAtlas / dcp

Data Coordination Platform manifest and integration tests.
3 stars 1 forks source link

General Availability Coordination #441

Closed brianraymor closed 5 years ago

brianraymor commented 5 years ago

This is an informational epic to track pending issues required for General Availability.

jahilton commented 5 years ago

Also trying to include these 2 issues in this Epic, but getting a 404 error when trying to save the modified Epic that includes either/both issue. https://github.com/DataBiosphere/azul/issues/1098 https://github.com/DataBiosphere/azul/issues/1119

jahilton commented 5 years ago

Also included in this... https://app.zenhub.com/workspaces/dcp-5ac7bcf9465cb172b77760d9/issues/databiosphere/azul/1128

brianraymor commented 5 years ago

Updated. I added the epic that contains both azul/1098 and azul/1119.

theathorn commented 5 years ago

Here's the general sequence of events that needs to be verified prior to GA: 1) Complete re-ingestion of all Primary data sets (remaining: Tabula Muris) 2) Complete Secondary Analysis (Optimus and/or SS2) for all eligible data sets 3) Matrix Service manually updates database for all Secondary Analysis projects 4) Matrix Service generates per-project matrix output files into Data Browser S3 bucket 5) Azul runs script to generate all pre-project metadata TSV files (automation can be post GA) 6) Data Store tombstones all old bundles for re-ingested projects 7) Data Store tombstones any failed Integration Test projects in prod 8) Verify all projects complete in Data Browser 9) Enable user facing URL (data.humancellatlas.org) and DNS redirect "prod.data." to "data."

jahilton commented 5 years ago

It was previously decided not to wait on re-ingestion for GA. Also, contrary to the standing procedure, we will tombstone the old versions of the datasets even if the re-ingestion is not complete. So Item 1 can be removed, and Item 6 could move up in the sequence.

parthshahva commented 5 years ago

I think the tracker paints a good visual view of this list. We can't go to GA until all red boxes are resolved from https://tracker.data.humancellatlas.org

claymfischer commented 5 years ago

I can't emphasize enough just how useful this tracker is, @parthshahva.

kbergin commented 5 years ago

Item 2 can be removed from this list, we've said in many previous PM meetings on this topic that analysis should not block GA because it will always be ongoing / in progress. Whatever is currently there is what should be in the GA. cc @TimothyTickle

jahilton commented 5 years ago

@mckinsel is going to refine matrix-service/351 to something like "as part of matrix service project ETL, project-level matrices are automatically written to the correct data browser bucket with the correct headers" So it's no longer a GA blocker - I have removed it from the Epic