v.2.0 dataset release - wf updates

[ergonyc]

For the August release of ASAP CRN datasets several small updates have been made to the PMBDS workflows.

IMPLIMENTED

• [x] data manifest updates: Pull Req #54
• [x] data integrity testing updates: Pull Request #56

WIP

Bug fixes

• [x] #62

Curation

• [x] pre-filtered metadata summary (to capture effect of filtering)
• [x] #68
• [x] #70
• [x] #71

Enhancements

• [x] #69

Upgrades

• unresolved. Slated for v3.0

Unresolved. Pushed to v3.0

• [ ] CellRanger/cellbender QC metrics added to metadata
• [ ] #63
• [ ] UPGRADES:
  • [ ] #60 (medium priority)
  • [ ] #61 (low priority)

curated artifacts summary

added artifacts:

entry point 1

• ${cohort_id}.initial_metadata.csv

• ${cohort_id}.merged_adata_object.h5ad

  feature metadata

• ${cohort_id}.all_genes.csv
• ${cohort_id}.hvg_genes.csv

scvi model

• ${cohort_id}_scvi_model.tar.gz

  entry point 2

• ${cohort_id}.final_adata.h5ad (renamed)
• ${cohort_id}.final_metadata.csv

asap-dev-data-{cohort,team-xxyy}
├── cohort_analysis
│   ├── ${cohort_id}.sample_list.tsv
│   ├── ${cohort_id}.initial_metadata.csv
│   ├── ${cohort_id}.merged_adata_object.h5ad
│   ├── ${cohort_id}.doublet_score.violin.png
│   ├── ${cohort_id}.n_genes_by_counts.violin.png
│   ├── ${cohort_id}.pct_counts_mt.violin.png
│   ├── ${cohort_id}.pct_counts_rb.violin.png
│   ├── ${cohort_id}.total_counts.violin.png
│   ├── ${cohort_id}.final_validation_metrics.csv
│   ├── ${cohort_id}.all_genes.csv
│   ├── ${cohort_id}.hvg_genes.csv
│   ├── ${cohort_id}_scvi_model.tar.gz
│   ├── ${cohort_id}.cell_types.csv
│   ├── ${cohort_id}.final_adata.h5ad
│   ├── ${cohort_id}.final_metadata.csv
│   ├── ${cohort_id}.scib_report.csv
│   ├── ${cohort_id}.scib_results.svg
│   ├── ${cohort_id}.features.umap.png
│   ├── ${cohort_id}.groups.umap.png
│   └── MANIFEST.tsv

[ergonyc]

We ought to compile the CellRanger QC metrics into tabular form and pass along as part of the metadata. Potentially also some of the auxilary CellBender metrics.

TODO: enumerate which metrics

[ergonyc]

I think we need to add a "pre-filtered" summary of metadata and possibly an unfiltered "merged" adata object. i.e the output of plot_qc_metrics.py. That will offer users an easy entry point for alternative processing / filtering. e.g. if the user wanted to develop an "atlas" of cells / cell-types for a specific brain region, they would harmonize a subset of the overall cohort.

[ergonyc]

We should also upate scvi-tools to v1.15 and python to 3.12 at some point. Maybe after the next release... maybe with the other updates / testing

[ergonyc]

There's a subtle bug in the cellassign code which estimates the library size parameter after subsetting to the marker genes. This should be calculated before subsetting.

[ergonyc]

Looks like grouping the v2.0 release changes into these categories will be useful:

Bug fixes

• [x] #62

Curation

• [ ] CellRanger QC metrics added to metadata
• [ ] pre-filtered metadata summary (to capture effect of filtering)
• [ ] artifact naming
• [ ] #63

Upgrades

• [ ] #61
• [ ] #60

[ergonyc]

Final code changes:

• [x] #66
• [ ] #67
• [x] #69
• [ ] integration performed over samples rather than batch_id #70
• [ ] #68
• [ ] #71

also updated in opening comment.

[ergonyc]

@kfang4

Here's a summary of the artifacts that should be curated...

curated artifacts summary

Added artifacts:

entry point 1

• ${cohort_id}.initial_metadata.csv

• ${cohort_id}.merged_adata_object.h5ad

  feature metadata

• ${cohort_id}.all_genes.csv
• ${cohort_id}.hvg_genes.csv

scvi model

• ${cohort_id}_scvi_model.tar.gz

  entry point 2

• ${cohort_id}.final_adata.h5ad (renamed)
• ${cohort_id}.final_metadata.csv

asap-dev-data-{cohort,team-xxyy}
├── cohort_analysis
│   ├── ${cohort_id}.sample_list.tsv
│   ├── ${cohort_id}.initial_metadata.csv
│   ├── ${cohort_id}.merged_adata_object.h5ad
│   ├── ${cohort_id}.doublet_score.violin.png
│   ├── ${cohort_id}.n_genes_by_counts.violin.png
│   ├── ${cohort_id}.pct_counts_mt.violin.png
│   ├── ${cohort_id}.pct_counts_rb.violin.png
│   ├── ${cohort_id}.total_counts.violin.png
│   ├── ${cohort_id}.final_validation_metrics.csv
│   ├── ${cohort_id}.all_genes.csv
│   ├── ${cohort_id}.hvg_genes.csv
│   ├── ${cohort_id}_scvi_model.tar.gz
│   ├── ${cohort_id}.cell_types.csv
│   ├── ${cohort_id}.final_adata.h5ad
│   ├── ${cohort_id}.final_metadata.csv
│   ├── ${cohort_id}.scib_report.csv
│   ├── ${cohort_id}.scib_results.svg
│   ├── ${cohort_id}.features.umap.png
│   ├── ${cohort_id}.groups.umap.png
│   └── MANIFEST.tsv