search

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
7 stars 2 forks source link

Support for HCA bionetworks and atlases #5519

Closed bvizzier-ucsc closed 3 months ago

bvizzier-ucsc commented 1 year ago

Add support to Azul for indexing of bionetworks and atlas per HCA issue 1520.

The Slack discussion surrounding this can be found at: https://humancellatlas.slack.com/archives/C01360XN04S/p1680708729364499

achave11-ucsc commented 1 year ago

Assignee to review if this needs more info or not.

achave11-ucsc commented 1 year ago

Assignee to start conversation with CC about specifying their needs.

bvizzier-ucsc commented 1 year ago

@NoopDog Do you have inputs to this request as to how the Data Browser would like to see this information presented?

NoopDog commented 1 year ago

Hi @bvizzier-ucsc I will check with you on Slack to verify I understand this request. The current mock is here:

image

I think we need to:

  1. Revisit the metadata changes that have been made to support the Atlas pages.
  2. Review any data that has been ingested using the new schema updates and compare that with the full set of data required to represent the current/upcoming Atlases.
bvizzier-ucsc commented 1 year ago

@hannes-ucsc: "@noopdog to provide the names of the metadata properties that CC needs exposed as additional facets."

NoopDog commented 1 year ago

Here is some background and an initial proposal on mapping the new metadata schema elements to new term facets and entity attributes to accomplish the intended use cases.

Use Cases

This effort is meant to enable the following use cases:

  1. See BioNetworks that have HCA Atlases in the HCA Data Repository
  2. List HCA Atlases in the HCA Data Repository
  3. List HCA Atlases in the HCA Data Repository published by a given BioNetwork
  4. Find projects in the HCA Data Repository used by a BioNetwork's HCA Atlases
  5. Find Projects in the HCA Data Repository used by/integrated into a given HCA Atlas

Data Model

The following new user concepts are introduced:

  1. HCA Altas - This is an "Integration Object" resulting from combining/processing data from multiple projects. The atlas format will initially be a cellxgene matrix in .h5ad and .rds formats. Atlases will be versioned and vary by tissue and whether the tissue or donor was healthy or diseased. HCA Atlases represent an HCA community consensus of the analysis they present. This extra curation/consensus distinguishes HCA Atlases from other Atlases published by the single-cell community.
image
  1. BioNetwork - there are 18 tissue-specific "Biological Networks". These are research organizations funded to produce HCA Atlases.
image

Relationship Between Biological Networks, HCA Atlases, and Projects

Biological Networks

  1. Biological Networks can have zero or more Atlases.

HCA Atlases

  1. HCA Atlases can use 0 or more Projects (sometimes Atlas datasets are not ingested into the HCA Data Repository or may even be unpublished).
  2. HCA Atlases can belong to a single Biological Network.

Projects

  1. Projects can be an atlas or not.
  2. Projects can be used by zero or more atlases.

Metadata

To enable this, the metadata schema was updated so that Projects can be associated with zero or more HCA Bionetwork entries.

These hca_bionetwork entries specify:

  1. If the project is an atlas, and if so, its name, version, and BioNetwork.
  2. If the project is used in an atlas, e.g. if it is a "Source Dataset" for a given atlas, and if so, the atlas name, version, and network.

Azul Indexer Support

Below is an initial proposal of new term facets and related entity attributes (for the project, sample, and file entities) to accomplish the abovementioned use cases.

  1. atlasProject - true if the project has an hca_bionetworks array entry where atlas_project is yes false otherwise.
  2. atlas - an array of the concatenation of project.hca_bionetworks[x].hca_tissue_atlas + " " + project.hca_bionetworks[x].hca_tissue_atlas_version for each of the project's hca_bionetworks array entries.
  3. bionetworkName - an array of the distinct project.hca_bionetworks[x].name values of the project's hca_bionetworks array entries.
achave11-ucsc commented 1 year ago

Spike for design based on above requirements.

hannes-ucsc commented 1 year ago

Are there any existing snapshots in TDR that use the updated schema for their project entities?

dsotirho-ucsc commented 1 year ago

Assignee to ask wranglers.

NoopDog commented 1 year ago

The current assignee is @bvizzier-ucsc. Do you mean him @dsotirho-ucsc? Using people's names when discussing assignments might be helpful it's also more personable! —just a thought.

Cheers, D

hannes-ucsc commented 1 year ago

Thank you, but the assignment of @bvizzier-ucsc was intentional.

dsotirho-ucsc commented 1 year ago

https://humancellatlas.slack.com/archives/C01360XN04S/p1697742636902909

dsotirho-ucsc commented 1 year ago

@hannes-ucsc: "In the above thread EBI reports that no projects with the new schema currently exist. Assignee to follow up with HCA."

bvizzier-ucsc commented 1 year ago

Update from Gaby:

"We’ll find out which projects already have this info, and send that over tomorrow."

bvizzier-ucsc commented 1 year ago

According to Ida Zucchi, the following datasets should have the atlas fields:

577c946d-6de5-4b55-a854-cd3fde40bff2 kidney
c893cb57-5c9f-4f26-9312-21b85be84313 gut
bc5512cc-9544-4aa4-8b75-8af445ee2257 lung
92892ab2-1334-4b1c-9761-14f5a73548ea lung
1fac187b-1c3f-41c4-b6b6-6a9a8c0489d1 kidney,gut
dsotirho-ucsc commented 1 year ago

Continue spike to review existing metadata in the snapshots for above projects.

hannes-ucsc commented 1 year ago 
$ grep -E '577c946d|c893cb57|bc5512cc|92892ab2|1fac187' deployments/prod/environment.py
    mksrc('datarepo-b0a40e19', 'hca_prod_577c946d6de54b55a854cd3fde40bff2__20211129_dcp1_20211129_dcp1', 6),
    mksrc('datarepo-6b1109e5', 'hca_prod_577c946d6de54b55a854cd3fde40bff2__20220117_dcp2_20220120_dcp12', 7),
    mksrc('datarepo-2b95c100', 'hca_prod_c893cb575c9f4f26931221b85be84313__20220118_dcp2_20220121_dcp12', 20),
    mksrc('datarepo-c89e2c59', 'hca_prod_1fac187b1c3f41c4b6b66a9a8c0489d1__20230427_dcp2_20230501_dcp27', 129),
    mksrc('datarepo-a0198d42', 'hca_prod_92892ab213344b1c976114f5a73548ea__20230616_dcp2_20230616_dcp29', 10),
    mksrc('datarepo-41efd06c', 'hca_prod_bc5512cc95444aa48b758af445ee2257__20230614_dcp2_20230616_dcp29', 168),
    mksrc('datarepo-47aed999', 'hca_prod_c893cb575c9f4f26931221b85be84313__20220118_dcp2_20230815_dcp30', 21),
    mksrc('datarepo-d6c0be70', 'hca_prod_577c946d6de54b55a854cd3fde40bff2__20220117_dcp2_20230905_dcp31', 21),
    mksrc('datarepo-4450b12b', 'hca_prod_92892ab213344b1c976114f5a73548ea__20230616_dcp2_20230905_dcp31', 10),
    mksrc('datarepo-0287a0ba', 'hca_prod_577c946d6de54b55a854cd3fde40bff2__20220117_dcp2_20231002_dcp32', 21),
          SELECT project_id, content FROM `datarepo-c89e2c59.hca_prod_1fac187b1c3f41c4b6b66a9a8c0489d1__20230427_dcp2_20230501_dcp27.project`
UNION ALL SELECT project_id, content FROM `datarepo-0287a0ba.hca_prod_577c946d6de54b55a854cd3fde40bff2__20220117_dcp2_20231002_dcp32.project`
UNION ALL SELECT project_id, content FROM `datarepo-4450b12b.hca_prod_92892ab213344b1c976114f5a73548ea__20230616_dcp2_20230905_dcp31.project`
UNION ALL SELECT project_id, content FROM `datarepo-41efd06c.hca_prod_bc5512cc95444aa48b758af445ee2257__20230614_dcp2_20230616_dcp29.project`
UNION ALL SELECT project_id, content FROM `datarepo-47aed999.hca_prod_c893cb575c9f4f26931221b85be84313__20220118_dcp2_20230815_dcp30.project`
[{
  "project_id": "577c946d-6de5-4b55-a854-cd3fde40bff2",
  "content": "{\"contributors\":[{\"name\":\"Benjamin,D,Humphreys\",\"email\":\"humphreysbd@wustl.edu\",\"phone\":\"(+1) 314-362-1575\",\"institution\":\"Washington University in St. Louis\",\"laboratory\":\"Department of Medicine, Division of Nephrology\",\"address\":\"600 S. Euclid Ave. CB 8126, St. Louis, MO, 63110\",\"country\":\"USA\",\"corresponding_contributor\":true,\"project_role\":{\"text\":\"principal investigator\",\"ontology\":\"EFO:0009736\",\"ontology_label\":\"principal investigator\"},\"orcid_id\":\"0000-0002-6420-8703\"},{\"name\":\"Parker,C,Wilson\",\"email\":\"parkerw@wustl.edu\",\"phone\":\"(+1) 314-362-7440\",\"institution\":\"Washington University in St. Louis\",\"laboratory\":\"Department of Pathology and Immunology\",\"address\":\"660 S. Euclid Ave CB 8118, St. Louis, MO, 63110\",\"country\":\"USA\",\"corresponding_contributor\":true,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"},\"orcid_id\":\"0000-0001-8647-9662\"},{\"name\":\"Helmut,G,Rennke\",\"email\":\"hrenke@bwh.harvard.edu\",\"phone\":\"(+1) 617-732-7510\",\"institution\":\"Brigham and Women\u0027s Hospital\",\"laboratory\":\"Department of Pathology, Brigham and Women’s Hospital\",\"address\":\"75 Francis Street, Amory 3, Boston, MA  02115\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"},\"orcid_id\":\"0000-0001-6442-3465\"},{\"name\":\"Paul,A,Welling\",\"email\":\"Pwelling@som.umaryland.edu\",\"phone\":\"(+1) 410-706-3851\",\"institution\":\"University of Maryland School of Medicine\",\"laboratory\":\"Department of Medicine, Division of Nephrology\",\"address\":\"655 West Baltimore St., BRB 5-029, Baltimore, MD, 21201\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"},\"orcid_id\":\"0000-0002-0719-8705\"},{\"name\":\"Sushrut,S, Waikar\",\"email\":\"swaikar@bwh.harvard.edu\",\"phone\":\"(+1) 617-732-8473\",\"institution\":\"Brigham and Women\u0027s Hospital\",\"laboratory\":\"Department of Medicine, Division of Renal Medicine\",\"address\":\"75 FRANCIS STREET, BOSTON, MA, 02115\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"},\"orcid_id\":\"0000-0003-4004-326X\"},{\"name\":\"Haojia,,Wu\",\"email\":\"haojiawu@wustl.edu\",\"phone\":\"(+1) 314-362-1575\",\"institution\":\"Washington University in St. Louis\",\"laboratory\":\"Department of Medicine, Division of Nephrology\",\"address\":\"600 S. Euclid Ave. CB 8126, St. Louis, MO, 63110\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"},\"orcid_id\":\"0000-0002-7866-2544\"},{\"name\":\"Yuhei,,Kirita\",\"email\":\"ykirita@wustl.edu\",\"phone\":\"(+1) 314-362-1575\",\"institution\":\"Washington University in St. Louis\",\"laboratory\":\"Department of Medicine, Division of Nephrology\",\"address\":\"600 S. Euclid Ave. CB 8126, St. Louis, MO, 63110\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"}},{\"name\":\"Kohei,,Uchimura\",\"email\":\"kohei.uchimura@wustl.edu\",\"phone\":\"(+1) 314-362-1575\",\"institution\":\"Washington University in St. Louis\",\"laboratory\":\"Department of Medicine, Division of Nephrology\",\"address\":\"600 S. Euclid Ave. CB 8126, St. Louis, MO, 63110\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"experimental scientist\",\"ontology\":\"EFO:0009741\",\"ontology_label\":\"experimental scientist\"}},{\"name\":\"Parisa,,Nejad\",\"email\":\"pnejad@ucsc.edu\",\"institution\":\"University of California, Santa Cruz\",\"laboratory\":\"Human Cell Atlas Data Coordination Platform\",\"country\":\"USA\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"Human Cell Atlas wrangler\",\"ontology\":\"EFO:0009737\",\"ontology_label\":\"data curator\"}}],\"describedBy\":\"https://schema.humancellatlas.org/type/project/17.1.1/project\",\"funders\":[{\"grant_title\":\"Humanized models of mouse diabetic nephropathy\",\"grant_id\":\"DK076169\",\"organization\":\"NIH-NIDDK\"},{\"grant_title\":\"Pilot and Feasability Program\",\"grant_id\":\"DK115255\",\"organization\":\"NIH-NIDDK\"},{\"grant_id\":\"173970\",\"organization\":\"Chan Zuckerburg Initiative\"},{\"grant_title\":\"Novel Human Biomarkers of Kidney Fibrosis\",\"grant_id\":\"DK104308\",\"organization\":\"NIH-NIDDK\"},{\"grant_title\":\"Molecular Mechanism of ROMK Channel Function\",\"grant_id\":\"DK054231\",\"organization\":\"NIH-NIDDK\"},{\"grant_title\":\"Potassium in Hypertension\",\"grant_id\":\"17CVD05\",\"organization\":\"Leducq Foundation\"}],\"geo_series_accessions\":[\"GSE131882\"],\"project_core\":{\"project_short_name\":\"Diabetic Nephropathy snRNA-seq\",\"project_title\":\"The Single Cell Transcriptomic Landscape of Early Human Diabetic Nephropathy\",\"project_description\":\"Diabetic nephropathy is characterized by damage to both the glomerulus and tubulointerstitium, but relatively little is known about accompanying cell-specific changes in gene expression. We performed unbiased single nucleus RNA sequencing (snRNAseq) on cryopreserved human diabetic kidney samples to generate 23,980 single nucleus transcriptomes from three control and three early diabetic nephropathy samples. All major cell types of the kidney were represented in the final dataset. Side by side comparison demonstrated cell-type-specific changes in gene expression that are important for ion transport, angiogenesis, and immune cell activation. In particular, we show that the diabetic thick ascending limb, late distal convoluted tubule, and principal cells all adopt a gene expression signature consistent with increased potassium secretion, including alterations in Na-K+-ATPase, WNK1, mineralocorticoid receptor and NEDD4L expression, as well as decreased paracellular calcium and magnesium reabsorption. We also identify strong angiogenic signatures in glomerular cell types, proximal convoluted tubule, distal convoluted tubule and principal cells. Taken together, these results suggest that increased potassium secretion and angiogenic signaling represent early kidney responses in human diabetic nephropathy.\"},\"publications\":[{\"authors\":[\"Humphreys BD\",\"Wilson PC\",\"Rennke HG\",\"Welling PA\",\"Waikar SS\",\"Wu H\",\"Kirita Y\",\"Uchimura K\"],\"title\":\"The Single Cell Transcriptomic Landscape of Early Human Diabetic Nephropathy\",\"doi\":\"10.1101/645424\",\"url\":\"https://www.pnas.org/content/116/39/19619\",\"official_hca_publication\":false}],\"schema_type\":\"project\",\"estimated_cell_count\":30000,\"hca_bionetworks\":[{\"name\":\"Kidney\",\"hca_tissue_atlas\":\"Kidney\",\"hca_tissue_atlas_version\":\"v1.0\",\"atlas_project\":false,\"schema_version\":\"1.0.1\"}],\"provenance\":{\"document_id\":\"577c946d-6de5-4b55-a854-cd3fde40bff2\",\"submission_date\":\"2019-09-04T18:59:09.150Z\",\"update_date\":\"2023-08-21T14:00:28.207Z\",\"schema_major_version\":17,\"schema_minor_version\":1}}"
}, {
  "project_id": "bc5512cc-9544-4aa4-8b75-8af445ee2257",
  "content": "{\"describedBy\":\"https://schema.humancellatlas.org/type/project/17.1.1/project\",\"schema_type\":\"project\",\"project_core\":{\"project_short_name\":\"RegenerativeLineagesLungMetastasis\",\"project_title\":\"Regenerative lineages and immune-mediated pruning in lung cancer metastasis\",\"project_description\":\"Developmental processes underlying normal tissue regeneration have been implicated in cancer, but the degree of their enactment during tumor progression and under the selective pressures of immune surveillance, remain unknown. Here we show that human primary lung adenocarcinomas are characterized by the emergence of regenerative cell types, typically seen in response to lung injury, and by striking infidelity among transcription factors specifying most alveolar and bronchial epithelial lineages. In contrast, metastases are enriched for key endoderm and lung-specifying transcription factors, SOX2 and SOX9, and recapitulate more primitive transcriptional programs spanning stem-like to regenerative pulmonary epithelial progenitor states. This developmental continuum mirrors the progressive stages of spontaneous outbreak from metastatic dormancy in a mouse model and exhibits SOX9-dependent resistance to natural killer cells. Loss of developmental stage-specific constraint in macrometastases triggered by natural killer cell depletion suggests a dynamic interplay between developmental plasticity and immune-mediated pruning during metastasis.\"},\"contributors\":[{\"name\":\"Ashley M,,Laughney\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Cancer Biology and Genetics Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0001-9435-953X\"},{\"name\":\"Jing,,Hu\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Cancer Biology and Genetics Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Nathaniel R,,Campbell\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Cancer Biology and Genetics Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0002-0355-0708\"},{\"name\":\"Samuel F,,Bakhoum\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Human Oncology and Pathogenesis Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Manu,,Setty\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Vincent-Philippe,,Lavallée\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Yubin,,Xie\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0003-2542-2544\"},{\"name\":\"Ignas,,Masilionis\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0001-7937-9014\"},{\"name\":\"Ambrose J,,Carr\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Sanjay,,Kottapalli\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Viola,,Allaj\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Molecular Pharmacology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Marissa,,Mattar\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Molecular Pharmacology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Natasha,,Rekhtman\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Department of Pathology, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Joao B,,Xavier\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"Linas,,Mazutis\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\"},{\"name\":\"John T,,Poirier\",\"institution\":\"New York University Langone Health\",\"laboratory\":\"Perlmutter Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0001-9795-5644\"},{\"name\":\"Charles M,,Rudin\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Molecular Pharmacology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0001-5204-3465\"},{\"name\":\"Dana,,Pe\u0027er\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Computational and Systems Biology Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0002-9259-8817\",\"corresponding_contributor\":true,\"email\":\"peerd@mskcc.org\"},{\"name\":\"Joan,,Massagué\",\"institution\":\"Sloan Kettering Institute\",\"laboratory\":\"Cancer Biology and Genetics Program, Memorial Sloan Kettering Cancer Center\",\"address\":\"New York, NY\",\"country\":\"USA\",\"orcid_id\":\"0000-0001-9324-8408\",\"corresponding_contributor\":true,\"email\":\"j-massague@ski.mskcc.org\"},{\"name\":\"Ida,,Zucchi\",\"institution\":\"EMBL-EBI\",\"country\":\"UK\",\"email\":\"idazucchi@ebi.ac.uk\",\"project_role\":{\"text\":\"data curator\",\"ontology\":\"EFO:0009737\",\"ontology_label\":\"data curator\"}}],\"publications\":[{\"authors\":[\"Laughney AM\",\"Hu J\",\"Campbell NR\",\"Bakhoum SF\",\"Setty M\",\"Lavallée VP\",\"Xie Y\",\"Masilionis I\",\"Carr AJ\",\"Kottapalli S\",\"Allaj V\",\"Mattar M\",\"Rekhtman N\",\"Xavier JB\",\"Mazutis L\",\"Poirier JT\",\"Rudin CM\",\"Pe\u0027er D\",\"Massagué J.\"],\"title\":\"Regenerative lineages and immune-mediated pruning in lung cancer metastasis.\",\"doi\":\"10.1038/s41591-019-0750-6\",\"pmid\":32042191,\"url\":\"https://doi.org/10.1038/s41591-019-0750-6\",\"official_hca_publication\":false}],\"hca_bionetworks\":[{\"name\":\"Lung\",\"hca_tissue_atlas\":\"Lung\",\"atlas_project\":false}],\"insdc_project_accessions\":[\"SRP173552\",\"SRP173553\"],\"geo_series_accessions\":[\"GSE123902\",\"GSE123903\"],\"insdc_study_accessions\":[\"PRJNA510251\",\"PRJNA510250\"],\"funders\":[{\"grant_id\":\"DP5 OD026395\",\"organization\":\"NIH HHS\"},{\"grant_id\":\"P01 CA129243\",\"organization\":\"NCI NIH HHS\"},{\"grant_id\":\"R01 CA164729\",\"organization\":\"NCI NIH HHS\"},{\"grant_id\":\"U54 CA209975\",\"organization\":\"NCI NIH HHS\"},{\"grant_id\":\"R01 CA229215\",\"organization\":\"NCI NIH HHS\"},{\"grant_id\":\"DP1 HD084071\",\"organization\":\"NICHD NIH HHS\"},{\"grant_id\":\"F30 CA220954\",\"organization\":\"NCI NIH HHS\"},{\"grant_id\":\"P30 CA008748\",\"organization\":\"NCI NIH HHS\"},{\"grant_id\":\"U2C CA233284\",\"organization\":\"NCI NIH HHS\"}],\"estimated_cell_count\":49253,\"provenance\":{\"document_id\":\"bc5512cc-9544-4aa4-8b75-8af445ee2257\",\"submission_date\":\"2021-06-11T13:58:49.740Z\",\"update_date\":\"2023-06-06T09:21:41.571Z\",\"schema_major_version\":17,\"schema_minor_version\":1}}"
}, {
  "project_id": "92892ab2-1334-4b1c-9761-14f5a73548ea",
  "content": "{\"describedBy\":\"https://schema.humancellatlas.org/type/project/17.1.1/project\",\"schema_type\":\"project\",\"project_core\":{\"project_short_name\":\"BALDiscriminatingCOVID-19\",\"project_title\":\"Discriminating mild from critical COVID-19 by innate and adaptive immune single-cell profiling of bronchoalveolar lavages\",\"project_description\":\"How the innate and adaptive host immune system miscommunicate to worsen COVID-19 immunopathology has not been fully elucidated. Here, we perform single-cell deep-immune profiling of bronchoalveolar lavage (BAL) samples from 5 patients with mild and 26 with critical COVID-19 in comparison to BALs from non-COVID-19 pneumonia and normal lung. We use pseudotime inference to build T-cell and monocyte-to-macrophage trajectories and model gene expression changes along them. In mild COVID-19, CD8+ resident-memory (TRM) and CD4+ T-helper-17 (TH17) cells undergo active (presumably antigen-driven) expansion towards the end of the trajectory, and are characterized by good effector functions, while in critical COVID-19 they remain more naïve. Vice versa, CD4+ T-cells with T-helper-1 characteristics (TH1-like) and CD8+ T-cells expressing exhaustion markers (TEX-like) are enriched halfway their trajectories in mild COVID-19, where they also exhibit good effector functions, while in critical COVID-19 they show evidence of inflammation-associated stress at the end of their trajectories. Monocyte-to-macrophage trajectories show that chronic hyperinflammatory monocytes are enriched in critical COVID-19, while alveolar macrophages, otherwise characterized by anti-inflammatory and antigen-presenting characteristics, are depleted. In critical COVID-19, monocytes contribute to an ATP-purinergic signaling-inflammasome footprint that could enable COVID-19 associated fibrosis and worsen disease-severity. Finally, viral RNA-tracking reveals infected lung epithelial cells, and a significant proportion of neutrophils and macrophages that are involved in viral clearance.\"},\"contributors\":[{\"name\":\"Els,,Wauters\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Respiratory Diseases and Thoracic Surgery (BREATHE), Department of Chronic Diseases and Metabolism\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Pierre,,Van Mol\",\"institution\":\"University Hospitals Leuven\",\"laboratory\":\"Department of Pneumology\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0003-1314-9413\"},{\"name\":\"Abhishek Dinkarnath,,Garg\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory for Cell Stress \u0026 Immunity (CSI), Department of Cellular and Molecular Medicine (CMM)\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0002-9976-9922\"},{\"name\":\"Sander,,Jansen\",\"institution\":\"Rega Institute, KU Leuven\",\"laboratory\":\"Laboratory of Virology and Chemotherapy, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0002-9344-131X\"},{\"name\":\"Yannick,,Van Herck\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Experimental Oncology, Department of Oncology\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Lore,,Vanderbeke\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Clinical Bacteriology and Mycology, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Ayse,,Bassez\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Translational Genetics, Department of Human Genetics\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Bram,,Boeckx\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Translational Genetics, Department of Human Genetics\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Bert,,Malengier-Devlies\",\"institution\":\"Rega Institute, KU Leuven\",\"laboratory\":\"Laboratory of Immunobiology, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0003-3527-1145\"},{\"name\":\"Anna,,Timmerman\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Translational Genetics, Department of Human Genetics\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Thomas,,Van Brussel\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Translational Genetics, Department of Human Genetics\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Tina,,Van Buyten\",\"institution\":\"Rega Institute, KU Leuven\",\"laboratory\":\"Laboratory of Virology and Chemotherapy, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Rogier,,Schepers\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Translational Genetics, Department of Human Genetics\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Elisabeth,,Heylen\",\"institution\":\"Rega Institute, KU Leuven\",\"laboratory\":\"Laboratory of Virology and Chemotherapy, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0003-4359-7225\"},{\"name\":\"Dieter,,Dauwe\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Intensive Care Medicine, Department of Cellular and Molecular Medicine\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0002-9771-2543\"},{\"name\":\"Christophe,,Dooms\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Respiratory Diseases and Thoracic Surgery (BREATHE), Department of Chronic Diseases and Metabolism\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Jan,,Gunst\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Intensive Care Medicine, Department of Cellular and Molecular Medicine\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Greet,,Hermans\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Intensive Care Medicine, Department of Cellular and Molecular Medicine\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Philippe,,Meersseman\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory for Clinical Infectious and Inflammatory Disorders, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Dries,,Testelmans\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Respiratory Diseases and Thoracic Surgery (BREATHE), Department of Chronic Diseases and Metabolism\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Jonas,,Yserbyt\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Respiratory Diseases and Thoracic Surgery (BREATHE), Department of Chronic Diseases and Metabolism\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Sabine,,Tejpar\",\"institution\":\"KU Leuven\",\"laboratory\":\"Molecular Digestive Oncology, Department of Oncology\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Walter,,De Wever\",\"institution\":\"KU Leuven\",\"laboratory\":\"Department of Imaging \u0026 Pathology\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Patrick,,Matthys\",\"institution\":\"Rega Institute, KU Leuven\",\"laboratory\":\"Laboratory of Immunobiology, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0002-9685-6836\"},{\"name\":\"Johan,,Neyts\",\"institution\":\"Rega Institute, KU Leuven\",\"laboratory\":\"Laboratory of Virology and Chemotherapy, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"orcid_id\":\"0000-0002-0033-7514\"},{\"name\":\"Joost,,Wauters\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory for Clinical Infectious and Inflammatory Disorders, Department of Microbiology, Immunology and Transplantation\",\"address\":\"Leuven\",\"country\":\"Belgium\"},{\"name\":\"Junbin,,Qian\",\"email\":\"dr_qian@zju.edu.cn\",\"institution\":\"Women\u0027s Hospital, Zhejiang University\",\"laboratory\":\"Department of Gynecologic Oncology, School of Medicine\",\"address\":\"Hangzhou, Zhejiang, 310006\",\"country\":\"China\",\"corresponding_contributor\":true},{\"name\":\"Diether,,Lambrechts\",\"email\":\"Diether.Lambrechts@kuleuven.vib.be\",\"institution\":\"KU Leuven\",\"laboratory\":\"Laboratory of Translational Genetics, Department of Human Genetics\",\"address\":\"Leuven\",\"country\":\"Belgium\",\"corresponding_contributor\":true,\"orcid_id\":\"0000-0002-3429-302X\"},{\"name\":\"Ida,,Zucchi\",\"email\":\"idazucchi@ebi.ac.uk\",\"institution\":\"EMBL-EBI\",\"country\":\"UK\",\"project_role\":{\"text\":\"data curator\",\"ontology\":\"EFO:0009737\",\"ontology_label\":\"data curator\"}}],\"supplementary_links\":[\"https://lambrechtslab.sites.vib.be/en/immune-atlas\"],\"publications\":[{\"authors\":[\"Wauters E\",\"Van Mol P\",\"Garg AD\",\"Jansen S\",\"Van Herck Y\",\"Vanderbeke L\",\"Bassez A\",\"Boeckx B\",\"Malengier-Devlies B\",\"Timmerman A\",\"Van Brussel T\",\"Van Buyten T\",\"Schepers R\",\"Heylen E\",\"Dauwe D\",\"Dooms C\",\"Gunst J\",\"Hermans G\",\"Meersseman P\",\"Testelmans D\",\"Yserbyt J\",\"Tejpar S\",\"De Wever W\",\"Matthys P\",\"CONTAGIOUS collaborators\",\"Neyts J\",\"Wauters J\",\"Qian J\",\"Lambrechts D.\"],\"title\":\"Discriminating mild from critical COVID-19 by innate and adaptive immune single-cell profiling of bronchoalveolar lavages\",\"doi\":\"10.1038/s41422-020-00455-9\",\"pmid\":33473155,\"url\":\"https://doi.org/10.1038/s41422-020-00455-9\",\"official_hca_publication\":false}],\"hca_bionetworks\":[{\"schema_version\":\"1.0.1\",\"name\":\"Lung\",\"hca_tissue_atlas\":\"Lung\",\"hca_tissue_atlas_version\":\"v1.0\",\"atlas_project\":false}],\"ega_accessions\":[\"EGAS00001004717\"],\"funders\":[{\"grant_id\":\"1S66020N\",\"organization\":\"Fonds Wetenschappelijk Onderzoek\"},{\"grant_id\":\"VR 2016 2312 Doc.1521/4\",\"organization\":\"Vlaams Instituut voor Biotechnologie\"},{\"grant_id\":\"1833317N\",\"organization\":\"Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)\"},{\"grant_id\":\"1833317N\",\"organization\":\"Fonds Wetenschappelijk Onderzoek\"},{\"grant_id\":\"1833317N\",\"organization\":\"Stichting Tegen Kanker\"},{\"grant_id\":\"1S66020N\",\"organization\":\"Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)\"},{\"grant_id\":\"VR 2016 2312 Doc.1521/4\",\"organization\":\"Vlaams Instituut voor Biotechnologie (Flanders Institute for Biotechnology)\"},{\"grant_id\":\"11E9819N\",\"organization\":\"Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)\"},{\"grant_id\":\"11E9819N\",\"organization\":\"Fonds Wetenschappelijk Onderzoek\"},{\"grant_id\":\"1833317N\",\"organization\":\"Stichting Tegen Kanker (Belgian Foundation Against Cancer)\"}],\"estimated_cell_count\":65166,\"provenance\":{\"document_id\":\"92892ab2-1334-4b1c-9761-14f5a73548ea\",\"submission_date\":\"2021-10-12T10:28:45.646Z\",\"update_date\":\"2023-08-16T16:32:24.327Z\",\"schema_major_version\":17,\"schema_minor_version\":1}}"
}, {
  "project_id": "1fac187b-1c3f-41c4-b6b6-6a9a8c0489d1",
  "content": "{\"describedBy\":\"https://schema.humancellatlas.org/type/project/17.0.0/project\",\"schema_type\":\"project\",\"project_core\":{\"project_short_name\":\"HumanCellLandscape\",\"project_title\":\"Construction of A Human Cell Landscape by Single-cell mRNA-seq\",\"project_description\":\"Single-cell RNA sequencing was used to determine the cell type composition of all major human organs and construct a basic scheme for the human cell landscape (HCL).\"},\"contributors\":[{\"name\":\"Xiaoping,,Han\",\"email\":\"xhan@zju.edu.cn\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\",\"corresponding_contributor\":true,\"orcid_id\":\"0000-0002-7557-7133\"},{\"name\":\"Guoji,,Guo\",\"email\":\"ggj@zju.edu.cn\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\",\"corresponding_contributor\":true,\"orcid_id\":\"0000-0002-1716-4621\"},{\"name\":\"Ziming,,Zhou\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Lijiang,,Fei\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Huiyu,,Sun\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Renying,,Wang\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Yao,,Chen\",\"institution\":\"Department of Reproductive Endocrinology, Women\u0027s Hospital, Zhejiang University School of Medicine.\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Haide,,Chen\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Jingjing,,Wang\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Huanna,,Tang\",\"institution\":\"Kidney Disease Center, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Wenhao,,Ge\",\"institution\":\"Department of Hepatobiliary and Pancreatic Surgery, The Second Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Yincong,,Zhou\",\"institution\":\"College of Life Sciences, Zhejiang University\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Fang,,Ye\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Mengmeng,,Jiang\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Junqing,,Wu\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Yanyu,,Xiao\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Xiaoning,,Jia\",\"institution\":\"Center for Neuroscience, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Tingyue,,Zhang\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Xiaojie,,Ma\",\"institution\":\"Life Sciences Institute, Zhejiang University\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Qi,,Zhang\",\"institution\":\"Department of Hepatobiliary and Pancreatic Surgery, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Xueli,,Bai\",\"institution\":\"Department of Hepatobiliary and Pancreatic Surgery, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Shujing,,Lai\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Chengxuan,,Yu\",\"institution\":\"Center for Stem Cell and Regenerative Medicine, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Lijun,,Zhu\",\"institution\":\"Department of Hepatobiliary and Pancreatic Surgery, The Second Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Rui,,Lin\",\"institution\":\"Hangzhou Repugene Technology\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Yuchi,,Gao\",\"institution\":\"Annoroad Gene Technology\",\"address\":\"Beijing\",\"country\":\"China\"},{\"name\":\"Min,,Wang\",\"institution\":\"Veritas Genetics Asia\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Yiqing,,Wu\",\"institution\":\"Department of Reproductive Endocrinology, Women\u0027s Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Jianming,,Zhang\",\"institution\":\"Department of Neurosurgery, The Second Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Renya,,Zhan\",\"institution\":\"Department of Neurosurgery, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Saiyong,,Zhu\",\"institution\":\"Life Sciences Institute, Zhejiang University\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Hailan,,Hu\",\"institution\":\"Center for Neuroscience, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Changchun,,Wang\",\"institution\":\"Institute of Cancer Research and Basic Medical Sciences of Chinese Academy of Sciences, Cancer Hospital of University of Chinese Academy of Sciences, Zhejiang Cancer Hospital\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Ming,,Chen\",\"institution\":\"College of Life Sciences, Zhejiang University\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"He,,Huang\",\"institution\":\"Bone Marrow Transplantation Center, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Tingbo,,Liang\",\"institution\":\"Department of Hepatobiliary and Pancreatic Surgery, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Jianghua,,Chen\",\"institution\":\"Kidney Disease Center, The First Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Weilin,,Wang\",\"institution\":\"Department of Hepatobiliary and Pancreatic Surgery, The Second Affiliated Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Dan,,Zhang\",\"institution\":\"Department of Reproductive Endocrinology, Women\u0027s Hospital, Zhejiang University School of Medicine\",\"address\":\"Hangzhou\",\"country\":\"China\"},{\"name\":\"Ami,,Day\",\"email\":\"ami@ebi.ac.uk\",\"institution\":\"EMBL-EBI\",\"address\":\"Wellcome Genome Campus Cambridge\",\"country\":\"UK\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"Human Cell Atlas Data Wrangler\",\"ontology\":\"EFO:0009737\",\"ontology_label\":\"data curator\"}}],\"supplementary_links\":[\"https://db.cngb.org/search/project/CNP0000325/\"],\"publications\":[{\"authors\":[\"Han X\",\"Zhou Z\",\"Fei L\",\"Sun H\",\"Wang R\",\"Chen Y\",\"Chen H\",\"Wang J\",\"Tang H\",\"Ge W\",\"Zhou Y\",\"Ye F\",\"Jiang M\",\"Wu J\",\"Xiao Y\",\"Jia X\",\"Zhang T\",\"Ma X\",\"Zhang Q\",\"Bai X\",\"Lai S\",\"Yu C\",\"Zhu L\",\"Lin R\",\"Gao Y\",\"Wang M\",\"Wu Y\",\"Zhang J\",\"Zhan R\",\"Zhu S\",\"Hu H\",\"Wang C\",\"Chen M\",\"Huang H\",\"Liang T\",\"Chen J\",\"Wang W\",\"Zhang D\",\"Guo G\"],\"title\":\"Construction of a human cell landscape at single-cell level.\",\"doi\":\"10.1038/s41586-020-2157-4\",\"pmid\":32214235,\"url\":\"https://www.nature.com/articles/s41586-020-2157-4\",\"official_hca_publication\":false}],\"insdc_project_accessions\":[\"SRP214806\"],\"geo_series_accessions\":[\"GSE134355\"],\"insdc_study_accessions\":[\"PRJNA554845\"],\"funders\":[{\"grant_id\":\"91842301.0\",\"organization\":\"National Natural Science Foundation of China\"},{\"grant_id\":\"81770188.0\",\"organization\":\"National Natural Science Foundation of China\"},{\"grant_id\":\"31722027.0\",\"organization\":\"National Natural Science Foundation of China\"},{\"grant_id\":\"31922049.0\",\"organization\":\"National Natural Science Foundation of China\"},{\"grant_id\":\"31701290.0\",\"organization\":\"National Natural Science Foundation of China\"},{\"grant_id\":\"31871473.0\",\"organization\":\"National Natural Science Foundation of China\"},{\"grant_id\":\"2018YFA0107804\",\"organization\":\"National Key Research and Development Program\"},{\"grant_id\":\"2018YFA0107801\",\"organization\":\"National Key Research and Development Program\"},{\"grant_id\":\"2018YFA0800503\",\"organization\":\"National Key Research and Development Program\"},{\"grant_id\":\"2018YFC1005003\",\"organization\":\"National Key Research and Development Program\"},{\"grant_id\":\"R17H080001\",\"organization\":\"Zhejiang Provincial Natural Science Foundation of China\"},{\"grant_id\":\"unspecified\",\"organization\":\"Fundamental Research Funds for the Central Universities\"}],\"estimated_cell_count\":702968,\"provenance\":{\"document_id\":\"1fac187b-1c3f-41c4-b6b6-6a9a8c0489d1\",\"submission_date\":\"2021-01-14T15:30:33.065Z\",\"update_date\":\"2023-04-13T16:32:49.135Z\",\"schema_major_version\":17,\"schema_minor_version\":0}}"
}, {
  "project_id": "c893cb57-5c9f-4f26-9312-21b85be84313",
  "content": "{\"describedBy\":\"https://schema.humancellatlas.org/type/project/17.1.1/project\",\"schema_type\":\"project\",\"project_core\":{\"project_short_name\":\"EpithelialDiversityHealthInflammation\",\"project_title\":\"Colonic single epithelial cell census reveals goblet cell drivers of barrier breakdown in inflammatory bowel disease.\",\"project_description\":\"Three single cell RNA-sequencing (10x, Chromium) experiments were performed on healthy and inflamed human colonic samples, including from patients with Ulcerative Colitis. The authors characterised the composition of the intestinal epithelial cells in these samples.\"},\"contributors\":[{\"name\":\"Kaushal,,Parikh\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Agne,,Antanaviciute\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"David,,Fawkner-Corbett\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Marta,,Jagielowicz\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Anna,,Aulicino\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Christoffer,,Lagerholm\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Wolfson Imaging Centre Oxford\",\"address\":\"Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Simon,,Davis\",\"institution\":\"Target Discovery Institute\",\"laboratory\":\"Nuffield Department of Medicine\",\"address\":\"University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"James,,Kinchen\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Hannah,H,Chen\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Nasullah Khalid,,Alham\",\"institution\":\"John Radcliffe Hospital, University of Oxford\",\"laboratory\":\"Nuffield Department of Surgical Sciences\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Neil,,Ashley\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Errin,,Johnson\",\"institution\":\"Sir William Dunn School of Pathology, University of Oxford\",\"address\":\"University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Philip,,Hublitz\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Leyuan,,Bao\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Joanna,,Lukomska\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Rajinder Singh,,Andev\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Elisabet,,Björklund\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Benedikt,M,Kessler\",\"institution\":\"Target Discovery Institute\",\"laboratory\":\"Nuffield Department of Medicine\",\"address\":\"University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Roman,,Fischer\",\"institution\":\"Target Discovery Institute\",\"laboratory\":\"Nuffield Department of Medicine\",\"address\":\"University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Robert,,Goldin\",\"institution\":\"St Mary\u0027s Hospital, Imperial College\",\"laboratory\":\"Centre for Pathology\",\"address\":\"St Mary\u0027s Hospital, Imperial College, London\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Hashem,,Koohy\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine\",\"laboratory\":\"MRC WIMM Centre For Computational Biology\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":false},{\"name\":\"Alison,,Simmons\",\"email\":\"alison.simmons@imm.ox.ac.uk\",\"institution\":\"MRC Weatherall Institute of Molecular Medicine (WIMM)\",\"laboratory\":\"Medical Research Council (MRC) Human Immunology Unit\",\"address\":\"John Radcliffe Hospital, University of Oxford, Oxford\",\"country\":\"UK\",\"corresponding_contributor\":true},{\"name\":\"Ami,,Day\",\"institution\":\"EMBL-EBI\",\"address\":\"Wellcome Genome Campus, Hinxton\",\"country\":\"UK\",\"corresponding_contributor\":false,\"project_role\":{\"text\":\"Human Cell Atlas Wrangler\",\"ontology\":\"EFO:0009737\",\"ontology_label\":\"data curator\"}}],\"supplementary_links\":[\"https://github.com/agneantanaviciute/colonicepithelium\",\"http://dx.doi.org/10.6019/PXD011655\"],\"publications\":[{\"authors\":[\"Parikh K\",\"Antanaviciute A\",\"Fawkner-Corbett D\",\"Jagielowicz M\",\"Aulicino A\",\"Lagerholm C\",\"Davis S\",\"Kinchen J\",\"Chen HH\",\"Alham NK\",\"Ashley N\",\"Johnson E\",\"Hublitz P\",\"Bao L\",\"Lukomska J\",\"Andev RS\",\"Björklund E\",\"Kessler BM\",\"Fischer R\",\"Goldin R\",\"Koohy H\",\"Simmons A\"],\"title\":\"Colonic epithelial cell diversity in health and inflammatory bowel disease.\",\"doi\":\"10.1038/s41586-019-0992-y\",\"pmid\":30814735,\"url\":\"https://www.nature.com/articles/s41586-019-0992-y\",\"official_hca_publication\":false}],\"insdc_project_accessions\":[\"SRP151260\"],\"geo_series_accessions\":[\"GSE116222\"],\"insdc_study_accessions\":[\"PRJNA477814\"],\"funders\":[{\"grant_id\":\"MR/M00919X/1\",\"organization\":\"Medical Research Council\"},{\"grant_id\":\"MR/S036377/1\",\"organization\":\"Medical Research Council\"},{\"grant_id\":\"MC_UU_00008/7\",\"organization\":\"Medical Research Council\"},{\"grant_id\":\"MC_UU_12010/7\",\"organization\":\"Medical Research Council\"},{\"grant_id\":\"NIHR-RP-R3-12-026\",\"organization\":\"Department of Health\"}],\"estimated_cell_count\":11175,\"hca_bionetworks\":[{\"name\":\"Gut\",\"hca_tissue_atlas\":\"Gut\",\"hca_tissue_atlas_version\":\"v1.0\",\"atlas_project\":false,\"schema_version\":\"1.0.1\"}],\"provenance\":{\"document_id\":\"c893cb57-5c9f-4f26-9312-21b85be84313\",\"submission_date\":\"2021-02-03T11:13:48.174Z\",\"update_date\":\"2023-07-03T22:02:34.763Z\",\"schema_major_version\":17,\"schema_minor_version\":1}}"
}]
$ cat tissue_atlas.json | jq -r '.[].content' | jq '.provenance.document_id,.describedBy,.hca_bionetworks'
"577c946d-6de5-4b55-a854-cd3fde40bff2"
"https://schema.humancellatlas.org/type/project/17.1.1/project"
[
  {
    "name": "Kidney",
    "hca_tissue_atlas": "Kidney",
    "hca_tissue_atlas_version": "v1.0",
    "atlas_project": false,
    "schema_version": "1.0.1"
  }
]
"bc5512cc-9544-4aa4-8b75-8af445ee2257"
"https://schema.humancellatlas.org/type/project/17.1.1/project"
[
  {
    "name": "Lung",
    "hca_tissue_atlas": "Lung",
    "atlas_project": false
  }
]
"92892ab2-1334-4b1c-9761-14f5a73548ea"
"https://schema.humancellatlas.org/type/project/17.1.1/project"
[
  {
    "schema_version": "1.0.1",
    "name": "Lung",
    "hca_tissue_atlas": "Lung",
    "hca_tissue_atlas_version": "v1.0",
    "atlas_project": false
  }
]
"1fac187b-1c3f-41c4-b6b6-6a9a8c0489d1"
"https://schema.humancellatlas.org/type/project/17.0.0/project"
null
"c893cb57-5c9f-4f26-9312-21b85be84313"
"https://schema.humancellatlas.org/type/project/17.1.1/project"
[
  {
    "name": "Gut",
    "hca_tissue_atlas": "Gut",
    "hca_tissue_atlas_version": "v1.0",
    "atlas_project": false,
    "schema_version": "1.0.1"
  }
]
hannes-ucsc commented 1 year ago

As of dcp32, 1fac187b-1c3f-41c4-b6b6-6a9a8c0489d1 doesn't have the required schema version or fields. The only snapshot for it was introduced in dcp29 so maybe there's an update in the works. Either way, given how sparse the data is, it seems a little premature for us to add support for this. The result would look a little sad. I don't see any technical issues with the spec or its implementation except with

atlas - an array of the concatenation of project.hca_bionetworks[x].hca_tissue_atlas + " " + project.hca_bionetworks[x].hca_tissue_atlas_version for each of the project's hca_bionetworks array entries.

When returning JSON, it is an anti-pattern to concatenate properties with a separator. It would be more flexible to return nested objects and there is precedent for this in Azul, for example with donor and organism ages. So instead of returning and array of strings, we will return an array of {"atlas": …, "version": …} objects. The UI can then render these as it pleases.

But again, given our other priorities, it seems premature to delve into this right now.

dsotirho-ucsc commented 1 year ago

@hannes-ucsc: "Assignee to work with wranglers to get more projects augmented with Bionetwork information. "

NoopDog commented 1 year ago

So instead of returning and array of strings, we will return an array of {"atlas": …, "version": …} objects. The UI can then render these as it pleases.

Awesome, Thanks, D

bvizzier-ucsc commented 1 year ago

I've asked the wranglers if we can start the process of getting the bionetworks and atlases sections of the schema.

bvizzier-ucsc commented 1 year ago

30* more datasets.

(* Note: 6874b7eb-3445-47ec-8773-75141430e169 does not appear to be in the current hand-off.)

They will be focusing on datasets identified by Dave and from discussions with John R.

bvizzier-ucsc commented 1 year ago

dcp34 is scheduled to have more datasets with the atlas information populated. dcp34 is scheduled to close for submissions on Dec 5, 2023.

bvizzier-ucsc commented 11 months ago

Still waiting on release of dcp34.

bvizzier-ucsc commented 11 months ago

dcp34 has been released and it is supposed to have more snapshots with the bionetworks and atlases associated fields completed in the schema. The did not specify how many of them have it.

Edit: Moving to Triage for review and next steps.

hannes-ucsc commented 10 months ago

Spike to determine how many projects have atlas information populated in dcp34.

hannes-ucsc commented 10 months ago

1) Someone else can do the spike.

2) Is it true that CC has implemented an out-of-band process to populate their atlas UI? If so, how important is it that we implement this now? If they continue to use that process, changes in Azul would remain unused.

NoopDog commented 10 months ago

Hi @bvizzier-ucsc and @hannes-ucsc, I feel this feature is still required and a priority. I believe this is also the view of the HCA office. Once implemented, the HCA Data Explorer will use the Azul implementation to enable users to find atlases, their source datasets, related analysis portals, etc., in the Data Explorer.

dsotirho-ucsc commented 10 months ago

@hannes-ucsc: "That makes sense. Assignee to proceed with spike and request from System Administrator to reduce production permissions (editor to viewer)."

achave11-ucsc commented 10 months ago

The following is an exhaustive list of projects (46) in the dcp34 catalog with the hca_bionetworks field populated, where the "atlas information" is defined:

datarepo-a2f2ced7.hca_prod_08fb10df32e5456c9882e33fcd49077a__20231212_dcp2_20231213_dcp34.project, [{"name":"Lung","atlas_project":false}]
datarepo-a066b1d5.hca_prod_01aacb6840764fd99eb9aba0f48c1b5a__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-60acbcdf.hca_prod_111d272bc25a49ac9b25e062b70d66e0__20230530_dcp2_20230530_dcp28.project, [{"name":"Lung","atlas_project":false,"hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0"}]
datarepo-6fec2801.hca_prod_1538d572bcb7426b8d2c84f3a7f87bb0__20220630_dcp2_20231213_dcp34.project, [{"name":"Immune","hca_tissue_atlas":"Blood","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"},{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-cac974a7.hca_prod_1c4cbdd433e34dedab435958de817123__20230614_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-bb040c00.hca_prod_1eba4d0b2d154ba7bb3cd4654dd94519__20230815_dcp2_20230815_dcp30.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-1d8f9fa4.hca_prod_2184e63d82d84ab2839ee93f8395f568__20231212_dcp2_20231213_dcp34.project, []
datarepo-5f2ddddc.hca_prod_272b760266cd4b02a86b2b7c9c51a9ea__20230526_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-db922a93.hca_prod_29b5416534ee4da5b257b4c1f7343656__20230530_dcp2_20230530_dcp28.project, [{"name":"Kidney"}]
datarepo-780b7376.hca_prod_2caedc30c8164b99a237b9f3b458c8e5__20230614_dcp2_20230616_dcp29.project, [{"name":"Kidney","atlas_project":false}]
datarepo-3dbccc52.hca_prod_2f67614380c24bc6b7b42613fe0fadf0__20220111_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-720eb4d9.hca_prod_31887183a72c43089eacc6140313f39c__20220111_dcp2_20231002_dcp32.project, [{"name":"Gut","hca_tissue_atlas":"Gut","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-8c4f04c0.hca_prod_326b36bd0975475f983b56ddb8f73a4d__20230905_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false,"hca_tissue_atlas_version":"v1.0","schema_version":"1.0.1"}]
datarepo-078dbc55.hca_prod_34c9a62ca6104e31b3438fb7be676f8c__20221101_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-7221e50b.hca_prod_453d7ee2319f496c986299d397870b63__20230905_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-6c9af5df.hca_prod_457d0bfe79e443f1be5d83bf080d809e__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0","atlas_project":false}]
datarepo-129b6bcc.hca_prod_4a95101c9ffc4f30a809f04518a23803__20220113_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-766c30b1.hca_prod_4ef86852aca04a9185229968e0e54dbe__20230313_dcp2_20231102_dcp33.project, [{"name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-0287a0ba.hca_prod_577c946d6de54b55a854cd3fde40bff2__20220117_dcp2_20231002_dcp32.project, [{"name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-37f63790.hca_prod_58028aa80ed249cab60f15e2ed5989d5__20220117_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-4d2eebce.hca_prod_5a54c6170eed486e8c1a8a8041fc1729__20230616_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-84d96baf.hca_prod_65cbfea55c544255a1d014549a86a5c1__20230616_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-e8198e31.hca_prod_6874b7eb344547ec877375141430e169__20231213_dcp2_20231213_dcp34.project, [{"name":"Skin","atlas_project":false}]
datarepo-7be1db07.hca_prod_6936da41369246bbbca1cd0f507991e9__20230905_dcp2_20231003_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-13c1e76b.hca_prod_8185730f411340d39cc3929271784c2b__20220117_dcp2_20231213_dcp34.project, [{"name":"Eye","hca_tissue_atlas":"Retina","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-4450b12b.hca_prod_92892ab213344b1c976114f5a73548ea__20230616_dcp2_20230905_dcp31.project, [{"schema_version":"1.0.1","name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-6f3c6cc2.hca_prod_92afaa56d501481ea027dddd72212ba8__20230526_dcp2_20230815_dcp30.project, [{"name":"Lung","atlas_project":false}]
datarepo-96344c2b.hca_prod_957261f72bd64358a6ed24ee080d5cfc__20220330_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-bc7bde81.hca_prod_95f07e6e6a734e1ba880c83996b3aa5c__20220118_dcp2_20231102_dcp33.project, [{"name":"Immune","hca_tissue_atlas":"Blood","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-7f53a7f2.hca_prod_abe1a013af7a45ed8c26f3793c24a1f4__20220118_dcp2_20231213_dcp34.project, [{"name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-32c91a7f.hca_prod_b91c623b19454727b1670a93027b0d3f__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","atlas_project":false}]
datarepo-41efd06c.hca_prod_bc5512cc95444aa48b758af445ee2257__20230614_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-2901e79b.hca_prod_c05184453b3b49c6b8fcc41daa4eacba__20220213_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-5b1e55df.hca_prod_c16a754f5da346ed8c1e6426af2ef625__20220519_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-94d17e05.hca_prod_c1a9a93dd9de4e659619a9cec1052eaa__20220118_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-d12842d9.hca_prod_c4077b3c5c984d26a614246d12c2e5d7__20220118_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-abaa9315.hca_prod_c844538b88544a95bd01aacbaf86d97f__20230427_dcp2_20231213_dcp34.project, [{"schema_version":"1.0.1","name":"Gut","hca_tissue_atlas":"Gut","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-47aed999.hca_prod_c893cb575c9f4f26931221b85be84313__20220118_dcp2_20230815_dcp30.project, [{"name":"Gut","hca_tissue_atlas":"Gut","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-c47b01c5.hca_prod_d8ae869c39c24cddb3fc2d0d8f60e7b8__20230313_dcp2_20230815_dcp30.project, [{"schema_version":"1.0.1","name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-0bc51bfc.hca_prod_daf9d9827ce643f6ab51272577290606__20220119_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-2abbf49d.hca_prod_e456c042f6b64ceca3381a8ef80bd779__20230905_dcp2_20230905_dcp31.project, [{"name":"Lung"}]
datarepo-99284b34.hca_prod_e526d91dcf3a44cb80c5fd7676b55a1d__20220119_dcp2_20231213_dcp34.project, [{"schema_version":"1.0.1","name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-72424b3d.hca_prod_e5fe827437694d7daa356d33c226ab43__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0","atlas_project":false}]
datarepo-145a904d.hca_prod_ef1e3497515e4bbe8d4c10161854b699__20220118_dcp2_20231213_dcp34.project, [{"schema_version":"1.0.1","name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-9176f1e6.hca_prod_fae72d894ac44aab9b93574775e168d4__20230530_dcp2_20230530_dcp28.project, [{"name":"Kidney","atlas_project":false}]
datarepo-6f4f0e4f.hca_prod_f7b464770f2a4bffa9b7719e000499a3__20230815_dcp2_20230815_dcp30.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0","atlas_project":false}]

Note that project datarepo-1d8f9fa4.hca_prod_2184e63d82d84ab2839ee93f8395f568__20231212_dcp2_20231213_dcp34 defines this field as an empty list.

However, if the field of interest is actually hca_tissue_atlas then the project count (37) is a subset of the previous list, and those are:

datarepo-a066b1d5.hca_prod_01aacb6840764fd99eb9aba0f48c1b5a__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-60acbcdf.hca_prod_111d272bc25a49ac9b25e062b70d66e0__20230530_dcp2_20230530_dcp28.project, [{"name":"Lung","atlas_project":false,"hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0"}]
datarepo-6fec2801.hca_prod_1538d572bcb7426b8d2c84f3a7f87bb0__20220630_dcp2_20231213_dcp34.project, [{"name":"Immune","hca_tissue_atlas":"Blood","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"},{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-cac974a7.hca_prod_1c4cbdd433e34dedab435958de817123__20230614_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-bb040c00.hca_prod_1eba4d0b2d154ba7bb3cd4654dd94519__20230815_dcp2_20230815_dcp30.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-5f2ddddc.hca_prod_272b760266cd4b02a86b2b7c9c51a9ea__20230526_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-3dbccc52.hca_prod_2f67614380c24bc6b7b42613fe0fadf0__20220111_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-720eb4d9.hca_prod_31887183a72c43089eacc6140313f39c__20220111_dcp2_20231002_dcp32.project, [{"name":"Gut","hca_tissue_atlas":"Gut","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-8c4f04c0.hca_prod_326b36bd0975475f983b56ddb8f73a4d__20230905_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false,"hca_tissue_atlas_version":"v1.0","schema_version":"1.0.1"}]
datarepo-078dbc55.hca_prod_34c9a62ca6104e31b3438fb7be676f8c__20221101_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-7221e50b.hca_prod_453d7ee2319f496c986299d397870b63__20230905_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-6c9af5df.hca_prod_457d0bfe79e443f1be5d83bf080d809e__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0","atlas_project":false}]
datarepo-129b6bcc.hca_prod_4a95101c9ffc4f30a809f04518a23803__20220113_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-766c30b1.hca_prod_4ef86852aca04a9185229968e0e54dbe__20230313_dcp2_20231102_dcp33.project, [{"name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-0287a0ba.hca_prod_577c946d6de54b55a854cd3fde40bff2__20220117_dcp2_20231002_dcp32.project, [{"name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-37f63790.hca_prod_58028aa80ed249cab60f15e2ed5989d5__20220117_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-4d2eebce.hca_prod_5a54c6170eed486e8c1a8a8041fc1729__20230616_dcp2_20231002_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-84d96baf.hca_prod_65cbfea55c544255a1d014549a86a5c1__20230616_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-7be1db07.hca_prod_6936da41369246bbbca1cd0f507991e9__20230905_dcp2_20231003_dcp32.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-13c1e76b.hca_prod_8185730f411340d39cc3929271784c2b__20220117_dcp2_20231213_dcp34.project, [{"name":"Eye","hca_tissue_atlas":"Retina","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-4450b12b.hca_prod_92892ab213344b1c976114f5a73548ea__20230616_dcp2_20230905_dcp31.project, [{"schema_version":"1.0.1","name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-96344c2b.hca_prod_957261f72bd64358a6ed24ee080d5cfc__20220330_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-bc7bde81.hca_prod_95f07e6e6a734e1ba880c83996b3aa5c__20220118_dcp2_20231102_dcp33.project, [{"name":"Immune","hca_tissue_atlas":"Blood","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-7f53a7f2.hca_prod_abe1a013af7a45ed8c26f3793c24a1f4__20220118_dcp2_20231213_dcp34.project, [{"name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-41efd06c.hca_prod_bc5512cc95444aa48b758af445ee2257__20230614_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","atlas_project":false}]
datarepo-2901e79b.hca_prod_c05184453b3b49c6b8fcc41daa4eacba__20220213_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-5b1e55df.hca_prod_c16a754f5da346ed8c1e6426af2ef625__20220519_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-94d17e05.hca_prod_c1a9a93dd9de4e659619a9cec1052eaa__20220118_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-d12842d9.hca_prod_c4077b3c5c984d26a614246d12c2e5d7__20220118_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-abaa9315.hca_prod_c844538b88544a95bd01aacbaf86d97f__20230427_dcp2_20231213_dcp34.project, [{"schema_version":"1.0.1","name":"Gut","hca_tissue_atlas":"Gut","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-47aed999.hca_prod_c893cb575c9f4f26931221b85be84313__20220118_dcp2_20230815_dcp30.project, [{"name":"Gut","hca_tissue_atlas":"Gut","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-c47b01c5.hca_prod_d8ae869c39c24cddb3fc2d0d8f60e7b8__20230313_dcp2_20230815_dcp30.project, [{"schema_version":"1.0.1","name":"Kidney","hca_tissue_atlas":"Kidney","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-0bc51bfc.hca_prod_daf9d9827ce643f6ab51272577290606__20220119_dcp2_20231213_dcp34.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false,"schema_version":"1.0.1"}]
datarepo-99284b34.hca_prod_e526d91dcf3a44cb80c5fd7676b55a1d__20220119_dcp2_20231213_dcp34.project, [{"schema_version":"1.0.1","name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-72424b3d.hca_prod_e5fe827437694d7daa356d33c226ab43__20230616_dcp2_20230616_dcp29.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0","atlas_project":false}]
datarepo-145a904d.hca_prod_ef1e3497515e4bbe8d4c10161854b699__20220118_dcp2_20231213_dcp34.project, [{"schema_version":"1.0.1","name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v1.0","atlas_project":false}]
datarepo-6f4f0e4f.hca_prod_f7b464770f2a4bffa9b7719e000499a3__20230815_dcp2_20230815_dcp30.project, [{"name":"Lung","hca_tissue_atlas":"Lung","hca_tissue_atlas_version":"v2.0","atlas_project":false}]
hannes-ucsc commented 10 months ago

I compiled atlas.json from the above.

hannes-ucsc commented 10 months ago

The .hits[].projects[0] slice from /index/projects endpoint response would then look like project.json.

The property names deviate from CC's specification but that should be acceptable, the structure is the same as what was ~specified~ agreed.

Here's an excerpt from that file:

    {
        "projectId": "1538d572-bcb7-426b-8d2c-84f3a7f87bb0",
        "isTissueAtlasProject": false,
        "tissueAtlas": [
            {
                "atlas": "Blood",
                "version": "v1.0"
            },
            {
                "atlas": "Lung",
                "version": "v1.0"
            }
        ],
        "bionetworkName": [
            "Immune",
            "Lung"
        ]
    },

As specified by CC, the correlation between the entries of bionetworkName and tissueAtlas are lost. But that's what was specified and I assume it's intentional and not problematic.

On other /index/… endpoints, isTissueAtlasProject would be an array, but it would always be a singleton.

Note that the schema is rather lax (allowing nulls, absent properties and empty arrays) so the implementation should be lenient in that respect. The atlas metadata (see atlas.json above) already appears to be only partially populated in some projects.

Here is the jq invocation that was used to compile project.json: cat atlas.json | jq '[.[]|{"projectId": .project, "isTissueAtlasProject":[.hca_bionetworks[].atlas_project]|any, "tissueAtlas":[.hca_bionetworks[]|{"atlas":.hca_tissue_atlas,"version":.hca_tissue_atlas_version}], "bionetworkName":[.hca_bionetworks[].name]}]' > project.json

hannes-ucsc commented 9 months ago

For demo, show the new properties in hits and termFacets for every /index/{entity_type} endpoint response in prod. For each of these endpoints, filter by every one of these three new facets. Perform a handover of a PFB manifest to Terra in prod, with a filter that selects files from a project with tissue atlas information populated.

Show that the tissueAtlas facet is populated.

nadove-ucsc commented 8 months ago

Looks like the Demo label was missing.

nadove-ucsc commented 8 months ago

The termFacets entry for tissueAtlas is null for projects that have this fields populated. https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp35&filters=%7B%0A%20%20%22entryId%22%3A%20%7B%0A%20%20%20%20%22is%22%3A%20%5B%0A%20%20%20%20%20%20%22577c946d-6de5-4b55-a854-cd3fde40bff2%22%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&size=10

nadove-ucsc commented 8 months ago

The missing terms aggregation is due to Nested being used as the field type of contents.projects.tissue_atlas, which breaks aggregation since there's no corresponding .keyword field (the .keyword field(s) only exist for the nested leaf fields). To fix this, I think we'd need to implement a strategy analogous to ValueAndUnit, which facilitates aggregation of the similarly structured contents.donors.organism_age field by representing it as a single string ("{value} {unit}") in the index. But such an implementation is complicated by the sparseness of the current tissue_atlas metadata.

NoopDog commented 7 months ago

Hi @hannes-ucsc @nadove-ucsc do you have any questions for us on this feature? Cheers, D

cc: @bvizzier-ucsc

nadove-ucsc commented 5 months ago

Current status:

I'm not sure how important it is to add tissueAtlas to the other indices, or how difficult that would be (I don't think there's any precedent aggregating Nested fields). But the broken filter definitely needs to be fixed.

nadove-ucsc commented 5 months ago

This is (at least part of) how to fix the broken filter:

Subject: [PATCH] WIP
---
Index: src/azul/plugins/metadata/hca/__init__.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/plugins/metadata/hca/__init__.py b/src/azul/plugins/metadata/hca/__init__.py
--- a/src/azul/plugins/metadata/hca/__init__.py (revision 560d8306d5a75dfc7e5fd77e5d3363d822e686c0)
+++ b/src/azul/plugins/metadata/hca/__init__.py (date 1717455338732)
@@ -102,6 +102,9 @@
                         'properties': {
                             'accessions': {
                                 'type': 'nested'
+                            },
+                            'tissue_atlas': {
+                                'type': 'nested'
                             }
                         }
                     }