AdultLungElo - A spatial multi-omics atlas of the human lung reveals a novel immune cell survival niche

[Wkt8]

Primary Wrangler: Wei Kheng Teh

Secondary Wrangler:

Associated files:

Paper: https://www.biorxiv.org/content/10.1101/2021.11.26.470108v1

Ingest: https://contribute.data.humancellatlas.org/projects/detail?uuid=957261f7-2bd6-4358-a6ed-24ee080d5cfc

Google Drive: https://drive.google.com/drive/folders/1Ty3xrV-MEcjCDklpUHSC1Yna7pcvv699 Google Sheet: https://docs.google.com/spreadsheets/d/1skxc1GJovUgfn4Ugbm3zZiuzvOugEA0H/edit?usp=sharing&ouid=114069941208994528669&rtpof=true&sd=true

Key Events

• [x] Send initial instructions about contributing, including questionnaire and T&Cs
• [x] Receive project questionnaire
• [x] Assign the Primary Wrangler
• [ ] Receive T&C form filled out by at least 1 representative of group/project
• [x] Send custom metadata template spreadsheet, how-to guide and directions for submitting data, matrix and cell annotation files.
• [ ] Receive spreadsheet (to a sufficient standard to submit dataset)
• [x] Receive data, matrix and cell annotation files (to a sufficient standard to submit dataset)
• [x] Curate metadata spreadsheet with ontologies
• [ ] Upload sheet to validate metadata
• [ ] Run graph validator & Check linking
• [ ] Validate data files
• [ ] Ask the Secondary Wrangler for an end-to-end review of the project. Ask the Expertise Wrangler to review specific tabs if needed
• [ ] Get final approval of submission from the contributor including release date, metadata spreadsheet and data files
• [ ] Submit to production and inform ingest-devs the project is ready for archiving
• [ ] Email UX to inform that they're ready to send survey

Please track the below as well as the key events:

1. Track date first spreadsheet received and final spreadsheet sent by editing ticket to include date next to event.
2. Track spreadsheet iterations by placing asterisks next to receive spreadsheet event.
3. Track any metadata issues/tickets made for dataset with a bulleted list of links under received spreadsheet event. Links should be to the ticket in the metadata repo.

Final Steps:

• [ ] secondary review
• [ ] export to HCA (after checking with Elo for publicity date)
• [ ] archiving of Visium to ArrayExpress
• [ ] archiving of data files and metadata via DSP to ENA

[Wkt8]

Had a meeting with contributors Elo and Peng together with @ESapenaVentura - Notes and Email sent out to the contributors here in this google doc: https://docs.google.com/document/d/1wJIjR55kWyGbcZ3FOK8BMOMCKkSMD6oMfcgYLt68mtA/edit

[Wkt8]

This is actually the ticket for the AdultLungElo project. This includes CBTM adult lung samples.

[Wkt8]

This is at a point where we are waiting for additional Visium metadata and for Elo to send a final annotated cell matrice.

[prabh-t]

Wei requires help to archive this dataset.

[Wkt8]

Needs:

• [ ] secondary review
• [ ] export to HCA (after checking with Elo for publicity date)
• [ ] archiving of Visium to ArrayExpress
• [ ] archiving of data files and metadata via DSP to ENA

[MightyAx]

@ami-day to secondary review.

[Wkt8]

Ami has secondary reviewed and given me feedback.

[gabsie]

no publication date yet. might be private for now.

[idazucchi]

Elo confirmed that the data can be published

[Wkt8]

https://contribute.data.humancellatlas.org/submissions/detail?uuid=9e880501-55f3-442d-9908-efae1dac9be9&project=957261f7-2bd6-4358-a6ed-24ee080d5cfc

[ipediez]

Stuck on metadata validating since yesterday, @prabh-t to help Wei to unstuck this, probably related to yesterday's problems with projects stuck

[Wkt8]

Prabhat says it's just a size issue - so will keep waiting on it.

[Wkt8]

Exported. Now needs archiving.

[Wkt8]

I am following this SOP: https://ebi-ait.github.io/hca-ebi-wrangler-central/SOPs/archiving_SOP.html#step-2-of-3---archiving-files-to-dsp

DSP archiving is stalled, I have created a DSP submission with the ingest submission UUID with a POST request (30th March, 2pm), but I receive 'null' when attempting to retrieve the DSP submission UUID via the ingest submission UUID.

[ofanobilbao]

@prabh-t to try and help on this

[prabh-t]

curl -X POST https://archiver.ingest.archive.data.humancellatlas.org/archiveSubmissions -H 'Content-Type: application/json' -H 'Api-Key: xxx' -d '{"submission_uuid": "9e880501-55f3-442d-9908-efae1dac9be9", "alias_prefix": "HCA"}'

curl -X GET  -H 'Api-Key: xxx' https://archiver.ingest.archive.data.humancellatlas.org/latestArchiveSubmission/9e880501-55f3-442d-9908-efae1dac9be9

https://contribute.data.humancellatlas.org/submissions/detail?uuid=9e880501-55f3-442d-9908-efae1dac9be9&project=957261f7-2bd6-4358-a6ed-24ee080d5cfc

[Wkt8]

Update required for export to the DCP that the curator is unspecified. Need to think about how this will work with the deleted processes!

[aaclan-ebi]

The metadata entities of this project were updated with accessions. They also need to be submitted to DCP.

[ESapenaVentura]

👀 duty:

• Waiting for files to appear in ENA
• Need to check accessions propagate back to ingest

[ofanobilbao]

@Wkt8 if I understand you've already pushed this to DCP and archived it and now you're waiting for data to be in ENA and then update the DCP submission with the accessions. Correct?

[Wkt8]

Yes.

[idazucchi]

needs minor update

• to change 10x labels and
• visium bulk cell?
• add wei as curator
• add cell count estimate

[Wkt8]

@amnonkhen or @MightyAx could you please set this dataset state from Archived to Metadata Valid so that I could edit and update it? Updates are not permitted in the Archived metadata state.

[amnonkhen]

@Wkt8 it is done. I used this SOP and this script.

[Wkt8]

Strangely this is still in the 'archived' state, maybe redeploying or some other work forced it back to 'archived'? @ke4 please could you have a look?

[ke4]

Done it with curl request and rollout deployment of ingest-state-tracking. It looks OK now and the submission is in Metadata Valid status.

[Wkt8]

Update made, import form submitted, moving to done

[Wkt8]

In the exported state.

[Wkt8]

Meeting with Amanda to discuss as some analysis files need to be updated - see here: https://docs.google.com/document/d/1a5rjrLw3Al2EEWAqewkV-aZbmge0qERR0vQSd2i_VvY/edit

[Wkt8]

Spreadsheet updated and available here: https://docs.google.com/spreadsheets/d/1skxc1GJovUgfn4Ugbm3zZiuzvOugEA0H/edit?usp=sharing&ouid=114069941208994528669&rtpof=true&sd=true

Remaining actions

• [x] Put new files into a hca-util upload area
• [x] Delete staging area
• [ ] Check Nathan has deleted the terra staging area
• [x] Wait until the 'deleting submissions and adding new submissions' fix has been put into production
• [x] Delete existing submission from https://contribute.data.humancellatlas.org/projects/detail?uuid=957261f7-2bd6-4358-a6ed-24ee080d5cfc
• [x] Submit new submission
• [ ] Export project
• [ ] Submit export form
• [ ] Notify downstream components
• [ ] Notify Amanda Oliver (ao15@sanger.ac.uk)
• [ ] Submit files to cellxgene

[idazucchi]

some files are stuck in validation --> ops ticket

[Wkt8]

Exporting!

[idazucchi]

@Wkt8 this is marked as release 21 in the manifest, can you change it to release 22?

[Wkt8]

yeah we need to change it oops

[Wkt8]

Verified on catalog dcp = 22. All good.

[Wkt8]

Making another update for publication links

[Wkt8]

now in graph validating

[Wkt8]

• [x] Exported
• [x] Submitted product import form

[idazucchi]

verified in the data browser

[ofanobilbao]

I can't see a reason for this Dataset to be open again and in Stalled. So I will close it. Does not seem to have any comments saying something is wrong with it.

[arschat]

Contributor asked to add the processed spaceranger output files both in HCA & in ArrayExpress. They have provided the files here.

I've opened a ticket in ArrayExpress helpdesk to re-open the the submission, and I will update in HCA for the next release 36.

[arschat]

• Uploaded the files through hca-util (upload area: 79e143e8-9e91-4afb-aa23-ac7ee7be46ae/) and sync designated to "Upload Area Location"
• Updated new files though api but also previous files to upgrade to analysis_file 7.0.0 from 6.5.0.
• Created new processes to link image_specimen to new analysis_files & protocols.
• Graph validated & exported & import form sent
  Code used to update files through API

from hca_ingest.api.ingestapi import IngestApi

api = IngestApi(url="https://api.ingest.archive.data.humancellatlas.org/") headers_json = {'Content-Type': 'application/json', 'Authorization': 'Bearer ' + token} api.set_token(f"bearer {token}")

query = [{ "field": "content.describedBy", "operator": "IS", "value": "https://schema.humancellatlas.org/type/file/6.5.0/analysis_file" }, { "field": "project.id", "operator": "IS", "value": "61a7c1dcc7b79307bb4863b7" } ]

analysis_files = api.post('https://api.ingest.archive.data.humancellatlas.org/files/query?operator=AND', json=query).json() for old_file in analysis_files['_embedded']['files']: json = {'content': old_file['content']} json['content']['describedBy'] = 'https://schema.humancellatlas.org/type/file/7.0.0/analysis_file' response = api.patch(old_file['_links']['self']['href'], headers=headers_json, json=json)

query = [{ "field": "content", "operator": "IS", "value": None }, { "field": "project.id", "operator": "IS", "value": "61a7c1dcc7b79307bb4863b7" } ]

files_list = api.post('https://api.ingest.archive.data.humancellatlas.org/files/query?operator=AND', json=query).json()

for file in files_list['_embedded']['files']: json = {"content": { "describedBy": "https://schema.humancellatlas.org/type/file/7.0.0/analysis_file", "schema_type": "file", "file_core": { "file_name": file['fileName'], "format": "zip", "content_description": [ { "text": "gene expression matrix", "ontology": "data:3112", "ontology_label": "gene expression matrix" } ], "file_source": "Contributor" }, "genome_assembly_version": "GRCh38" } } response = api.patch(file['_links']['self']['href'], headers=headers_json, json=json) if response.ok: print(file['_links']['self']['href']) else: print("ERROR HERE " + file['_links']['self']['href']) print(response.status_code) break

[arschat]

AE opened the experiment for edits, and files were uploaded & assigned to samples. I added the peer-reviewed publication, too. As per Silvie's guidelines, I re-submitted the experiment. Once

• [ ] the AE submission is processed without errors, and
• [x] the HCA submission is verified in the browser I will close this ticket.

[arschat]

Verified in browser. Waiting for AE submission to be live before closing.

[arschat]

Problem in AE submission, needs re-upload of fastq files as well as the processed. Shared Silvie's guidelines with Krzysztof, and wait for them to upload.

[arschat]

Upload of data files completed. Project was validated and re-submitted and waiting for re-validation to complete update.

[arschat]

AE submission is now public. Update has been completed. Closing ticket.