Update cell suspension -> sequence file -> analysis file to cell suspension -> analysis file

[aaclan-ebi]

https://github.com/ebi-ait/hca-ebi-wrangler-central/issues/228

https://contribute.data.humancellatlas.org/submissions/detail?uuid=e6dca5ce-60b2-4087-97bd-fb32b20db01a&project=dc1a41f6-9e09-42a6-959e-3be23db6da56

[ami-day]

@aaclan-ebi picking this up

[idazucchi]

@aaclan-ebi and @idazucchi will meet today to confirm this is working

[aaclan-ebi]

Consulted the DCP2 channel: https://embl-ebi-ait.slack.com/archives/C01360XN04S/p1645456700650769 The submission is ready to be exported once we get a go signal.

Thanks for the discussion @idazucchi !

[ke4]

@idazucchi I was told by @aaclan-ebi that you are going to continue this task and there is no dev task left on this ticket. So I am going to reassign it from @aaclan-ebi to you. I hope it is OK with you.

[aaclan-ebi]

Yes, so from the thread, if they say they’re okay importing from prod staging area to dev Terra dataset, we’ll just click the submit button for this and export (only the metadata) and let them know we have them already exported.

[aaclan-ebi]

If all goes well with their testing, they'll just reimport the prod Terra staging area to prod Terra dataset.

[aaclan-ebi]

Let's keep an eye on that thread.

[gabsie]

Data needs to be in dev first, as per import team. @idazucchi da will move this data to that dev bucket, and will coordinate with @ke4 and @aaclan-ebi on this.

[aaclan-ebi]

The submission was stuck in Exporting because the 4 new processes which were manually created via API were not linked to the project:

TOKEN='insert-token-from-ui'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028c966cf347099900d78f/project" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028ca26cf347099900d790/project" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028ca56cf347099900d791/project" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028ca66cf347099900d792/project" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028c966cf347099900d78f/projects" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028ca26cf347099900d790/projects" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028ca56cf347099900d791/projects" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

curl -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/uri-list" "https://api.ingest.archive.data.humancellatlas.org/processes/62028ca66cf347099900d792/projects" -d 'https://api.ingest.archive.data.humancellatlas.org/projects/6000644a8ec45c6438fd9613'

After fixing the processes, I did the following steps to reexport:

Workaround to reexport a submission stuck

# force set submission to be Graph Valid, so that it can be submitted again
curl -X PUT -H "Authorization: Bearer $TOKEN" https://api.ingest.archive.data.humancellatlas.org/submissionEnvelopes/60d9dad4c78d1b6cc5ca5c91/commitGraphValidEvent

# redeploy state tracker pod, try to make sure it's idle before redeploy to not affect any existing submission

$ stern state-tracker

# find and delete the state-tracker pod
$ kubectl get pods | grep state
ingest-state-tracking-7ffcf64b85-skzgl                  1/1     Running            0          17d
$ kubectl delete pod ingest-state-tracking-7ffcf64b85-skzgl
pod "ingest-state-tracking-7ffcf64b85-skzgl" deleted

# wait for the state tracker to initialise successfully before reexporting the submission
$ stern state-tracker

# via the UI, submit the submission and tick the checkbox to only export metadata

[aaclan-ebi]

After successful export, copied project's prod staging area to dev:

gsutil -m cp -r gs://broad-dsp-monster-hca-prod-ebi-storage/prod/dc1a41f6-9e09-42a6-959e-3be23db6da56/ gs://broad-dsp-monster-hca-dev-ebi-staging/dev/

[ESapenaVentura]

Waiting on import team

[aaclan-ebi]

Azul indexing didn't encounter any issues in dev.

I've tried downloading the metadata tsv and found it hard to compare as Data Portal doesn't provide all the metadata we have submitted yet. I noticed that the experiment4_human_st_lowremove_scp_X_diffmap_pca_coords.txt file now belongs to a different bundle_uuid (which corresponds to the process which produced the file) from 2a67ab7a-bb3c-468e-965b-e36b55d6feb6 to fa41438f-bde4-4914-abb3-526d2d9c65d4. So, I believe the linking changes were submitted fine.

@idazucchi could you do some checks to see if there are any issues in the info from the Data Portal. Thanks!

dev: https://dev.singlecell.gi.ucsc.edu/explore/projects/dc1a41f6-9e09-42a6-959e-3be23db6da56/project-metadata

prod: https://data.humancellatlas.org/explore/projects/dc1a41f6-9e09-42a6-959e-3be23db6da56/project-metadata

[MightyAx]

@Wkt8 please do a visual scan of data portal to ensure export was successful. @idazucchi to double check on return from holiday.

[idazucchi]

I'm assuming that the bundle IDs indicate different subgraphs. The tsv from dev looks fine, the analysis files are grouped in the correct way @aaclan-ebi what do I need to write in the import form?