search

HumanCellAtlas / secondary-analysis

Secondary Analysis Service of the Human Cell Atlas Data Coordination Platform
https://pipelines.data.humancellatlas.org/ui/
BSD 3-Clause "New" or "Revised" License
3 stars 2 forks source link

Analyze 10x KidneySingleCellAtlas dataset with Optimus in production #784

Closed kbergin closed 5 years ago

kbergin commented 5 years ago

Project uuid is abe1a013-af7a-45ed-8c26-f3793c24a1f4 and it should start 47 workflows. Note this project was ingested twice so please ensure this is the one that gets analyzed.

https://tracker.data.humancellatlas.org/

┆Issue is synchronized with this Jira Story

ESapenaVentura commented 5 years ago

I'm the data wrangler for this dataset. Does anyone know what has happened with the analysis? It's saying in the tracker that there are 49 workflows, 3 of which have failed. I have no access to the job manager, so this is all I know.

If anyone could please give me an indication of what's happening, that would be lovely. Thanks!

samanehsan commented 5 years ago

Hi @ESapenaVentura! We have 46/47 successful workflows for that dataset and are looking into the failed workflow. There is no way to "restart" a workflow -- we just submit a new workflow to process the data, which is why there are 49 total at the moment.

Also this ticket has instructions for getting set up with job manager: https://github.com/HumanCellAtlas/secondary-analysis/issues/237

ESapenaVentura commented 5 years ago

Thanks @samanehsan! I didn't know it, that is nice to know :)

Also, @mshadbolt ^^ We should get set up with the job manager

ESapenaVentura commented 5 years ago

Hi @samanehsan! Is there any plan to submit the last workflow soon?

samanehsan commented 5 years ago

@ESapenaVentura we just re-submitted it with a change that should log more detailed info to help us with debugging, since it was unclear to us why it was failing: https://job-manager.caas-prod.broadinstitute.org/jobs/f77f6137-2744-47bf-a15f-3ef7a1526165?q=caas-collection-name%3Dlira-prod%26project_shortname%3DKidneySingleCellAtlas

samanehsan commented 5 years ago

Oops made a mistake with manually re-running the workflow so I restarted it: https://job-manager.caas-prod.broadinstitute.org/jobs/594d367e-d542-4ff1-bb73-1405de7f45f7?q=caas-collection-name%3Dlira-prod%26project_shortname%3DKidneySingleCellAtlas

ESapenaVentura commented 5 years ago

Any news on this?

kbergin commented 5 years ago

➤ Saman Ehsan commented:

Yes, I was able to confirm the failing steps do not have enough memory so I'm going to sync up with our comp bios about this tomorrow! Apologies for the delay here!

kbergin commented 5 years ago

➤ Saman Ehsan commented:

I created a PR to increase the memory for the failing steps here: https://github.com/HumanCellAtlas/skylab/pull/255 ( https://github.com/HumanCellAtlas/skylab/pull/255|smart-link )

kbergin commented 5 years ago

➤ Saman Ehsan commented:

Re-running the workflow with more memory now: https://job-manager.caas-prod.broadinstitute.org/jobs/7c2a4d69-c75c-4d82-bfaa-c9cfb6d682e5?q=caas-collection-name%3Dlira-prod%26project_shortname%3DKidneySingleCellAtlas ( https://job-manager.caas-prod.broadinstitute.org/jobs/7c2a4d69-c75c-4d82-bfaa-c9cfb6d682e5?q=caas-collection-name%3Dlira-prod%26project_shortname%3DKidneySingleCellAtlas )