search

lilab-bcb / cumulus

Cloud-based scalable and efficient single-cell genomics workflows
https://cumulus.readthedocs.io
BSD 3-Clause "New" or "Revised" License
55 stars 30 forks source link

Docker Registry version for SpaceRanger & mkfastq #402

Closed chenebroad closed 2 months ago

chenebroad commented 7 months ago

We've been running Spaceranger on Terra, running it in full (run_count & run_mkfastq) causes an error when trying to run version "3.0.0".

Spaceranger mkfastq points to gcr.io/broad-cumulus. Latest tag being 2.1.1 Spaceranger counts pulls from quay.io which has 3.0.0

Can we have a change to allow for separate versions for dockers for mkfastq and spaceranger count?

yihming commented 7 months ago

Hi @chenebroad . This issue is a bit more descent, because we no longer have write access to gcr.io/broad-cumulus registry, so it could be hard for us to sync it there.

I know that for Cellranger, users in Broad KCO run their own bcl_convert workflow to generate FASTQ files, and then run our Cellranger workflow with run_mkfastq set to false.

Have you tried this approach? I'm actually a little surprised to see this error, as I thought only mkfastq would be executed in gcr.io/broad-cumulus/spaceranger docker image. Did you confirm that the error is because of this version inconsistency?

chenebroad commented 7 months ago

Hi @yihming,

It's more like how the wdl pulls from the dockers? I believe there is only one variable "spanceranger_version" that dictates which docker tag to pull (since it can't find broad-cumulus 3.0.0 it will fail on Terra).

Your suggested solution would've been the next thing we tried, but ultimately what ended up working was FASTQ generation: run_mkfastq -> True run_count -> False spaceranger_version -> 2.1.1

Spaceranger count run_mkfastq -> False run_count -> True spaceranger_version -> 3.0.0

I think in the future we'll opt to go with bcl-convert -> spaceranger_count, I wasn't aware that you weren't able to commit to broad-cumulus.

yihming commented 7 months ago

I see what you mean now. Then I would go the same way as yours around the issue.

We haven't included bcl-convert yet, and currently still uses bcl2fastq. It looks like Illumina only provides active update and support on bcl-convert. So if the bcl-convert workflow at Broad is already established, please go for it.

At our team's side, we haven't decided yet when to support bcl-convert.

yihming commented 2 months ago

Just close this thread, as the issue could be solved by using Broad KCO's bcl-convert workflow to cover the mkfastq step.

I'll close this thread. Please feel free to reach out for any future issue.