possible bug with Wave containers and process `MTX_TO_SEURAT`?

[klkeys]

Description of the bug

when running the nf-core/scrnaseq full test on a Nextflow Tower Forge deployment with Wave containers, I get the following error:

Error executing process > 'NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (pbmc8k)'

Caused by:
  Task failed to start - CannotPullContainerError: Error response from daemon: unauthorized: repository 'docker.io/satijalab/seurat:4.1.0' unauthorized (401)

the Docker registry path looks kosher. a 401 error indicates a lack of permissions to pull. is this coming from DockerHub itself? do we need special permissions to pull from satijalab/seurat?

FWIW it looks like all previous pipeline steps completed successfully, including pulling the nfcore/cellranger image from DockerHub.

Command used and terminal output

nextflow run 'https://github.com/nf-core/scrnaseq'
         -name gigantic_swartz_3
         -params-file 'https://api.tower.nf/ephemeral/ibz7hOsMyj2bMILA9YAOtg.json'
         -with-tower
         -r c86646e4a818397f4bddfffd641b34240423f3ea
         -profile docker,test_full
         -resume 561aed79-1e75-463d-8ff0-9ccdf9b0b5d4

Relevant files

nf-2DpvR2NIJ8kerP.log

System information

• nextflow version: 22.10.6
• hardware: cloud
• executor: awsbatch
• container engine: docker
• OS: linux?
• version of nf-core/scrnaseq: 2.1.0

[grst]

Maybe you exceeded the docker hub rate limit?

On Mon, 20 Feb 2023 at 00:45, Kevin L. Keys @.***> wrote:

Description of the bug

when running the nf-core/scrnaseq full test on a Nextflow Tower Forge deployment with Wave containers, I get the following error:

Error executing process > 'NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (pbmc8k)'

Caused by: Task failed to start - CannotPullContainerError: Error response from daemon: unauthorized: repository 'docker.io/satijalab/seurat:4.1.0' unauthorized (401)

the Docker registry path looks kosher. a 401 error indicates a lack of permissions to pull. is this coming from DockerHub itself? do we need special permissions to pull from satijalab/seurat?

FWIW it looks like all previous pipeline steps completed successfully, including pulling the nfcore/cellranger image from DockerHub. Command used and terminal output

nextflow run 'https://github.com/nf-core/scrnaseq' -name gigantic_swartz_3 -params-file 'https://api.tower.nf/ephemeral/ibz7hOsMyj2bMILA9YAOtg.json' -with-tower -r c86646e4a818397f4bddfffd641b34240423f3ea -profile docker,test_full -resume 561aed79-1e75-463d-8ff0-9ccdf9b0b5d4

Relevant files

nf-2DpvR2NIJ8kerP.log https://github.com/nf-core/scrnaseq/files/10778264/nf-2DpvR2NIJ8kerP.log System information

• nextflow version: 22.10.6
• hardware: cloud
• executor: awsbatch
• container engine: docker
• OS: linux?
• version of nf-core/scrnaseq: 2.1.0

— Reply to this email directly, view it on GitHub https://github.com/nf-core/scrnaseq/issues/199, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVZRV3RT6ANQ2XKBNVQ2NDWYKV75ANCNFSM6AAAAAAVBHYEA4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[klkeys]

wouldn't that yield a 429 response error instead?

[grst]

hmm, fair enough. Can you pull the container manually on the same system? Locally, I can pull that container without issues and without authentication.

[klkeys]

I tried pulling on a node sitting in the same private subnet where the pipeline runs:

docker pull satijalab/seurat:4.1.0

executes without problem.

in case it matters, the container image source is actually from wave.seqera.io/wt/a4170753d92b/satijalab/seurat:4.1.0. I'm not sure how to pull from there outside of NF Tower since I think that pulls require a Tower token...?

[grst]

I'm starting to believe this could be an issue with tower/wave rather than scrnaseq. Maybe you could try reaching out to sequera through one of their channels?

[klkeys]

to test this, I deployed a second NF Tower Forge setup, but without Wave containers

it executed successfully with container image satijalab/seurat:4.1.0

the (problematic) Wave container image is wave.seqera.io/wt/73c655ad69a2/satijalab/seurat:4.1.0

I dug into the Wave docs and found this:

Wave currently does not support ADD, COPY and other Dockerfile commands that access files in the host file system.

and if you glance at the Dockerfile for Seurat 4.1.0 it has lines like

ADD file:5c44a80f547b7d68b550b0e64aef898b361666857abf9a5c8f3f8d0567b8e8e4 in / 
#...
COPY dir:fd50c611faf56007789273a44c3936561b29e8e6b75b727017cbe7a98c88d412 in /rocker_scripts
#...

therefore it would seem for now that Wave and the Seurat container are simply not compatible.

[pditommaso]

I guess you need to provide your Docker creds in the Tower workspace where you are launching this pipeline

[robsyme]

Wave can be used in two ways: 1) to append layers to existing container images (to allow tasks to use Fusion, for example), or 2) to build container images denovo using conda or Dockerfiles checked into the workflow repository.

The limitation for preventing ADD and COPY commands is only relevant when using Wave to build containers denovo. In your case, you're pulling the already-built image layers from docker.io, so the ADD/COPY limitation is moot.

If you supply Docker.io login details to Tower in the "Credentials" tab shown below, those credentials will be passed to Wave which will use them to pull the satijalab/seurat:4.1.0 manifest. Supplying credentials allows for both retrieval of private images, and also allows Wave to pull public container manifests without worrying about rate limiting, which may have been the issue here.

[klkeys]

I see. in that case, I'll try rerunning with DockerHub credentials in my NF Tower environment. 👍🏼

[robsyme]

Did adding the DockerHub credentials resolve the issue, @klkeys?

[klkeys]

apologies @robsyme, I set that workflow to run overnight and went to bed 😅

it looks like adding DockerHub credentials did the trick.

I'm confused why the additional credentials are necessary. a Wave-less run of this workflow without Docker credentials pulled that Docker container just fine. adding a note about Docker credentials might be a good addition to the Wave docs.

either way, problem solved, so I'll close this issue. thank you @robsyme @pditommaso @grst!

[robsyme]

No apologies necessary - I was genuinely interested in the result. Your suggestion for the docs is valid and we'll endeavour to make why and when you would attach those credentials more explicit.

For clarity, when you are pulling a container via Wave, the flow is outlined in the figure below (pulled from the Wave blog post)

The request to pull satijalab/seurat:4.1.0 is made by the Wave service, shared by all other Wave users. While your single request may be a single pull which would not require authentication, it is indistinguishable from all of the other pulls made by other (unauthenticated) Wave users. All of these unauthenticated pulls together trigger API limits on DockerHub.

When you supply Docker credentials to Tower, Wave will borrow those credentials from Tower and attach them to its request for satijalab/seurat:4.1.0. In doing so, it distinguishes the Wave actions on your behalf from the pool of actions requested by unauthenticated Wave user, separating it from the pack and ensuring it does not fall afoul of any throttling or limits by DockerHub.

Bat-signalling @llewellyn-sl - this is context for a documentation update I'm about to ping you about.