nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.73k stars 625 forks source link

docker.enable error when executor=k8s #4832

Open Chris-KS-C opened 7 months ago

Chris-KS-C commented 7 months ago

Bug report

When using executor='k8s', docker.enable=true appears ineffective for pulling images from public registry. However, specifying docker.registry='docker.io' resolves the issue.

Expected behaviour and actual behaviour

Expected Outcome: Successfully retrieving images from Docker Hub.

Actual Outcome: Failing to retrieve images during the pull process.

Steps to reproduce the problem

nextflow.nf

process PYTHON_DOCKER_HUB{
    output:
    stdout

    container 'python:3.13.0a4-slim'

    """
    #!/usr/bin/env python
    import os
    import sys

    print(os.path.realpath(sys.executable))
    """
}

workflow {    
    PYTHON_DOCKER_HUB()|view
}

nextflow.config

process {
    executor = 'k8s'
}

k8s {
    storageClaimName = 'nextflow'
    storageMountPath = '/mnt'
    workDir = '/mnt/work'
    projectDir = '/mnt/project'
}

docker.enable = true 

Program output

executor > k8s (1)
[c3/66e976] process > PYTHON_DOCKER_HUB [ 0%] 0 of 1
WARN: Failed to copy log for pod nf-c366e976ded2e194743bb417506f4e27
ERROR ~ Error executing process > 'PYTHON_DOCKER_HUB'
Caused by:
K8s pod image cannot be pulled -- Back-off pulling image "image-registry.openshift-image-registry.svc:5000/ht05542-mxs29/python:3.13.0a4-slim"

Environment

Additional context

I tried to pull an image from the private registry Harbor, but it seems that neither the docker.enable nor docker.registry values had any effect on the image pulling process. Despite setting docker.enable=false and docker.registry=null, the image was still successfully pulled.

bentsherman commented 7 months ago

With K8s you don't need to specify docker.enabled = true because K8s is container native. I believe it does use docker.registry to augment the container image name. Not sure where else it would get the registry if you didn't hard-code it in the container directive