Charliecloud simultaneous pull error

fa2k commented 1 year ago

Bug report

Expected behavior and actual behavior

Running an nf-core pipeline would download the charliecloud images on-demand.

When multiple different processes are scheduled at the same time, requiring different images, there is an error message and nextflow stops (error shown below).

Steps to reproduce the problem

When there are no local cached charliecloud images, the following command can reproduce the error:

nextflow run nf-core/raredisease -r dev  --outdir raredisease --genome GRCh38 --analysis_type wgs  -profile test,charliecloud

Program output

Error executing process > 'NFCORE_RAREDISEASE:RAREDISEASE:CHECK_INPUT:SAMPLESHEET_CHECK (1)'

Caused by:
  Charliecloud failed to pull image
  command: ch-image pull -s /data/ephemeral/charlie quay.io/biocontainers/python:3.9--1 > /dev/null
  status : 1
  message:
    error: storage directory is already in use
    hint: concurrent instances of ch-image cannot share the same storage directory

Environment

Nextflow version: 23.01.0-edge
Java version: openjdk 11.0.13
Operating system: Linux
Bash version: 5.1.16(1)-release
Charliecloud version: 0.30 (from conda)

Additional context

It's possible to work around the problem by manually pulling all the required images before running nextflow.

reidpr commented 1 year ago

Just FYI older versions of Charliecloud wouldn't throw this error, but the lack of concurrency was still present, so they could just silently break stuff.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

reidpr commented 1 year ago

Bot complaints notwithstanding, Charliecloud still does not support parallel access to its storage directory. So unless Nextflow has changed something, this bug is still present.

nextflow-io / nextflow