Failed to find loop device[BUG]

[Zifeng-L]

Describe the bug I tried the nextflow with singularity and there were something wrong with nextflow.container.SingularityCache.

To Reproduce Steps to reproduce the behavior:

1. Configure with these options:

   
   manifest {
   name = 'vib-singlecell-nf/vsn-pipelines'
   description = 'A repository of pipelines for single-cell data in Nextflow DSL2'
   homePage = 'https://github.com/vib-singlecell-nf/vsn-pipelines'
   version = '0.26.1'
   mainScript = 'main.nf'
   defaultBranch = 'master'
   nextflowVersion = '!>=20.10.0'
   }

params { global { project_name = '10x_PBMC' outdir = 'out' } misc { test { enabled = false } } utils { container = 'vibsinglecellnf/utils:0.4.0' publish { compressionLevel = 6 annotateWithBatchVariableName = false mode = 'link' } } sc { file_converter { off = 'h5ad' tagCellWithSampleId = true remove10xGEMWell = false useFilteredMatrix = true makeVarIndexUnique = false } scanpy { container = 'vibsinglecellnf/scanpy:0.5.2' report { annotations_to_plot = [] } feature_selection { report_ipynb = '/src/scanpy/bin/reports/sc_select_variable_genes_report.ipynb' method = 'mean_disp_plot' minMean = 0.0125 maxMean = 3 minDisp = 0.5 off = 'h5ad' } feature_scaling { method = 'zscore_scale' maxSD = 10 off = 'h5ad' } neighborhood_graph { nPcs = 50 off = 'h5ad' } dim_reduction { report_ipynb = '/src/scanpy/bin/reports/sc_dim_reduction_report.ipynb' pca { method = 'pca' nComps = 50 off = 'h5ad' } umap { method = 'umap' off = 'h5ad' } tsne { method = 'tsne' off = 'h5ad' } } clustering { preflight_checks = true report_ipynb = '/src/scanpy/bin/reports/sc_clustering_report.ipynb' method = 'louvain' resolution = 0.8 off = 'h5ad' } marker_genes { method = 'wilcoxon' ngenes = 0 groupby = 'louvain' off = 'h5ad' } filter { report_ipynb = '/src/scanpy/bin/reports/sc_filter_qc_report.ipynb' cellFilterStrategy = 'fixedthresholds' cellFilterMinNGenes = 200 cellFilterMaxNGenes = 4000 cellFilterMaxPercentMito = 0.15 geneFilterMinNCells = 3 off = 'h5ad' outdir = 'out' } data_transformation { method = 'log1p' off = 'h5ad' } normalization { method = 'cpx' countsPerCellAfter = 10000 off = 'h5ad' } } scope { genome = '' tree { level_1 = '' level_2 = '' level_3 = '' } } } data { tenx { cellranger_mex = 'data/10x/1k_pbmc/1kpbmc*/outs/' } } }

process { executor = 'local' cpus = 2 memory = '60 GB' clusterOptions = '-A cluster_account' withLabel:compute_resourcesdefault { time = '1h' } withLabel:compute_resources__minimal { cpus = 1 memory = '1 GB' } withLabel:compute_resourcesmem { cpus = 4 memory = '160 GB' } withLabel:compute_resourcescpu { cpus = 20 memory = '80 GB' } withLabel:compute_resources__report { maxForks = 2 cpus = 1 memory = '160 GB' } withLabel:compute_resources24hqueue { time = '24h' } }

timeline { enabled = true file = 'out/nextflow_reports/execution_timeline.html' }

report { enabled = true file = 'out/nextflow_reports/execution_report.html' }

trace { enabled = true file = 'out/nextflow_reports/execution_trace.txt' }

dag { enabled = true file = 'out/nextflow_reports/pipeline_dag.svg' }

min { enabled = false }

vsc { enabled = false }

singularity { enabled = true autoMounts = true runOptions = '--cleanenv -H $PWD -B ${HOME}' }

...

2. Run using this entry point:
<!-- Please specify the run command used (if applicable, otherwise delete this block): -->

!/bin/bash

SBATCH --job-name=test

SBATCH --nodes=1

SBATCH --ntasks-per-node=40

SBATCH --time=2-00:00:00

SBATCH --comment project_name

################################################################# source activate scnf export NXF_SINGULARITY_CACHEDIR=/home/u21111240011/.singularity cd /home/u21111240011/test/single_sample_test_1 nextflow -C single_sample.config \ run vib-singlecell-nf/vsn-pipelines \ -entry single_sample

3. See error:
<!-- Please paste the full error output in this code block (if applicable, otherwise delete this block): -->

[1k_pbmc_v3_chemistry, /home/u21111240011/test/single_sample_test_1/data/10x/1k_pbmc/1k_pbmc_v3_chemistry/outs, 10x_cellranger_mex_outs, h5ad, NULL] [1k_pbmc_v2_chemistry, /home/u21111240011/test/single_sample_test_1/data/10x/1k_pbmc/1k_pbmc_v2_chemistry/outs, 10x_cellranger_mex_outs, h5ad, NULL] Error executing process > 'single_sample:SINGLE_SAMPLE:SC__FILE_CONVERTER (2)'

Caused by: Process single_sample:SINGLE_SAMPLE:SC__FILE_CONVERTER (2) terminated with an error exit status (255)

Command executed:

/home/u21111240011/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/src/utils/bin/sc_file_converter.py --sample-id "1k_pbmc_v2_chemistry" --tag-cell-with-sample-id true --input-format 10x_cellranger_mex --output-format h5ad /home/u21111240011/test/single_sample_test_1/data/10x/1k_pbmc/1k_pbmc_v2_chemistry/outs/filtered_feature_bc_matrix "1k_pbmc_v2_chemistry.SC__FILE_CONVERTER.h5ad"

Command exit status: 255

Command output: (empty)

Command error: FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

Work dir: /home/u21111240011/test/single_sample_test_1/work/60/045b8d1ad8acd214dd3af8f2bcb582

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.

**Expected behavior**
It seemed something wrong with singularity. When I tested it in local, it worked well. When I submit it into slurm system, it can not work.
**Screenshots**
If applicable, add screenshots to help explain your problem.

**Please complete the following information:**
 - OS: Linux workstation 4.18.0-80.7.1.el8_0.x86_64
 - Nextflow Version: 21.04.0
 - vsn-pipelines Version: 962111a16c