search

nf-core / circdna

Pipeline for the identification of extrachromosomal circular DNA (ecDNA) from Circle-seq, WGS, and ATAC-seq data that were generated from cancer and other eukaryotic cells.
https://nf-co.re/circdna
MIT License
25 stars 14 forks source link

Test run fails with singularity `-r 1.0.1` #35

Closed madagiurgiu25 closed 1 year ago

madagiurgiu25 commented 1 year ago

Description of the bug

Hi together,

I tried to run the circdna test with singularity nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna and it fails for me on the HPC cluster. Below you have the error messages:

Command used and terminal output

# HPC cluster

nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna

# error
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/circdna] Pipeline completed with errors-
WARN: Access to undefined parameter `singularity_pull_docker_container` -- Initialise it to a default value eg. `params.singularity_pull_docker_container = some_value`
Error executing process > 'NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)'

Caused by:
  Process `NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)` terminated with an error exit status (127)

Command executed:

  check_samplesheet.py \
      samplesheet.csv \
      samplesheet.valid.csv \
      FASTQ

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK":
      python: $(python --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  WARNING: Skipping mount /fast/work/users/giurgium_c/miniconda/envs/nextflow/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
  .command.sh: line 3: check_samplesheet.py: command not found
  INFO:    Cleaning up image...

Work dir:
  /fast/work/users/giurgium_c/amplicon-architect/TR14_nextflow/work/49/26322661fc9e58b820527ba24a85c7

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

Relevant files

No response

System information

HPC cluster

madagiurgiu25 commented 1 year ago

I also tried it locally but same result (however the error is different):

Specifications:

nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna

Pulling Singularity image https://depot.galaxyproject.org/singularity/python:3.8.3 [cache /home/mada/Projects/tools/circdna/work/singularity/depot.galaxyproject.org-singularity-python-3.8.3.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/bwa:0.7.17--hed695b0_7 [cache /home/mada/Projects/tools/circdna/work/singularity/depot.galaxyproject.org-singularity-bwa-0.7.17--hed695b0_7.img]
WARN: Access to undefined parameter `singularity_pull_docker_container` -- Initialise it to a default value eg. `params.singularity_pull_docker_container = some_value`
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /home/mada/Projects/tools/circdna/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a different location
Error executing process > 'NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name depot.galaxyproject.org-singularity-python-3.8.3.img.pulling.1664432786081 https://depot.galaxyproject.org/singularity/python:3.8.3 > /dev/null
  status : 255
  message:
    ERROR: pull is only supported for shub URIs

Thank you. Best, Madalina

madagiurgiu25 commented 1 year ago

I tried again now, by pulling fresh the repository on the HPC cluster:

Versions:

nf-core/circdna

nextflow pull nf-core/circdna
Checking nf-core/circdna ...
done - revision: 50681adaf7 [1.0.1]

Command:

nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna

Output:

Same error as above execution_trace_2022-09-29_08-41-30.txt

Execution cancelled -- Finishing pending tasks before exit
-[nf-core/circdna] Pipeline completed with errors-
WARN: Access to undefined parameter `singularity_pull_docker_container` -- Initialise it to a default value eg. `params.singularity_pull_docker_container = some_value`
Error executing process > 'NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)'

Caused by:
  Process `NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)` terminated with an error exit status (127)

Command executed:

  check_samplesheet.py \
      samplesheet.csv \
      samplesheet.valid.csv \
      FASTQ

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_CIRCDNA:CIRCDNA:INPUT_CHECK:SAMPLESHEET_CHECK":
      python: $(python --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  WARNING: Skipping mount /fast/work/users/giurgium_c/miniconda/envs/nextflow/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
  .command.sh: line 3: check_samplesheet.py: command not found
  INFO:    Cleaning up image...

Work dir:
  /fast/work/users/giurgium_c/amplicon-architect/TR14_nextflow/work/b0/57c08a57995cd4294cad51507e07f6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
DSchreyer commented 1 year ago

Hey, I still cannot replicate it.

I also ran nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna on our HPC with nextflow version 22.04.0 and singularity version 3.8.7 and on a second server with nextflow version 21.10.6. Both worked fine for me and the pipeline was able to pull the singularity containers and run the test profile.

What is different to the run on the 18.09.2022 and your last run? Did your singularity configs change?

~/src/test$ nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna
N E X T F L O W  ~  version 22.04.0
Launching `https://github.com/nf-core/circdna` [desperate_engelbart] DSL2 - revision: 50681adaf7 [1.0.1]

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/circdna v1.0.1
------------------------------------------------------
Core Nextflow options
  revision                  : 1.0.1
  runName                   : desperate_engelbart
  containerEngine           : singularity
...
  profile                   : test,singularity
...

Finished Run:


Completed at: 11-Oct-2022 12:01:55
Duration    : 10m 41s
CPU hours   : 1.1
Succeeded   : 97
madagiurgiu25 commented 1 year ago

Hi,

So my versions are:

Your's are:

I can't make it run. I now create a fresh conda environment:

conda create -n test_aa python=3.7
conda activate test_aa
mamba install -y nextflow==22.04.0 
mamba install -y libarchive
mamba install -c "conda-forge/label/broken" singularity==3.8.7 -y

(test_aa) $ nextflow -v
nextflow version 22.04.0.5697
(test_aa) $ singularity --version
singularity version 3.8.7
(test_aa) $ python --version
Python 3.7.12
(test_aa) $ conda --version
conda 4.10.3

When I run your command:

nextflow run nf-core/circdna -r 1.0.1 -profile test,singularity --outdir test_circdna

I receive the following output:

N E X T F L O W  ~  version 22.04.0
Pulling nf-core/circdna ...
Cannot invoke "org.eclipse.jgit.lib.ObjectId.name()" because the return value of "org.eclipse.jgit.lib.Ref.getObjectId()" is null
madagiurgiu25 commented 1 year ago
Screenshot 2022-10-11 at 15 36 17
DSchreyer commented 1 year ago

Did you find a solution yet? I changed the singularity version to 3.8.0 and it still ran through normally. Did you try to run it on another system? If yes and it works, then maybe your configurations are not optimal.

Sorry, but I seem not be able to replicate your issue.

madagiurgiu25 commented 1 year ago

Hi, no not yet, now I receive still the above reported error.

I fixed the issue:

Cannot invoke "org.eclipse.jgit.lib.ObjectId.name()" because the return value of "org.eclipse.jgit.lib.Ref.getObjectId()" is null

by setting correctly the java VM:

export JAVA_HOME=<path>/miniconda/envs/aa/lib/jvm
export PATH=<path>/miniconda/envs/aa/bin:$PATH
export JAVA_VER=17
DSchreyer commented 1 year ago

Can you try and use conda?

DSchreyer commented 1 year ago

I will close this issue now due to inactivity. Please feel free to re-open it when the issue still exists. I hope some of the issues were addressed in the newest release!