search

nf-core / circdna

Pipeline for the identification of extrachromosomal circular DNA (ecDNA) from Circle-seq, WGS, and ATAC-seq data that were generated from cancer and other eukaryotic cells.
https://nf-co.re/circdna
MIT License
25 stars 14 forks source link

Error executing process > 'NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF #37

Closed liu-zhiyang closed 1 year ago

liu-zhiyang commented 1 year ago

Description of the bug

Hi, I ran circdna pipeline on bam files using identifier circle_finder, it broken with exit status 140 while executing process > 'NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF, and here is the error message:

Error executing process > 'NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF (e06588d9bca6003c0f0ec945f109b30a)'

Caused by:
  Process `NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF (e06588d9bca6003c0f0ec945f109b30a)` terminated with an error exit status (140)

Command executed:

  samtools sort -n -@ 6 -o e06588d9bca6003c0f0ec945f109b30a.qname.sorted.bam -T e06588d9bca6003c0f0ec945f109b30a.qname.sorted e06588d9bca6003c0f0ec945f109b30a.md.filtered.sorted.bam
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF":
      samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
  END_VERSIONS

Command exit status:
  140

Command output:
  (empty)

Command error:
  [bam_sort_core] merging from 504 files and 6 in-memory blocks...

Work dir:
  /gpfs/share/home/1710305101/testNextflow/circfindertest/work/3e/d2171697cb00563ea61eda353bc918

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

The Command error seems not like an error. How can I solve this problem? Hope for your reply. Thanks!

Command used and terminal output

command used:
nextflow run nf-core/circdna -profile singularity -r 1.0.1 -resume --circle_identifier ampliconarchitect,circle_finder --email liuzhiyang@pku.edu.cn --input samplesheet.csv --input_format BAM --outdir . --fasta /gpfs/share/home/1710305101/ref/genome/ICGC/GRCh37/genome.fa --igenomes_base /gpfs/share/home/1710305101/ref/igenomes --custom_config_base /gpfs/share/home/1710305101/.nextflow/configs/singularity --aa_data_repo /gpfs/share/home/1710305101/ref/aa_repo --mosek_license_dir /gpfs/share/home/1710305101/mosek/mosek.lic --reference_build GRCh37
terminal output:
executor >  slurm (47)
[a9/a25249] process > NFCORE_CIRCDNA:CIRCDNA:INPU... [100%] 1 of 1 ✔
[e3/70d852] process > NFCORE_CIRCDNA:CIRCDNA:SAMT... [100%] 2 of 2 ✔
[6c/4bc278] process > NFCORE_CIRCDNA:CIRCDNA:SAMT... [100%] 2 of 2 ✔
[dc/adcc33] process > NFCORE_CIRCDNA:CIRCDNA:BAM_... [100%] 2 of 2 ✔
[ce/9e995e] process > NFCORE_CIRCDNA:CIRCDNA:BAM_... [100%] 2 of 2 ✔
[8c/011c42] process > NFCORE_CIRCDNA:CIRCDNA:BAM_... [100%] 2 of 2 ✔
[07/59c8d8] process > NFCORE_CIRCDNA:CIRCDNA:MARK... [100%] 2 of 2 ✔
[92/b57b98] process > NFCORE_CIRCDNA:CIRCDNA:MARK... [100%] 2 of 2 ✔
[22/7a8e3d] process > NFCORE_CIRCDNA:CIRCDNA:MARK... [100%] 2 of 2 ✔
[42/28ac59] process > NFCORE_CIRCDNA:CIRCDNA:MARK... [100%] 2 of 2 ✔
[87/5918e4] process > NFCORE_CIRCDNA:CIRCDNA:MARK... [100%] 2 of 2 ✔
[a2/dd7889] process > NFCORE_CIRCDNA:CIRCDNA:SAMT... [100%] 2 of 2 ✔
[a8/0c631d] process > NFCORE_CIRCDNA:CIRCDNA:SAMT... [100%] 2 of 2 ✔
[d4/1413fd] process > NFCORE_CIRCDNA:CIRCDNA:SAMT... [100%] 2 of 2 ✔
[b5/2101d4] process > NFCORE_CIRCDNA:CIRCDNA:CNVK... [100%] 2 of 2 ✔
[a2/a6b51d] process > NFCORE_CIRCDNA:CIRCDNA:CNVK... [100%] 2 of 2 ✔
[eb/1415bd] process > NFCORE_CIRCDNA:CIRCDNA:COLL... [100%] 2 of 2 ✔
[4b/7d6393] process > NFCORE_CIRCDNA:CIRCDNA:AMPL... [100%] 2 of 2 ✔
[50/bbe7f6] process > NFCORE_CIRCDNA:CIRCDNA:AMPL... [100%] 2 of 2 ✔
[b9/3ecfa3] process > NFCORE_CIRCDNA:CIRCDNA:AMPL... [100%] 2 of 2 ✔
[9b/ad753e] process > NFCORE_CIRCDNA:CIRCDNA:AMPL... [100%] 2 of 2 ✔
[08/c49e51] process > NFCORE_CIRCDNA:CIRCDNA:SUMM... [100%] 2 of 2 ✔
[7b/5f5ca5] process > NFCORE_CIRCDNA:CIRCDNA:SAMT... [100%] 2 of 2, failed: 2 ✘
[-        ] process > NFCORE_CIRCDNA:CIRCDNA:SAMB... -
[-        ] process > NFCORE_CIRCDNA:CIRCDNA:BEDT... -
[9b/e4dd70] process > NFCORE_CIRCDNA:CIRCDNA:BEDT... [100%] 2 of 2 ✔
[-        ] process > NFCORE_CIRCDNA:CIRCDNA:CIRC... -
[-        ] process > NFCORE_CIRCDNA:CIRCDNA:CUST... -
[-        ] process > NFCORE_CIRCDNA:CIRCDNA:MULTIQC -
-[nf-core/circdna] Sent summary e-mail to liuzhiyang@pku.edu.cn (sendmail)-
-[nf-core/circdna] Pipeline completed with errors-
Error executing process > 'NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF (e06588d9bca6003c0f0ec945f109b30a)'

Caused by:
  Process `NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF (e06588d9bca6003c0f0ec945f109b30a)` terminated with an error exit status (140)

Command executed:

  samtools sort -n -@ 6 -o e06588d9bca6003c0f0ec945f109b30a.qname.sorted.bam -T e06588d9bca6003c0f0ec945f109b30a.qname.sorted e06588d9bca6003c0f0ec945f109b30a.md.filtered.sorted.bam
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_CIRCDNA:CIRCDNA:SAMTOOLS_SORT_QNAME_CF":
      samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
  END_VERSIONS

Command exit status:
  140

Command output:
  (empty)

Command error:
  [bam_sort_core] merging from 504 files and 6 in-memory blocks...

Work dir:
  /gpfs/share/home/1710305101/testNextflow/circfindertest/work/c4/cd12e5099a0ac7eb86cac6f62d36e8

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Relevant files

nextflow.log

System information

nextflow version: 22.10.0.5826 hardware: HPC executor: slurm container: singularity 3.5.2 OS: Red Hat 4.8.5 version of nf-core/circdna: 1.0.1

DSchreyer commented 1 year ago

Hi, thanks for reporting this. I tried to simulate your error:

  1. I checked if it could be caused by --input_format BAM but it does run on my instance with a test bam file.

  2. I then checked singularity using --profile singularity and it also works as expected. I did not encounter any errors.

  3. I then used slurm to identify if it could be an HPC issue with the command:

/nfs/software/slurm/current/spool/slurmd/job42306/slurm_script run main.nf -profile singularity,test_AA --circle_identifier circle_finder,ampliconarchitect --email [daniel.schreyer@glasgow.ac.uk](mailto:daniel.schreyer@glasgow.ac.uk) --input runs/samplesheet_test_AA.csv --outdir results --aa_data_repo data_repo/ --input_format BAM

Again, no error detected and the pipeline ran successfully.

Therefore, could you try to run 

  samtools sort -n -@ 6 -o e06588d9bca6003c0f0ec945f109b30a.qname.sorted.bam -T e06588d9bca6003c0f0ec945f109b30a.qname.sorted e06588d9bca6003c0f0ec945f109b30a.md.filtered.sorted.bam

in the work directory /gpfs/share/home/1710305101/testNextflow/circfindertest/work/c4/cd12e5099a0ac7eb86cac6f62d36e8

Let me know if this works. If you get the same error. Can you install a different samtools version. I am unsure where this could come from, but if it works in the work directory then there might be issues with the configuration.

I found an interesting article about exit code 140 (http://research.libd.org/SPEAQeasy/help.html). Could you try and allocate more memory and cpu towards the test run. It is a shot in the dark, but it could be worth it.

Thanks a lot!

liu-zhiyang commented 1 year ago

@DSchreyer Very thanks for your reply. I have tried to manually run samtools sort -n -@ 6 -o e06588d9bca6003c0f0ec945f109b30a.qname.sorted.bam -T e06588d9bca6003c0f0ec945f109b30a.qname.sorted e06588d9bca6003c0f0ec945f109b30a.md.filtered.sorted.bam with 6 cores (according to -@ parameter of samtools and ~ 24 GB memory available). And the process was completed sucessfully after more than 16 hours 30 minutes. I checked that SAMTOOLS_SORT module has the process_medium label with time limit of 8 hours and considering maxRetries = 1 the time limit will be 16 hours for 2nd attempt. So I think the error above maybe caused by time limit. I will try to add more resources and resume the pipeline.

liu-zhiyang commented 1 year ago

@DSchreyer I ‘ve tried to allocate more resources (12 cpus and 24h time limit) and completely rerun the pipeline and all works completed successfully. And I believe this is really a resource limit problem. Thank you again for your advice!

DSchreyer commented 1 year ago

That is wonderful. Glad to hear that it was resolved easily.