nf-core / mag

Assembly and binning of metagenomes
https://nf-co.re/mag
MIT License
217 stars 110 forks source link

Spades assembly failed with the error code 250 #429

Closed mathavanpu closed 1 year ago

mathavanpu commented 1 year ago

Description of the bug

Hi there, I am trying to run the mag (2.3.0)pipeline, it got failed with the code 250.

I tried to change spdes cpu values 10 to 100 and memory 64 to 164GB in the base.config file but while running mag it throws an error . uncomitted changes nfcore. Kindly help how to change number of cpu in the config file.

Command used and terminal output

## nf-core/mag execution completed unsuccessfully! ##
####################################################
The exit status of the task that caused the workflow execution to fail was: 250.
The full error message was:

Error executing process > 'NFCORE_MAG:MAG:SPADES (CookingOil-1_Trimmed_R)'

Caused by:
  Process `NFCORE_MAG:MAG:SPADES (CookingOil-1_Trimmed_R)` terminated with an error exit status (250)

Command executed:

  metaspades.py             --only-assembler             --threads "16"             --memory 128             --pe1-1 CookingOil-1_Trimmed_R.phix_removed.unmapped_1.fastq.gz             --pe1-2 CookingOil-1_Trimmed_R.phix_removed.unmapped_2.fastq.gz             -o spades
  mv spades/assembly_graph_with_scaffolds.gfa SPAdes-CookingOil-1_Trimmed_R_graph.gfa
  mv spades/scaffolds.fasta SPAdes-CookingOil-1_Trimmed_R_scaffolds.fasta
  mv spades/contigs.fasta SPAdes-CookingOil-1_Trimmed_R_contigs.fasta
  mv spades/spades.log SPAdes-CookingOil-1_Trimmed_R.log
  gzip "SPAdes-CookingOil-1_Trimmed_R_contigs.fasta"
  gzip "SPAdes-CookingOil-1_Trimmed_R_graph.gfa"
  gzip -c "SPAdes-CookingOil-1_Trimmed_R_scaffolds.fasta" > "SPAdes-CookingOil-1_Trimmed_R_scaffolds.fasta.gz"

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAG:MAG:SPADES":
      python: $(python --version 2>&1 | sed 's/Python //g')
      metaspades: $(metaspades.py --version | sed "s/SPAdes genome assembler v//; s/ \[.*//")
  END_VERSIONS

Command exit status:
  250

Command output:
    1:20:18.270    35G / 43G   INFO    General                 (pair_info_count.cpp       : 390)   Filtering data for library #0
    1:20:18.295    35G / 43G   INFO    General                 (pair_info_count.cpp       :  48)   Selecting usual mapper
    1:21:00.583    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 200000 reads
    1:21:01.201    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 400000 reads
    1:21:01.424    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 600000 reads
    1:21:01.793    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 800000 reads
    1:21:09.940    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1000000 reads
    1:21:10.399    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1200000 reads
    1:21:15.076    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 2200000 reads
    1:22:03.760    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 4200000 reads
    1:23:14.626    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 8400000 reads
    1:25:16.003    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 16800000 reads
    1:28:50.653    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  95)   Total 27798810 reads processed
    1:28:50.686    35G / 43G   INFO    General                 (pair_info_count.cpp       : 402)   Mapping library #0
    1:28:50.702    35G / 43G   INFO    General                 (pair_info_count.cpp       : 404)   Mapping paired reads (takes a while)
    1:28:50.718    35G / 43G   INFO    General                 (pair_info_count.cpp       : 299)   Left insert size quantile 159, right insert size quantile 371, filtering threshold 1, rounding threshold 0
    1:28:50.742    35G / 43G   INFO    General                 (pair_info_count.cpp       :  48)   Selecting usual mapper
    1:30:49.205    36G / 48G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 200000 reads
    1:30:50.366    36G / 48G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 400000 reads
    1:30:51.339    37G / 48G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 600000 reads
    1:30:59.359    38G / 49G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 800000 reads
    1:31:01.226    39G / 49G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1000000 reads
    1:31:03.231    40G / 49G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1200000 reads
    1:31:07.031    41G / 50G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 2200000 reads
    1:32:54.558    41G / 57G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 4200000 reads
    1:35:07.768    43G / 64G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 8400000 reads
    1:40:02.002    46G / 78G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 16800000 reads
    1:46:02.964    88G / 90G   INFO    General                 (sequence_mapper_notifier.h:  95)   Total 27798810 reads processed
    1:46:09.380    85G / 90G   INFO   StageManager             (stage.cpp                 : 185)   STAGE == Preliminary Distance Estimation (id: distance_estimation_preliminary)
    1:46:09.445    85G / 90G   INFO    General                 (distance_estimation.cpp   : 179)   Processing library #0
    1:46:09.476    85G / 90G   INFO    General                 (distance_estimation.cpp   : 150)   Weight Filter Done
    1:46:09.506    85G / 90G   INFO   DistanceEstimator        (distance_estimation.hpp   : 116)   Using SIMPLE distance estimator
  mimalloc: error: unable to allocate OS memory (268435456 bytes, error code: 12 [Cannot allocate memory], address: (nil), large only: 0, allow large: 1)
  mimalloc: process info:
     elapsed:   10480.121 s
     process: user: 127496.175 s, system: 5372.646 s, faults: 0
         rss: current: 137221435392, peak: 135711707136
      commit: current: 137221435392, peak: 137221435392

  == Error ==  system call for: "['/usr/local/bin/spades-core', 'spades/K55/configs/config.info', 'spades/K55/configs/mda_mode.info', 'spades/K55/configs/meta_mode.info']" finished abnormally, OS return value: -6
  None

  In case you have troubles running SPAdes, you can write to spades.support@cab.spbu.ru
  or report an issue on our GitHub repository github.com/ablab/spades
  Please provide us with params.txt and spades.log files from the output directory.

  SPAdes log can be found here: spades/spades.log

  Thank you for using SPAdes!

Command error:
    1:20:18.270    35G / 43G   INFO    General                 (pair_info_count.cpp       : 390)   Filtering data for library #0
    1:20:18.295    35G / 43G   INFO    General                 (pair_info_count.cpp       :  48)   Selecting usual mapper
    1:21:00.583    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 200000 reads
    1:21:01.201    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 400000 reads
    1:21:01.424    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 600000 reads
    1:21:01.793    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 800000 reads
    1:21:09.940    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1000000 reads
    1:21:10.399    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1200000 reads
    1:21:15.076    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 2200000 reads
    1:22:03.760    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 4200000 reads
    1:23:14.626    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 8400000 reads
    1:25:16.003    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 16800000 reads
    1:28:50.653    35G / 43G   INFO    General                 (sequence_mapper_notifier.h:  95)   Total 27798810 reads processed
    1:28:50.686    35G / 43G   INFO    General                 (pair_info_count.cpp       : 402)   Mapping library #0
    1:28:50.702    35G / 43G   INFO    General                 (pair_info_count.cpp       : 404)   Mapping paired reads (takes a while)
    1:28:50.718    35G / 43G   INFO    General                 (pair_info_count.cpp       : 299)   Left insert size quantile 159, right insert size quantile 371, filtering threshold 1, rounding threshold 0
    1:28:50.742    35G / 43G   INFO    General                 (pair_info_count.cpp       :  48)   Selecting usual mapper
    1:30:49.205    36G / 48G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 200000 reads
    1:30:50.366    36G / 48G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 400000 reads
    1:30:51.339    37G / 48G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 600000 reads
    1:30:59.359    38G / 49G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 800000 reads
    1:31:01.226    39G / 49G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1000000 reads
    1:31:03.231    40G / 49G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 1200000 reads
    1:31:07.031    41G / 50G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 2200000 reads
    1:32:54.558    41G / 57G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 4200000 reads
    1:35:07.768    43G / 64G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 8400000 reads
    1:40:02.002    46G / 78G   INFO    General                 (sequence_mapper_notifier.h:  77)   Processed 16800000 reads
    1:46:02.964    88G / 90G   INFO    General                 (sequence_mapper_notifier.h:  95)   Total 27798810 reads processed
    1:46:09.380    85G / 90G   INFO   StageManager             (stage.cpp                 : 185)   STAGE == Preliminary Distance Estimation (id: distance_estimation_preliminary)
    1:46:09.445    85G / 90G   INFO    General                 (distance_estimation.cpp   : 179)   Processing library #0
    1:46:09.476    85G / 90G   INFO    General                 (distance_estimation.cpp   : 150)   Weight Filter Done
    1:46:09.506    85G / 90G   INFO   DistanceEstimator        (distance_estimation.hpp   : 116)   Using SIMPLE distance estimator
  mimalloc: error: unable to allocate OS memory (268435456 bytes, error code: 12 [Cannot allocate memory], address: (nil), large only: 0, allow large: 1)
  mimalloc: process info:
     elapsed:   10480.121 s
     process: user: 127496.175 s, system: 5372.646 s, faults: 0
         rss: current: 137221435392, peak: 135711707136
      commit: current: 137221435392, peak: 137221435392

  == Error ==  system call for: "['/usr/local/bin/spades-core', 'spades/K55/configs/config.info', 'spades/K55/configs/mda_mode.info', 'spades/K55/configs/meta_mode.info']" finished abnormally, OS return value: -6
  None

  In case you have troubles running SPAdes, you can write to spades.support@cab.spbu.ru
  or report an issue on our GitHub repository github.com/ablab/spades
  Please provide us with params.txt and spades.log files from the output directory.

  SPAdes log can be found here: spades/spades.log

  Thank you for using SPAdes!

Work dir:
  /goast/Work/WGS/LCA-WGS2-BY-859/mag_trail1/work/1e/fd1eb0013b2799fa09526a047b657a

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

The workflow was completed at 2023-04-28T19:05:02.282228060+05:30 (duration: 9h 16m 55s)

The command used to launch the workflow was as follows:

  nextflow run nf-core/mag -r 2.2.1 -bg -name Cooking -profile docker -params-file nf-params.json

Pipeline Configuration:

Relevant files

No response

System information

mag (2.3.0)pipeline

jfy133 commented 1 year ago

Hi @mathavanpu

You should not be modifying teh pipeline code!

To adjust resources, you need to use a custom config file - docs are here: https://nf-co.re/docs/usage/configuration#tuning-workflow-resources

If using a custom config still doesn't work, please feel free to re-open (with your config file attached)

mathavanpu commented 1 year ago

Hi, Thank you for the response Again I got an error I am not able to solve it.

I created the custom config files

Spades

process { withName: SPADES { cpus = 100 memory = 240.GB time = 16.h } }

Concot

process { withName: CONCOCT_CONCOCT { cpus = 2 memory = 12.GB time = 24.h } }

Command executed: nextflow run nf-core/mag -r 2.3.0 -bg -name Project_Meta_859 -profile docker -params-file nf-params.json -c concot.config spades.config

Error message The exit status of the task that caused the workflow execution to fail was: 250. The full error message was:

Error executing process > 'NFCORE_MAG:MAG:SPADES (CookingOil-1_Trimmed_R)'

Caused by: Process NFCORE_MAG:MAG:SPADES (CookingOil-1_Trimmed_R) terminated with an error exit status (250)

metaspades.py --only-assembler --threads "10" --memory 64 --pe1-1 CookingOil-1_Trimmed_R.phix_removed.unmapped_1.fastq.gz --pe1-2 CookingOil-1_Trimmed_R.phix_removed.unmapped_2.fastq.gz -o spades mv spades/assembly_graph_with_scaffolds.gfa SPAdes-CookingOil-1_Trimmed_R_graph.gfa mv spades/scaffolds.fasta SPAdes-CookingOil-1_Trimmed_R_scaffolds.fasta mv spades/contigs.fasta SPAdes-CookingOil-1_Trimmed_R_contigs.fasta mv spades/spades.log SPAdes-CookingOil-1_Trimmed_R.log gzip "SPAdes-CookingOil-1_Trimmed_R_contigs.fasta" gzip "SPAdes-CookingOil-1_Trimmed_R_graph.gfa" gzip -c "SPAdes-CookingOil-1_Trimmed_R_scaffolds.fasta" > "SPAdes-CookingOil-1_Trimmed_R_scaffolds.fasta.gz"

cat <<-END_VERSIONS > versions.yml "NFCORE_MAG:MAG:SPADES": python: $(python --version 2>&1 | sed 's/Python //g') metaspades: $(metaspades.py --version | sed "s/SPAdes genome assembler v//; s/ [.*//") END_VERSIONS

Command exit status: 250 The config file CPU values is not exhibited in the command metaspades.py --only-assembler --threads "10" --memory 64 --pe1-1 CookingOil-1_Trimmed_R.phix_removed.unmapped_1.fastq.gz --pe1-2 CookingOil-1_Trimmed_R.phix_removed.unmapped_2.fastq.gz -o spades

it took only 10 threads Kindly help

d4straub commented 1 year ago

-c concot.config spades.config doesnt work, every config needs a -c, use -c concot.config -c spades.config Such questions might be better asked in the nf-core slack channel #mag, see https://nf-co.re/join.