search

nf-core / hic

Analysis of Chromosome Conformation Capture data (Hi-C)
https://nf-co.re/hic
MIT License
80 stars 55 forks source link

Exit error 140 #167

Closed ninashenker closed 5 months ago

ninashenker commented 1 year ago

Description of the bug

Hi,

Great work on this pipeline!

I keep getting an exit 140 error for the bowtie alignment step. I've tried increasing the time as it seems all the failed jobs fail at 16 hours. I've tried adding a nextflow.config and running it as: nextflow run nf-core/hic --input sample_sheet.csv --outdir Arima_results_updated --genome hg38 -profile bigpurple --digestion arima -c nextflow.config

I've also tried to manually edit the configs/conf/bigpurple.config and the ./workflow/conf/base.config but still no luck. Any idea what could be causing the error and what is the correct way to edit the resource requirement?

Best, Nina

Command used and terminal output

Command: nextflow run nf-core/hic --input sample_sheet.csv --outdir Arima_results_updated --genome hg38 -profile bigpurple --digestion arima -resume

executor >  slurm (22)
[ae/43f96c] process > NFCORE_HIC:HIC:INPUT_CHECK:SAMPLESHEET_CHECK (sample_sheet.csv)                        [100%] 1 of 1, cached: 1 ✔
[f6/9a253d] process > NFCORE_HIC:HIC:PREPARE_GENOME:CUSTOM_GETCHROMSIZES (genome.fa)                         [100%] 1 of 1, cached: 1 ✔
[4b/d9e9ff] process > NFCORE_HIC:HIC:PREPARE_GENOME:GET_RESTRICTION_FRAGMENTS (^GATC,G^ANTC)                 [100%] 1 of 1, cached: 1 ✔
[01/1f4254] process > NFCORE_HIC:HIC:FASTQC (Chordoma-0TS2033219-Arima)                                      [100%] 46 of 46, cached: 46 ✔
[7a/c4be2e] process > NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN (Chordoma-0TS2028097-Arima)         [ 95%] 46 of 48, cached: 30, failed: 12
[9f/55218f] process > NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:TRIM_READS (Chordoma-0TS2022608-Arima)            [100%] 30 of 30, cached: 30
[59/15bac6] process > NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN_TRIMMED (Chordoma-0TS1842376-Arima) [100%] 30 of 30, cached: 29
[12/0e1f10] process > NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:MERGE_BOWTIE2 (Chordoma-0TS1842367-Arima)         [100%] 29 of 29, cached: 24 ✔
[-        ] process > NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:COMBINE_MATES                                     [  0%] 0 of 2
[-        ] process > NFCORE_HIC:HIC:HICPRO:GET_VALID_INTERACTION                                            -
[-        ] process > NFCORE_HIC:HIC:HICPRO:MERGE_VALID_INTERACTION                                          -
[-        ] process > NFCORE_HIC:HIC:HICPRO:MERGE_STATS                                                      [  0%] 0 of 2
[-        ] process > NFCORE_HIC:HIC:HICPRO:HICPRO2PAIRS                                                     -
[89/2476c9] process > NFCORE_HIC:HIC:COOLER:COOLER_MAKEBINS (null})                                          [100%] 3 of 3, cached: 3 ✔
[-        ] process > NFCORE_HIC:HIC:COOLER:COOLER_CLOAD                                                     -
[-        ] process > NFCORE_HIC:HIC:COOLER:COOLER_BALANCE                                                   -
[-        ] process > NFCORE_HIC:HIC:COOLER:COOLER_ZOOMIFY                                                   -
[-        ] process > NFCORE_HIC:HIC:COOLER:COOLER_DUMP                                                      -
[-        ] process > NFCORE_HIC:HIC:COOLER:SPLIT_COOLER_DUMP                                                -
[-        ] process > NFCORE_HIC:HIC:HIC_PLOT_DIST_VS_COUNTS                                                 -
[-        ] process > NFCORE_HIC:HIC:COMPARTMENTS:COOLTOOLS_EIGSCIS                                          -
[-        ] process > NFCORE_HIC:HIC:TADS:COOLTOOLS_INSULATION                                               -
[-        ] process > NFCORE_HIC:HIC:CUSTOM_DUMPSOFTWAREVERSIONS                                             -
[-        ] process > NFCORE_HIC:HIC:MULTIQC                                                                 -
-[nf-core/hic] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN (Chordoma-0TS2033219-Arima)'
Command executed:

  INDEX=`find -L ./ -name "*.rev.1.bt2" | sed "s/\.rev.1.bt2$//"`
  [ -z "$INDEX" ] && INDEX=`find -L ./ -name "*.rev.1.bt2l" | sed "s/\.rev.1.bt2l$//"`
  [ -z "$INDEX" ] && echo "Bowtie2 index files not found" 1>&2 && exit 1

  bowtie2 \
      -x $INDEX \
      -U TS20-33219_rep2_R1.fastq.gz \
      --threads 12 \
      --un-gz Chordoma-0TS2033219-Arima_1_R1.unmapped.fastq.gz \
      --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder \
      2> Chordoma-0TS2033219-Arima_1_R1.bowtie2.log \
      | samtools view -F 4 --threads 12 -o Chordoma-0TS2033219-Arima_1_R1.bam -

  if [ -f Chordoma-0TS2033219-Arima_1_R1.unmapped.fastq.1.gz ]; then
      mv Chordoma-0TS2033219-Arima_1_R1.unmapped.fastq.1.gz Chordoma-0TS2033219-Arima_1_R1.unmapped_1.fastq.gz
  fi

  if [ -f Chordoma-0TS2033219-Arima_1_R1.unmapped.fastq.2.gz ]; then
      mv Chordoma-0TS2033219-Arima_1_R1.unmapped.fastq.2.gz Chordoma-0TS2033219-Arima_1_R1.unmapped_2.fastq.gz
  fi

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_HIC:HIC:HICPRO:HICPRO_MAPPING:BOWTIE2_ALIGN":
      bowtie2: $(echo $(bowtie2 --version 2>&1) | sed 's/^.*bowtie2-align-s version //; s/ .*$//')
Command exit status:
  140

Command output:
  (empty)

Work dir:
  /gpfs/data/abl/home/shenkn01/Chordoma/nfcore/nf-core-hic-2.0.0/work/7d/b188ee8f8a1954c53d71727985ccc8

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Relevant files

nextflow_hic.zip

System information

Nextflow = 23.01.1 Java = 17.0.2 Container = Singularity: 3.1 Hardware = HPC Executor - SLURM (bigpurple)

Package Version

attmap 0.13.2 attrs 23.1.0 cattrs 22.2.0 certifi 2023.5.7 cfgv 3.3.1 charset-normalizer 3.1.0 click 8.1.3 distlib 0.3.6 exceptiongroup 1.1.1 filelock 3.12.0 filetype 1.2.0 future 0.18.3 gitdb 4.0.10 GitPython 3.1.31 identify 2.5.24 idna 3.4 importlib-metadata 6.6.0 iniconfig 2.0.0 Jinja2 3.1.2 jsonschema 4.17.3 logmuse 0.2.7 Markdown 3.4.3 markdown-it-py 2.2.0 MarkupSafe 2.1.2 mdurl 0.1.2 nextflow 23.4.1 nf-core 2.8 nodeenv 1.7.0 numpy 1.24.3 oyaml 1.0 packaging 23.1 pandas 2.0.1 pip 23.0.1 piper 0.12.3 platformdirs 3.5.0 pluggy 1.0.0 pre-commit 3.3.1 prompt-toolkit 3.0.38 psutil 5.9.5 pyfaidx 0.7.2.1 Pygments 2.15.1 pyrsistent 0.19.3 pytest 7.3.1 pytest-workflow 2.0.1 python-dateutil 2.8.2 pytz 2023.3 PyYAML 6.0 questionary 1.10.0 refgenconf 0.12.2 refgenie 0.12.1 requests 2.30.0 requests-cache 1.0.1 rich 13.3.5 rich-click 1.6.1 setuptools 66.0.0 six 1.16.0 smmap 5.0.0 tabulate 0.9.0 tomli 2.0.1 tqdm 4.65.0 tzdata 2023.3 ubiquerg 0.6.2 url-normalize 1.4.3 urllib3 2.0.2 virtualenv 20.23.0 wcwidth 0.2.6 wheel 0.38.4 yacman 0.9.0 zipp 3.15.0

ninashenker commented 1 year ago

Update: I used the suggestion I was given here https://github.com/nf-core/hic/issues/158 which worked nicely but some of the samples are still failing to complete the bowtie alignment step within 48 hrs - is there anyway to shorten this alignment step/is it normal to take this long?

nservant commented 1 year ago

Hi, this mainly depands on the number of reads you have in input. I would suggest to look at the --split_fastq option, to work per chunk.