Error executing process > 'bwa '

More92 commented 2 years ago

hi， I am a novice with eager2. I'm trying to run captured sample, I got the following error： The full error message was: Error executing process > 'bwa (SAMPLE)'

Caused by: Process exceeded running time limit (4h)

Command executed: bwa aln ....

How can I do? Im using singularity profile and the nf-core/eager version is v2.4.2 Could you give me some suggestions about this error thanks Stefania

TCLamnidis commented 2 years ago

Hi Stefania,

The job ran for more than 4h (the walltime specified for the job), so the executor of your cluster killed the job.

Could you post the command you used to lauch your eager job?

Best, Thiseas

More92 commented 2 years ago

Hi Thiseas, Absolutely yes! nextflow run nf-core/eager -profile singularity -resume eager2_catt_01_04 --input '/home/student4/EAGER_Catturati/INPUT/_S4_R{1,2}_001.fastq.gz' --udg_type half --single_stranded false --colour_chemistry 2 --bam false --snpcapture_bed /home/student4/SNP_list_ancestry_GRCh37.p13.bed --fasta /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta --bwa_index /home/share/public/hg_19_dbSNP_1-4-19/ --fasta_index /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta.fai --seq_dict /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.dict --outdir /home/student4/EAGER_Catturati/results -w /home/student4/EAGER_Catturati/INPUT/work --skip_collapse true --mapper bwaaln --bwaalnn 0.01 --bwaalnl 16500 --run_bam_filtering --bam_mapping_quality_threshold 30 --bam_unmapped_type bam --dedupper dedup --dedup_all_merged true --damageprofiler_length 100 --run_trim_bam false

thanks Stefania

TCLamnidis commented 2 years ago

Did the above error completely kill your run, or just one of the submitted jobs?

By default, eager will retry execution of failed jobs up to 3 times with increasing resources and wall time before giving up and failing an entire run.

Are you running this on a cluster?

If so, it looks like you not loading any cluster profile, or one does not exist for your cluster yet. These profiles generally describe the cluster you are working in, and let nextflow optimise resource allocation for submitted jobs.

If you are running locally, then you can create a config file to increase the walltime and/or resources allocated to different jobs to custom values instead.

If you need help setting up any of the profiles/config files mentioned above, you can join us in the nf-core slack workspace. We are happy to help.

More92 commented 2 years ago

The error completeley kill my run Yes, Im running this on a cluster but i not created a config file! I try to do it thanks, for your help! Stefania

jfy133 commented 2 years ago

This may also help (you don't have to make an official nf-core one)

https://nf-co.re/usage/tutorials/step_by_step_institutional_profile

More92 commented 2 years ago

I've realized that the error that completely killed my run was associated to only one of my samples. So I ran again my samples, without the one that was causing problems, and everything worked well. But, as expected, if I run the 'damaged' sample alone I get the same error and the run dies. Should I consider the problem to be in the sample or should I need to create the configuration file anyways? thanks, for your help! Stefania

jfy133 commented 2 years ago

Hey Stefania,

If you're just getting the 'out of time' error, the problem is unlikely the sample - you just may have large amount of data or high endogenous.

A minimal config file you could try would be, in a my_custom.config file, add the following

profiles {
  long_bwa {
     process {
       withName: bwa {
        time = 24.h
      }
    }
  }
}

Or something like that

Then supply to your nf-core/eager command with something along the lines of

nextflow run nf-core/eager -r <X.X.X> -c my_custom.config -profile <conda,docker,singularity>,long_bwa <rest of your parameters>

More92 commented 2 years ago

Hi, I tried as you told to me, but the error persists! I think that my_custom.config file isn't correct. I created my_custom.config such as test file with your suggestions, you think i must write other command? Sorry but i didn't use a config file thanks, for your help! Stefania

jfy133 commented 2 years ago

Hi, I tried as you told to me, but the error persists! I think that my_custom.config file isn't correct. I created my_custom.config such as test file with your suggestions, you think i must write other command? Sorry but i didn't use a config file thanks, for your help! Stefania

Hi Stefania,

you need to give us more information when reporting an issue.

Please post the new command you tried and also the config file you made.

Can you also post the .nextflow.log file.

More92 commented 2 years ago

hi, yes of course! this is the pipeline report with the error and the new command.

####################################################

nf-core/eager execution completed unsuccessfully!

#################################################### The exit status of the task that caused the workflow execution to fail was: null. The full error message was:

Error executing process > 'bwa (C-178-C2_S46_R)'

Caused by: Process exceeded running time limit (4h)

Command executed:

bwa aln -t 4 hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta C-178-C2_S46_R1_001.fastq_L0.pe.pair1.truncated.gz -n 0.01 -l 16500 -k 2 -o 2 -f C-178-C2_S46_R.r1.sai bwa aln -t 4 hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta C-178-C2_S46_R1_001.fastq_L0.pe.pair2.truncated.gz -n 0.01 -l 16500 -k 2 -o 2 -f C-178-C2_S46_R.r2.sai bwa sampe -r "@RG\tID:ILLUMINA-C-178-C2_S46_R\tSM:C-178-C2_S46_R\tPL:illumina\tPU:ILLUMINA-C-178-C2_S46_R-PE" hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta C-178-C2_S46_R.r1.sai C-178-C2_S46_R.r2.sai C-178-C2_S46_R1_001.fastq_L0.pe.pair1.truncated.gz C-178-C2_S46_R1_001.fastq_L0.pe.pair2.truncated.gz | samtools sort -@ 3 -O bam - > C-178-C2_S46R"PE".mapped.bam samtools index "C-178-C2_S46R""PE".mapped.bam

Command exit status:

Command output: (empty)

Command error: [bwa_aln] 176bp reads: max_diff = 9 [bwa_aln] 207bp reads: max_diff = 10 [bwa_aln] 239bp reads: max_diff = 11 [bwa_aln_core] calculate SA coordinate... 4421.66 sec [bwa_aln_core] write to the disk... 0.08 sec [bwa_aln_core] 262144 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4571.36 sec [bwa_aln_core] write to the disk... 0.07 sec [bwa_aln_core] 524288 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4568.74 sec [bwa_aln_core] write to the disk... 0.07 sec [bwa_aln_core] 786432 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4637.99 sec [bwa_aln_core] write to the disk... 0.07 sec [bwa_aln_core] 1048576 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4655.33 sec [bwa_aln_core] write to the disk... 0.07 sec [bwa_aln_core] 1310720 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4595.60 sec [bwa_aln_core] write to the disk... 0.07 sec [bwa_aln_core] 1572864 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4716.88 sec [bwa_aln_core] write to the disk... 0.07 sec [bwa_aln_core] 1835008 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 204.92 sec [bwa_aln_core] write to the disk... 0.00 sec [bwa_aln_core] 1845411 sequences have been processed. [main] Version: 0.7.17-r1188 [main] CMD: bwa aln -t 4 -n 0.01 -l 16500 -k 2 -o 2 -f C-178-C2_S46_R.r1.sai hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta C-178-C2_S46_R1_001.fastq_L0.pe.pair1.truncated.gz [main] Real time: 9542.303 sec; CPU: 32378.976 sec [bwa_aln] 17bp reads: max_diff = 2 [bwa_aln] 22bp reads: max_diff = 3 [bwa_aln] 42bp reads: max_diff = 4 [bwa_aln] 64bp reads: max_diff = 5 [bwa_aln] 90bp reads: max_diff = 6 [bwa_aln] 117bp reads: max_diff = 7 [bwa_aln] 146bp reads: max_diff = 8 [bwa_aln] 176bp reads: max_diff = 9 [bwa_aln] 207bp reads: max_diff = 10 [bwa_aln] 239bp reads: max_diff = 11 [bwa_aln_core] calculate SA coordinate... 4200.04 sec [bwa_aln_core] write to the disk... 0.08 sec [bwa_aln_core] 262144 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4229.88 sec [bwa_aln_core] write to the disk... 0.08 sec [bwa_aln_core] 524288 sequences have been processed. [bwa_aln_core] calculate SA coordinate... 4212.75 sec [bwa_aln_core] write to the disk... 0.08 sec [bwa_aln_core] 786432 sequences have been processed. [bwa_aln_core] calculate SA coordinate...

Work dir: /home/student4/Eager2_178_2/INPUT/work/6b/b6e2c56967406519951d0f1b38ffa7

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

The workflow was completed at 2022-04-08T18:38:49.181+02:00 (duration: 4h 11s)

The command used to launch the workflow was as follows:

nextflow run nf-core/eager -r dev -c /home/student4/Desktop/my_custom.config -profile singularity -resume catt_178 --input '/home/student4/Eager2_178_2/Input/C-178-C2_S46_R{1,2}_001.fastq.gz' --udg_type half --single_stranded false --colour_chemistry 2 --bam false --snpcapture_bed /home/student4/SNP_list_ancestry_GRCh37.p13.bed --fasta /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta --bwa_index /home/share/public/hg_19_dbSNP_1-4-19/ --fasta_index /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta.fai --seq_dict /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.dict --outdir /home/student4/Eager2_178_2/results -w /home/student4/Eager2_178_2/INPUT/work --skip_collapse true --mapper bwaaln --bwaalnn 0.01 --bwaalnl 16500 --run_bam_filtering --bam_mapping_quality_threshold 30 --bam_unmapped_type bam --dedupper dedup --dedup_all_merged true --damageprofiler_length 100 --run_trim_bam false

Pipeline Configuration:

Pipeline Release: dev
Run Name: stupefied_joliot
Input: /home/student4/Eager2_178_2/Input/C-178-C2_S46_R{1,2}_001.fastq.gz
Fasta Ref: /home/share/public/hg_19_dbSNP_1-4-19/Homo_sapiens_assembly19.fasta
Max Resources: 128 GB memory, 16 cpus, 10d time per job
Container: singularity - nfcore/eager:dev
Output dir: /home/student4/Eager2_178_2/results
Launch dir: /home/student4/Desktop
Working dir: /home/student4/Eager2_178_2/INPUT/work
Script dir: /home/student4/.nextflow/assets/nf-core/eager
User: student4
Config Profile: singularity
Config Files: /home/student4/.nextflow/assets/nf-core/eager/nextflow.config, /home/student4/Desktop/my_custom.config
Date Started: 2022-04-08T14:38:38.360+02:00
Date Completed: 2022-04-08T18:38:49.181+02:00
Pipeline script file path: /home/student4/.nextflow/assets/nf-core/eager/main.nf
Pipeline script hash ID: 5817cc8d9df4f5a571d6a974f455e82f
Pipeline repository Git URL: https://github.com/nf-core/eager
Pipeline repository Git Commit: d447ac4e0285b6a4979aa8b0549250968ec2f6ce
Pipeline Git branch/tag: dev
Nextflow Version: 21.10.3
Nextflow Build: 5655
Nextflow Compile Timestamp: 23-11-2021 07:29 UTC ################################ this the my config.file singularity { withLabel: bwa { time = 24.h } }

thanks, for your help! Stefania

jfy133 commented 2 years ago

Hey Stefania,

Thanks!

Could you try again but how I described above, with the profiles block in your config file, and also calling the profile within your command.

Note I did make a mistake the first time with the example with the link, however I've edited the example to be correct so please try that again.

Note your current config file for some reason puts bwa into a singularity block, however the singularity scope block only affects singularity container settings, NOT the tool. You must make sure the withLabel block goes into process (as this then affects the bwa job)

jfy133 commented 2 years ago

I'm going to close this as it's not a bug - please ask on slack if you need more help :)

nf-core / eager