search

nf-core / sarek

Analysis pipeline to detect germline or somatic variants (pre-processing, variant calling and annotation) from WGS / targeted sequencing
https://nf-co.re/sarek
MIT License
351 stars 387 forks source link

ERROR: terminated for an unknown reason -- Likely it has been terminated by the external system #1133

Open amnghn opened 1 year ago

amnghn commented 1 year ago

Description of the bug

I was running Sarek for about 100 samples, and I ran out of disk quota at some point. After free up more space on the cluster (Uppmax), I -resume the previous run, but I got this error message for several samples, and the pipeline shot down.

error [nextflow.exception.ProcessFailedException]: ProcessNFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_MPILEUP:BCFTOOLS_MPILEUP (sample_ID)terminated for an unknown reason -- Likely it has been terminated by the external system

Command used and terminal output

nextflow run nf-core/sarek -r 3.2.2 -resume -profile uppmax -params-file nf-params.json --project naissXXXX-X-XXX

Relevant files

[username@rackham4 b747de569aa5ee9e086c3234d3d971]$ cat .command.log INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred INFO: Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid [mpileup] 1 samples in 1 input files [mpileup] maximum number of reads per input file set to -d 250 slurmstepd: error: JOB 38839689 ON r151 CANCELLED AT 2023-06-22T13:18:59

System information

HPC: Uppmax Congfig: -profile uppmax

FriederikeHanssen commented 11 months ago

Hi! Sorry for the late reply, looks like slurm terminated your job. I am guessing you need to assign more time to it with a custom config: https://nf-co.re/docs/usage/troubleshooting#my-pipeline-crashes-part-way-through-a-run-at-a-certain-step-with-a-non-0-exit-code