search

nf-core / sarek

Analysis pipeline to detect germline or somatic variants (pre-processing, variant calling and annotation) from WGS / targeted sequencing
https://nf-co.re/sarek
MIT License
395 stars 402 forks source link

Sarek stops and cannot connect to amazonaws #1295

Open vmelichar opened 11 months ago

vmelichar commented 11 months ago

Description of the bug

As seen in the attached log file, the pipeline stops working and cycles in error with amazonaws. I am new to nf-core so I am unfamiliar with how all the parts of the pipeline work. Did this happen to anyone else?

Command used and terminal output

nextflow run nf-core/sarek --input fullpath/samplesheet.csv --outdir fullpath/results --tools freebayes,mutect2,strelka,snpeff,vep,merge -profile singularity --max_cpus 30 --step mapping --igenomes_base fullpath/references --use_annotation_cache_key

executor >  local (6977)
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:BWAMEM1_INDEX                                                                                 -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:BWAMEM2_INDEX                                                                                 -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:DRAGMAP_HASHTABLE                                                                             -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:GATK4_CREATESEQUENCEDICTIONARY                                                                -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:MSISENSORPRO_SCAN                                                                             -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:SAMTOOLS_FAIDX                                                                                -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:TABIX_DBSNP                                                                                   -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:TABIX_GERMLINE_RESOURCE                                                                       -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:TABIX_KNOWN_SNPS                                                                              -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:TABIX_KNOWN_INDELS                                                                            -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:TABIX_PON                                                                                     -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_ALLELES                                                                                 -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_LOCI                                                                                    -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_GC                                                                                      -
[-        ] process > NFCORE_SAREK:SAREK:PREPARE_GENOME:UNZIP_RT                                                                                      -
[d3/8aa26a] process > NFCORE_SAREK:SAREK:PREPARE_INTERVALS:CREATE_INTERVALS_BED (wgs_calling_regions_noseconds.hg38.bed)                              [100%] 1 of 1 ✔
[9a/080ce7] process > NFCORE_SAREK:SAREK:PREPARE_INTERVALS:TABIX_BGZIPTABIX_INTERVAL_SPLIT (chr6_95070791-167591393)                                  [100%] 21 of 21 ✔
[e0/0f1bfc] process > NFCORE_SAREK:SAREK:PREPARE_INTERVALS:TABIX_BGZIPTABIX_INTERVAL_COMBINED (wgs_calling_regions_noseconds.hg38)                    [100%] 1 of 1 ✔
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:SAMTOOLS_VIEW_MAP_MAP                                                                    -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:SAMTOOLS_VIEW_UNMAP_UNMAP                                                                -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:SAMTOOLS_VIEW_UNMAP_MAP                                                                  -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:SAMTOOLS_VIEW_MAP_UNMAP                                                                  -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:SAMTOOLS_MERGE_UNMAP                                                                     -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:COLLATE_FASTQ_UNMAP                                                                      -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:COLLATE_FASTQ_MAP                                                                        -
[-        ] process > NFCORE_SAREK:SAREK:CONVERT_FASTQ_INPUT:CAT_FASTQ                                                                                -
[58/65590e] process > NFCORE_SAREK:SAREK:FASTQC (R198-a)                                                                                              [100%] 38 of 38 ✔
[dc/9d8f01] process > NFCORE_SAREK:SAREK:FASTP (R15-a)                                                                                                [100%] 38 of 38 ✔
[bd/ab3cba] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM (R15-a)                                                 [100%] 456 of 456 ✔
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM2_MEM                                                         -
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:DRAGMAP_ALIGN                                                       -
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:SENTIEON_BWAMEM                                                     -
[43/e11fc4] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES (R212)                                                               [100%] 38 of 38 ✔
[1f/6884db] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:CRAM_QC_MOSDEPTH_SAMTOOLS:SAMTOOLS_STATS (R212)                                           [100%] 38 of 38 ✔
[e2/10b8d7] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:CRAM_QC_MOSDEPTH_SAMTOOLS:MOSDEPTH (R212)                                                 [100%] 38 of 38 ✔
[-        ] process > NFCORE_SAREK:SAREK:CRAM_TO_BAM                                                                                                  -
[59/02d181] process > NFCORE_SAREK:SAREK:BAM_BASERECALIBRATOR:GATK4_BASERECALIBRATOR (R212)                                                           [100%] 798 of 798 ✔
[f9/ff8a81] process > NFCORE_SAREK:SAREK:BAM_BASERECALIBRATOR:GATK4_GATHERBQSRREPORTS (R212)                                                          [100%] 38 of 38 ✔
[f8/f3d9b1] process > NFCORE_SAREK:SAREK:BAM_APPLYBQSR:GATK4_APPLYBQSR (R212)                                                                         [100%] 798 of 798 ✔
[7d/51e68d] process > NFCORE_SAREK:SAREK:BAM_APPLYBQSR:CRAM_MERGE_INDEX_SAMTOOLS:MERGE_CRAM (R212)                                                    [100%] 38 of 38 ✔
[fe/f6f190] process > NFCORE_SAREK:SAREK:BAM_APPLYBQSR:CRAM_MERGE_INDEX_SAMTOOLS:INDEX_CRAM (R212)                                                    [100%] 38 of 38 ✔
[61/f96072] process > NFCORE_SAREK:SAREK:CRAM_QC_RECAL:SAMTOOLS_STATS (R212)                                                                          [100%] 38 of 38 ✔
[f1/8d2956] process > NFCORE_SAREK:SAREK:CRAM_QC_RECAL:MOSDEPTH (R212)                                                                                [100%] 38 of 38 ✔
[-        ] process > NFCORE_SAREK:SAREK:CRAM_TO_BAM_RECAL                                                                                            -
[63/574e87] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_FREEBAYES:FREEBAYES (R83)                               [100%] 399 of 399 ✔
[be/39f200] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_FREEBAYES:BCFTOOLS_SORT (R83)                           [100%] 399 of 399 ✔
[2d/1c2ce6] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_FREEBAYES:MERGE_FREEBAYES (R83)                         [100%] 19 of 19 ✔
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_FREEBAYES:TABIX_VC_FREEBAYES                            -
[8b/8b43ef] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:STRELKA_SINGLE (R83)                     [100%] 399 of 399 ✔
[d7/220eb4] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERGE_STRELKA (R83)                      [100%] 19 of 19 ✔
[cb/ad1f55] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERGE_STRELKA_GENOME (R83)               [100%] 19 of 19 ✔
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_FREEBAYES:FREEBAYES                                   -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_FREEBAYES:BCFTOOLS_SORT                               -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_FREEBAYES:MERGE_FREEBAYES                             -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_FREEBAYES:TABIX_VC_FREEBAYES                          -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:MUTECT2                            -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:MERGE_MUTECT2                      -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:MERGEMUTECTSTATS                   -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:LEARNREADORIENTATIONMODEL          -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:GETPILEUPSUMMARIES                 -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:GATHERPILEUPSUMMARIES              -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:CALCULATECONTAMINATION             -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_TUMOR_ONLY_MUTECT2:FILTERMUTECTCALLS                  -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:STRELKA_SINGLE                         -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERGE_STRELKA                          -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERGE_STRELKA_GENOME                   -
[2e/47d601] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_FREEBAYES:FREEBAYES (R212_vs_R210)                       [100%] 399 of 399 ✔
[93/3c8e8c] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_FREEBAYES:BCFTOOLS_SORT (R212_vs_R210)                   [100%] 399 of 399 ✔
[63/74013b] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_FREEBAYES:MERGE_FREEBAYES (R212_vs_R210)                 [100%] 19 of 19 ✔
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_FREEBAYES:TABIX_VC_FREEBAYES                             -
[0b/9cf2ed] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_STRELKA:STRELKA_SOMATIC (R212_vs_R210)           [100%] 399 of 399 ✔
[d3/ed61e0] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_STRELKA:MERGE_STRELKA_INDELS (R212_vs_R210)      [100%] 19 of 19 ✔
[bd/0306e8] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_STRELKA:MERGE_STRELKA_SNVS (R212_vs_R210)        [100%] 19 of 19 ✔
[57/d6f8c8] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (R212_vs_R210)            [100%] 399 of 399 ✔
[01/72d767] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MERGE_MUTECT2 (R212_vs_R210)             [100%] 19 of 19 ✔
[73/b9901e] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MERGEMUTECTSTATS (R212_vs_R210)          [100%] 19 of 19 ✔
[82/bffe19] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:LEARNREADORIENTATIONMODEL (R212_vs_R210) [100%] 19 of 19 ✔
[8e/30c650] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:GETPILEUPSUMMARIES_NORMAL (R210)         [100%] 399 of 399 ✔
[7d/f5a375] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:GETPILEUPSUMMARIES_TUMOR (R212)          [100%] 399 of 399 ✔
[f5/6e564f] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:GATHERPILEUPSUMMARIES_NORMAL (R210)      [100%] 19 of 19 ✔
[fb/102294] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:GATHERPILEUPSUMMARIES_TUMOR (R212)       [100%] 19 of 19 ✔
[0e/63a4ca] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:CALCULATECONTAMINATION (R212_vs_R210)    [100%] 19 of 19 ✔
[a9/da52fb] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:FILTERMUTECTCALLS (R212_vs_R210)         [100%] 19 of 19 ✔
[42/6f01f9] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:BCFTOOLS_STATS (R212_vs_R210)                                                       [100%] 114 of 114 ✔
[43/218914] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:VCFTOOLS_TSTV_COUNT (R212_vs_R210)                                                  [100%] 114 of 114 ✔
[9f/57ab17] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:VCFTOOLS_TSTV_QUAL (R212_vs_R210)                                                   [100%] 114 of 114 ✔
[83/e2f11b] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:VCFTOOLS_SUMMARY (R212_vs_R210)                                                     [100%] 114 of 114 ✔
[8d/b7c4ab] process > NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_SNPEFF:SNPEFF_SNPEFF (R15_vs_R16)                                              [100%] 114 of 114 ✔
[d0/ea632a] process > NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_SNPEFF:TABIX_BGZIPTABIX (R15_vs_R16)                                           [100%] 114 of 114 ✔
[-        ] process > NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_MERGE:ENSEMBLVEP_VEP                                                           -
[-        ] process > NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_MERGE:TABIX_TABIX                                                              -
[-        ] process > NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_ENSEMBLVEP:ENSEMBLVEP_VEP                                                      -
[-        ] process > NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_ENSEMBLVEP:TABIX_TABIX                                                         -
[-        ] process > NFCORE_SAREK:SAREK:CUSTOM_DUMPSOFTWAREVERSIONS                                                                                  -
[-        ] process > NFCORE_SAREK:SAREK:MULTIQC

Relevant files

.nextflow.log

System information

nextflow 23.04.2 nf-core/sarek 3.3.2 System: Linux Container: singularity

FriederikeHanssen commented 11 months ago

Hi! We are you running your pipeline? It could help to predownload all images using nf-core download