search

cancerit / dockstore-cgpwgs

Dockstore implementation of CGP core WGS analysis
GNU Affero General Public License v3.0
30 stars 14 forks source link

Wrapper error when running cgpwgs-1.1.5 with Singularity #65

Closed mjakobs closed 3 years ago

mjakobs commented 3 years ago

Hi there, I'm trying to run cgpwgs-1.1.5 using singularity and I keep running into the same error associated with the CaVEMan wrapper and the alleleCount wrapper.

Prior to running cgpwgs-1.1.5 I processed my samples using cgpmap-2.0.3.

The code I used to run cgpwgs-1.1.5 is:

singularity exec -i --bind \
pipelines/Sanger_hg37/Scripts_for_testing_Sanger_hg37:/mnt/in/,pipelines/Sanger_hg37/Scripts_for_testing_Sanger_hg37/output:/mnt/out,pipelines/Sanger_hg37/cgpwgs_ref_grch37:/mnt/reference \
 --workdir /path/to/workdir/ \
 --home pipelines/Sanger_hg37/Scripts_for_testing_Sanger_hg37:/home \
 pipelines/Sanger_hg37/dockstore-cgpwgs_1.1.5.sif ds-wrapper.pl \
 -reference="/mnt/reference/core_ref_GRCh37d5.tar.gz" \
 -annot="/mnt/reference/VAGrENT_ref_GRCh37d5_ensembl_75.tar.gz" \
 -snv_indel="/mnt/reference/SNV_INDEL_ref_GRCh37d5.tar.gz" \
 -cnv_sv="/mnt/reference/CNV_SV_ref_GRCh37d5_brass6+.tar.gz" \
 -subcl="/mnt/reference/SUBCL_ref_GRCh37d5.tar.gz" \
 -tumour="/mnt/in/Test_G0481L09_S1_mapped.bam" -normal="/mnt/in/Test_G0481L09_S17_L004_mapped.bam" \
 -exclude="NC_007605,hs37d5,GL%" -species="human" \
 -assembly="GRCh37d5" \
 -cavereads="350000"

The output of the job is:

> Options loaded: 

> $VAR1 = {
>           'wt_sm' => 'e2e40072-f6c3-11eb-b585-e143ccb5f655',
>           'sp' => 'human',
>           'cr' => 350000,
>           'a' => '/mnt/reference/VAGrENT_ref_GRCh37d5_ensembl_75.tar.gz',
>           'r' => '/mnt/reference/core_ref_GRCh37d5.tar.gz',
>           'e' => 'NC_007605,hs37d5,GL%',
>           'as' => 'GRCh37d5',
>           'cs' => '/mnt/reference/CNV_SV_ref_GRCh37d5_brass6+.tar.gz',
>           'si' => '/mnt/reference/SNV_INDEL_ref_GRCh37d5.tar.gz',
>           'n' => '/mnt/in/Test_G0481L09_S17_L004_mapped.bam',
>           'pc' => 8,
>           't' => '/mnt/in/Test_G0481L09_S1_mapped.bam',
>           'mt_sm' => '3be90900-f6c5-11eb-b585-e143ccb5f655',
>           'sc' => '/mnt/reference/SUBCL_ref_GRCh37d5.tar.gz'
>         };
> 
> core_ref_GRCh37d5/genome.fa.fai
> core_ref_GRCh37d5/genome.fa.dict
> core_ref_GRCh37d5/genome.fa
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/vagrent.fa.fai
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/vagrent.cache.gz
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/vagrent.cache.gz.tbi
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/exon_regions.sub.bed.gz.tbi
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/exon_regions.indel.bed.gz
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/vagrent.fa
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/README.txt
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/codingexon_regions.sub.bed.gz.tbi
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/gene_regions.bed.gz
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/gene_regions.bed.gz.tbi
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/codingexon_regions.indel.bed.gz
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/exon_regions.sub.bed.gz
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/codingexon_regions.sub.bed.gz
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/codingexon_regions.indel.bed.gz.tbi
> VAGrENT_ref_GRCh37d5_ensembl_75/vagrent/exon_regions.indel.bed.gz.tbi
> SNV_INDEL_ref/pindel/
> SNV_INDEL_ref/pindel/WGS_Rules.lst
> SNV_INDEL_ref/pindel/pindel_np.gff3.gz.tbi
> SNV_INDEL_ref/pindel/pindel_np.gff3.gz
> SNV_INDEL_ref/pindel/WXS_Rules.lst
> SNV_INDEL_ref/pindel/HiDepth.bed.gz.tbi
> SNV_INDEL_ref/pindel/softRules.lst
> SNV_INDEL_ref/pindel/HiDepth.bed.gz
> SNV_INDEL_ref/pindel/simpleRepeats.bed.gz.tbi
> SNV_INDEL_ref/pindel/simpleRepeats.bed.gz
> SNV_INDEL_ref/caveman/
> SNV_INDEL_ref/caveman/flagging/
> SNV_INDEL_ref/caveman/flagging/snps.bed.gz.tbi
> SNV_INDEL_ref/caveman/flagging/flag.to.vcf.convert.ini
> SNV_INDEL_ref/caveman/flagging/centromeric_repeats.bed.gz.tbi
> SNV_INDEL_ref/caveman/flagging/simple_repeats.bed.gz.tbi
> SNV_INDEL_ref/caveman/flagging/flag.vcf.config.ini
> SNV_INDEL_ref/caveman/flagging/snps.bed.gz
> SNV_INDEL_ref/caveman/flagging/centromeric_repeats.bed.gz
> SNV_INDEL_ref/caveman/flagging/hi_seq_depth.bed.gz
> SNV_INDEL_ref/caveman/flagging/hi_seq_depth.bed.gz.tbi
> SNV_INDEL_ref/caveman/flagging/simple_repeats.bed.gz
> SNV_INDEL_ref/caveman/unmatchedNormal.bed.gz
> SNV_INDEL_ref/caveman/HiDepth.tsv
> SNV_INDEL_ref/caveman/flag.vcf.config.WGS.ini
> SNV_INDEL_ref/caveman/flag.vcf.config.WXS.ini
> SNV_INDEL_ref/caveman/unmatchedNormal.bed.gz.tbi
> CNV_SV_ref/ascat/
> CNV_SV_ref/ascat/README.txt
> CNV_SV_ref/ascat/SnpGcCorrections.tsv
> CNV_SV_ref/brass/
> CNV_SV_ref/brass/viral.genomic.fa.2bit
> CNV_SV_ref/brass/all_ncbi_bacteria.1.fa.2bit
> CNV_SV_ref/brass/brass_np.groups.gz.tbi
> CNV_SV_ref/brass/cytoband.txt
> CNV_SV_ref/brass/500bp_windows.gc.bed.gz
> CNV_SV_ref/brass/CentTelo.tsv
> CNV_SV_ref/brass/brass_np.groups.gz
> CNV_SV_ref/brass/README.txt
> CNV_SV_ref/brass/all_ncbi_bacteria.4.fa.2bit
> CNV_SV_ref/brass/all_ncbi_bacteria.3.fa.2bit
> CNV_SV_ref/brass/all_ncbi_bacteria.2.fa.2bit
> CNV_SV_ref/brass/HiDepth.bed.gz
> SUBCL/battenberg/
> SUBCL/battenberg/impute/
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chrX_PAR2_impute.legend
> SUBCL/battenberg/impute/impute_info.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr8_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr7_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr13_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr13_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr8_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr21_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr4_impute.legend
> SUBCL/battenberg/impute/genetic_map_chrX_PAR1_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr19_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr15_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr9_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr18_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr11_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr5_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chrX_PAR2_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr14_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr5_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr17_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr10_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr7_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr19_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr1_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr2_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr14_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr8_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr3_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr12_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chrX_nonPAR_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr13_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr12_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr19_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr17_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr12_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr4_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr10_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr9_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr11_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr16_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr20_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr22_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr5_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr3_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr16_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr2_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr20_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr6_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr22_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr18_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr14_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chrX_PAR1_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr4_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr1_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chrX_PAR2_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr15_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr1_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr10_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr6_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chrX_PAR1_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chrX_nonPAR_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr7_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr16_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr6_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr22_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr20_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr3_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr21_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr21_impute.legend
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr17_impute.legend
> SUBCL/battenberg/impute/genetic_map_chr11_combined_b37.txt
> SUBCL/battenberg/impute/genetic_map_chr15_combined_b37.txt
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chrX_nonPAR_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr18_impute.hap.gz
> SUBCL/battenberg/impute/ALL_1000G_phase1integrated_v3_chr2_impute.hap.gz
> SUBCL/battenberg/impute/genetic_map_chr9_combined_b37.txt
> SUBCL/battenberg/1000genomesloci/
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr14.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr16.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr2.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr10.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr17.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr4.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr6.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr22.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr3.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr18.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr21.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr13.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr8.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr20.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr19.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr23.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr9.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr12.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr7.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr11.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr5.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr15.txt
> SUBCL/battenberg/1000genomesloci/1000genomesloci2012_chr1.txt
> SUBCL/battenberg/ignore_contigs.txt
> SUBCL/battenberg/probloci.txt
> 
> Start workflow: Thu Aug 12 17:34:05 BST 2021
> 
> Loading user options from: /home/run.params
> PCAP_THREADED_LOADBACKOFF=1
> HOSTNAME=hnode010.pri.csf3.alces.network
> SGE_TASK_STEPSIZE=undefined
> SHELL=/bin/bash
> TERM=
> NHOSTS=1
> PCAP_THREADED_NO_SCRIPT=1
> SGE_O_WORKDIR=/path/to/workdir/pipelines/Sanger_hg37/Scripts_for_testing_Sanger_hg37
> TMPDIR=/tmp/2799758.1.serial.q
> PERL5LIB=/opt/wtsi-cgp/lib/perl5
> SGE_O_HOME=/path/to/my/homedir
> SGE_CELL=default
> SGE_ARCH=lx-amd64
> SYSTEMCTL_SKIP_REDIRECT=
> SINGULARITY_APPNAME=
> RESTARTED=0
> ARC=lx-amd64
> USER_PATH=/tmp/2799758.1.serial.q:/usr/local/bin:/bin:/usr/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
> USER=b05055gj
> LD_LIBRARY_PATH=/opt/wtsi-cgp/lib:/.singularity.d/libs
> SGE_TASK_LAST=undefined
> SINGULARITY_NAME=dockstore-cgpwgs_1.1.5.sif
> QUEUE=serial.q
> PCAP_THREADED_REM_LOGS=1
> PCAP_THREADED_FORCE_SYNC=1
> SGE_TASK_ID=undefined
> SGE_BINARY_PATH=/opt/site/sge/bin/lx-amd64
> PATH=/opt/wtsi-cgp/bin:/opt/wtsi-cgp/bin:/opt/wtsi-cgp/bin:/opt/wtsi-cgp/bin:/opt/wtsi-cgp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> SGE_STDERR_PATH=/path/to/workdir/pipelines/Sanger_hg37/Scripts_for_testing_Sanger_hg37//Step6_run_cgpwgs_20210812.sh.e2799758
> PWD=/home
> R_LIBS=/opt/wtsi-cgp/R-lib
> SGE_STDOUT_PATH=/path/to/workdir/pipelines/Sanger_hg37/Scripts_for_testing_Sanger_hg37//Step6_run_cgpwgs_20210812.sh.o2799758
> SGE_ACCOUNT=sge
> OPT=/opt/wtsi-cgp
> SGE_EXECD_PORT=6445
> SGE_QMASTER_PORT=6444
> JOB_NAME=Step6_run_cgpwgs_20210812.sh
> JOB_SCRIPT=/opt/site/sge/default/spool/hnode010/job_scripts/2799758
> SGE_ROOT=/opt/site/sge
> REQNAME=Step6_run_cgpwgs_20210812.sh
> SGE_JOB_SPOOL_DIR=/opt/site/sge/default/spool/hnode010/active_jobs/2799758.1
> ENVIRONMENT=BATCH
> HOME=/home
> SHLVL=3
> NQUEUES=1
> SGE_CWD_PATH=/path/to/workdir/
> SGE_O_LOGNAME=mjakobs
> SGE_O_MAIL=/var/spool/mail/mjakobs
> TMP=/tmp/2799758.1.serial.q
> LOGNAME=b05055gj
> JOB_ID=2799758
> SYSTEMCTL_IGNORE_DEPENDENCIES=
> SGE_TASK_FIRST=undefined
> OMP_NUM_THREADS=1
> PROMPT_COMMAND=PS1="Singularity> "; unset PROMPT_COMMAND
> SINGULARITY_CONTAINER=/path/to/workdir/pipelines/Sanger_hg37/dockstore-cgpwgs_1.1.5.sif
> SGE_O_PATH=/path/to/workdir/shared/software/miniconda/bin:/mnt/bmh01-rds/UoOxford_David_W/shared/software/miniconda/condabin:/opt/site/sge/bin:/opt/site/sge/bin/lx-amd64:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/path/to/my/homedir/.local/bin:/path/to/my/homedir/bin:/path/to/workdir/pipelines/Sanger_hg37/tools/bin:/path/to/workdir/software/go/bin:
> SGE_O_HOST=hlogin2
> SGE_O_SHELL=/bin/bash
> REQUEST=Step6_run_cgpwgs_20210812.sh
> R_LIBS_USER=/opt/wtsi-cgp/R-lib
> NSLOTS=1
> SGE_STDIN_PATH=/dev/null
> _=/usr/bin/env
>         BAM_MT : /mnt/in/Test_G0481L09_S1_mapped.bam
>         BAM_WT : /mnt/in/Test_G0481L09_S17_L004_mapped.bam
>         NAME_MT : 3be90900-f6c5-11eb-b585-e143ccb5f655
>         NAME_WT : e2e40072-f6c3-11eb-b585-e143ccb5f655
> Setting up Parallel block 1
>         [Parallel block 1] CaVEMan setup added...
>         [Parallel block 1] BB splitlocifiles added...
>         [Parallel block 1] Genotype Check added...
>         [Parallel block 1] VerifyBam Normal added...
> Starting Parallel block 1: Thu Aug 12 17:34:05 BST 2021
>         Starting geno
>         Starting CaVEMan_setup
>         Starting splitlocifiles
>         Starting verify_WT
> 
> Setting up Parallel block 2
>         [Parallel block 2] BB alleleCount added...
>         [Parallel block 2] CaVEMan split added...
> Starting Parallel block 2: Thu Aug 12 17:40:48 BST 2021
>         Starting CaVEMan_split
>         Starting alleleCount

The error from the job is:

+ set +x
+ bash -c '/usr/bin/time -v compareBamGenotypes.pl  -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/genotyped  -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam  -j /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/genotyped/result.json  -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam >& /home/timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.geno ; echo '\''WRAPPER_EXIT: '\''$?'
+ set +x
+ bash -c '/usr/bin/time -v caveman.pl  -r /home/reference_files/genome.fa.fai  -ig /home/reference_files/caveman/HiDepth.tsv  -b /home/reference_files/caveman/flagging  -ab /home/reference_files/vagrent  -u /home/reference_files/caveman  -s '\''human'\''  -sa GRCh37d5  -t 16  -st WGS  -tc /home/tmp/tum.cn.bed  -nc /home/tmp/norm.cn.bed  -td 5 -nd 2  -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam  -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam  -c /home/flag.vcf.config.WXS.ini  -f /home/reference_files/caveman/flagging/flag.to.vcf.convert.ini  -e 350000  -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman  -p setup >& /home/timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.CaVEMan_setup ; echo '\''WRAPPER_EXIT: '\''$?'
+ set +x
+ bash -c '/usr/bin/time -v battenberg.pl     -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg     -u /home/reference_files/battenberg/1000genomesloci     -e /home/reference_files/battenberg/impute/impute_info.txt     -c /home/reference_files/battenberg/probloci.txt     -r /home/reference_files/genome.fa.fai     -ig /home/reference_files/battenberg/ignore_contigs.txt     -ge XX     -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam     -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam     -p splitlocifiles     -nl 50     -t 16 >& /home/timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.splitlocifiles ; echo '\''WRAPPER_EXIT: '\''$?'
+ set +x
+ bash -c '/usr/bin/time -v verifyBamHomChk.pl -d 25   -o /home/WGS_e2e40072-f6c3-11eb-b585-e143ccb5f655/contamination   -b /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam   -j /home/WGS_e2e40072-f6c3-11eb-b585-e143ccb5f655/contamination/result.json >& /home/timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.verify_WT ; echo '\''WRAPPER_EXIT: '\''$?'
+ set +x
+ bash -c '/usr/bin/time -v caveman.pl  -r /home/reference_files/genome.fa.fai  -ig /home/reference_files/caveman/HiDepth.tsv  -b /home/reference_files/caveman/flagging  -ab /home/reference_files/vagrent  -u /home/reference_files/caveman  -s '\''human'\''  -sa GRCh37d5  -t 16  -st WGS  -tc /home/tmp/tum.cn.bed  -nc /home/tmp/norm.cn.bed  -td 5 -nd 2  -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam  -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam  -c /home/flag.vcf.config.WXS.ini  -f /home/reference_files/caveman/flagging/flag.to.vcf.convert.ini  -e 350000  -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman  -p split >& /home/timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.CaVEMan_split ; echo '\''WRAPPER_EXIT: '\''$?'
+ set +x
+ bash -c '/usr/bin/time -v battenberg.pl     -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg     -u /home/reference_files/battenberg/1000genomesloci     -e /home/reference_files/battenberg/impute/impute_info.txt     -c /home/reference_files/battenberg/probloci.txt     -r /home/reference_files/genome.fa.fai     -ig /home/reference_files/battenberg/ignore_contigs.txt     -ge XX     -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam     -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam     -p allelecount     -nl 50     -t 16 >& /home/timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.alleleCount ; echo '\''WRAPPER_EXIT: '\''$?'
ERRORS OCCURRED:
/home/CaVEMan_split.wrapper.log
/home/alleleCount.wrapper.log

The contents of CaVEMann_split.wrapper.log are:

> WRAPPER_EXIT: 25

The contents of alleleCount.wrapper.log are:

>  WRAPPER_EXIT: 25

The contents of /timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.CaVEMan_split are:

> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
>         LANGUAGE = (unset),
>         LC_ALL = (unset),
>         LANG = "en_GB.UTF-8"
>     are supported and installed on your system.
> perl: warning: Falling back to the standard locale ("C").
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.1 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.2 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.3 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.4 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.5 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.6 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.7 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.8 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.9 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.10 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.11 as previously successful
> Thread 12 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e
> 143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.12.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.13 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.14 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.15 as previously successful
> Thread 16 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e
> 143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.16.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.17 as previously successful
> Thread 18 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e
> 143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.18.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.19 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.20 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.21 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.22 as previously successful
> Thread 23 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.23.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Thread 24 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.24.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.25 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.26 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.27 as previously successful
> Skipping Sanger_CGP_Caveman_Implement_caveman_split.28 as previously successful
> Thread error: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.12.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> 

> 
> Perl exited with active threads:
>         0 running and unjoined
>         16 finished and unjoined
>         0 running and detached
> Command exited with non-zero status 25
>         Command being timed: "caveman.pl -r /home/reference_files/genome.fa.fai -ig /home/reference_files/caveman/HiDepth.tsv -b /home/reference_files/caveman/flagging -ab /home/reference_files/vagrent -u /home/reference_files/caveman -s human -sa GRCh37d5 -t 28 -st WGS -tc /home/tmp/tum.cn.bed -nc /home/tmp/norm.cn.bed -td 5 -nd 2 -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam -c /home/flag.vcf.config.WXS.ini -f /home/reference_files/caveman/flagging/flag.to.vcf.convert.ini -e 350000 -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman -p split"
>         User time (seconds): 1.07
>         System time (seconds): 0.47
>         Percent of CPU this job got: 0%
>         Elapsed (wall clock) time (h:mm:ss or m:ss): 2:49.63
>         Average shared text size (kbytes): 0
>         Average unshared data size (kbytes): 0
>         Average stack size (kbytes): 0
>         Average total size (kbytes): 0
>         Maximum resident set size (kbytes): 225580
>         Average resident set size (kbytes): 0
>         Major (requiring I/O) page faults: 0
>         Minor (reclaiming a frame) page faults: 159608
>         Voluntary context switches: 204
>         Involuntary context switches: 37
>         Swaps: 0
>         File system inputs: 224
>         File system outputs: 464
>         Socket messages sent: 0
>         Socket messages received: 0
>         Signals delivered: 0
>         Page size (bytes): 4096
>         Exit status: 25

I have tried deleting the scripts in /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman/tmpCaveman/logs/ and re-run the code but ended up with the same errors.

The contents of /timings/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655.time.alleleCountare:

> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
>         LANGUAGE = (unset),
>         LC_ALL = (unset),
>         LANG = "en_GB.UTF-8"
>     are supported and installed on your system.
> perl: warning: Falling back to the standard locale ("C").
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.1 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.2 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.3 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.4 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.5 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.6 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.7 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.8 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.9 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.10 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.11 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.12 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.13 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.14 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.15 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.16 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.17 as previously successful
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.18 as previously successful
> Thread 19 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.19.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.20 as previously successful
> Thread 21 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.21.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Skipping Sanger_CGP_Battenberg_Implement_battenberg_allelecount.22 as previously successful
> Thread 23 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.23.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Thread 24 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.24.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Thread 25 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.25.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Thread 26 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.26.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Thread 27 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.27.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Thread 28 terminated abnormally: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.28.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> 
> Thread error: ERROR: Script already present, delete to proceed: /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg/tmpBattenberg/logs/Sanger_CGP_Battenberg_Implement_battenberg_allelecount.19.sh at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 289.
> Perl exited with active threads:
>         0 running and unjoined
>         9 finished and unjoined
>         0 running and detached
> Command exited with non-zero status 25
>         Command being timed: "battenberg.pl -o /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/battenberg -u /home/reference_files/battenberg/1000genomesloci -e /home/reference_files/battenberg/impute/impute_info.txt -c /home/reference_files/battenberg/probloci.txt -r /home/reference_files/genome.fa.fai -ig /home/reference_files/battenberg/ignore_contigs.txt -ge XX -tb /home/tmp/3be90900-f6c5-11eb-b585-e143ccb5f655.bam -nb /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam -p allelecount -nl 50 -t 28"
>         User time (seconds): 1.51
>         System time (seconds): 0.56
>         Percent of CPU this job got: 1%
>         Elapsed (wall clock) time (h:mm:ss or m:ss): 2:50.18
>         Average shared text size (kbytes): 0
>         Average unshared data size (kbytes): 0
>         Average stack size (kbytes): 0
>         Average total size (kbytes): 0
>         Maximum resident set size (kbytes): 289368
>         Average resident set size (kbytes): 0
>         Major (requiring I/O) page faults: 0
>         Minor (reclaiming a frame) page faults: 218529
>         Voluntary context switches: 270
>         Involuntary context switches: 46
>         Swaps: 0
>         File system inputs: 0
>         File system outputs: 472
>         Socket messages sent: 0
>         Socket messages received: 0
>         Signals delivered: 0
>         Page size (bytes): 4096
>         Exit status: 25

The contents of /WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman/tmpCaveman/logs/Sanger_CGP_Caveman_Implement_caveman_split.12.err are:

> + /opt/wtsi-cgp/bin/caveman split -i 12 -f /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655/caveman/tmpCaveman/caveman.cfg.ini -e 350000
> Warning: The index file is older than the data file: /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam.bai
> Warning: The index file is older than the data file: /home/tmp/e2e40072-f6c3-11eb-b585-e143ccb5f655.bam.bai

Sorry, I know it's a bit of a long one. Any help or advice would be much appreciated!

Best, Maria

keiranmraine commented 3 years ago

Hi, It looks like you've requested 28 CPUs, how much memory have you assigned to job.

mjakobs commented 3 years ago

Hi Keiran, thanks for looking into this. I've tried submitting the job to a node with 16 cores at 32GB per core.

keiranmraine commented 3 years ago

I think I've spotted what's wrong. If the job fails you need to find and remove all **/tmp*/logs/*.sh files if you want to attempt a restart as there are multiple trees of work ocurring in parallel.

Please verify this command gives the appropriate files with echo in place of rm-ffirst:

find /home/WGS_3be90900-f6c5-11eb-b585-e143ccb5f655_vs_e2e40072-f6c3-11eb-b585-e143ccb5f655 -type f -name \*.log | grep -P '/tmp[^/]+/logs/[^/]+.sh' | xargs rm -f
mjakobs commented 3 years ago

Hi Keiran,

Thank you for your help! Sorry for the delay in responding to this - I had to resubmit my job a few times with different resource requirements. I ended up deleting all of the files that were produced and then resubmitting.

The pipeline has finished running with an exit status of 0, so I'm happy to close this issue, but I did get a final message in my output log that I wasn't quite sure about:

Run POST_EXEC: Fri Aug 20 15:24:14 BST 2021
No POST_EXEC defined

Workflow end: Fri Aug 20 15:24:14 BST 2021

Do I need to worry about this?

keiranmraine commented 3 years ago

POST_EXEC is not relevant and is actually removed in more recent releases.