Strelka Asserting error.

[snashraf]

Hi Brad, I am here again for your help !! Sorry for troubling you again and again !!

I am getting below error now which seems like happening at the step of combining sample regions from strelka.

[2017-10-17T19:20Z] ngslogin1: multiprocessing: combine_sample_regions [2017-10-17T19:20Z] ngslogin1: Identified 204 parallel analysis blocks Block sizes: min: 145363 5%: 6373368.45 25%: 15481367.25 median: 15503393.5 75%: 15643451.0 95%: 16882500.65 99%: 32695647.92 max: 43716635 Between block sizes: min: 308 5%: 549.65 25%: 2102.75 median: 11546.5 75%: 43531.25 95%: 498717.85 99%: 1668411.53 max: 2239111

[2017-10-17T19:20Z] ngslogin1: Timing: hla typing [2017-10-17T19:21Z] ngslogin1: Timing: alignment post-processing [2017-10-17T19:21Z] ngslogin1: ipython: piped_bamprep [2017-10-17T19:23Z] ngslogin1: Timing: variant calling [2017-10-17T19:23Z] ngslogin1: ipython: variantcall_sample [2017-10-17T19:27Z] nsnode47: Unexpected error Traceback (most recent call last): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 50, in _setup_logging yield config File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 368, in variantcall_sample return ipython.zip_args(apply(genotype.variantcall_sample, *args)) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/variation/genotype.py", line 347, in variantcall_sample call_file = caller_fn(align_bams, items, ref_file, assoc_files, region, call_file) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 23, in run call_file = _run_somatic(paired, ref_file, assoc_files, region, out_file) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 176, in _run_somatic for f in ["somatic.snvs.vcf.gz", "somatic.indels.vcf.gz"]], File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 161, in _postprocess_somatic parts[7].split(";")) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 128, in _tumor_normal_genotypes tumor_gt = name_to_gt(sgt_val) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 109, in name_to_gt assert val.lower() in ["ref", "conflict"], val AssertionError: GG

Thanks Najeeb

[chapmanb]

Najeeb; Sorry about the issue. bcbio has custom code which is trying to convert the way that strelka2 represents the tumor and normal genotypes in the INFO field into standard FORMAT based GT genotypes. It looks like this is a case we haven't seen before. I pushed a change to the latest development version which provides additional information when this fails. Is it possible to update and re-run with this and report what it outputs? That should help give us an idea about how best to resolve it. Thanks for the help debugging.

[snashraf]

Hi Brad,

Thanks for your kind help !! When I run the program after updating it, it goes to the next step which is CNV calling using CNVKIT. So I was not able to get any error message for Strelka again. And even I run Somatic calling in the ensemble mood but didn't get any ensemble VCF as well. Is there anything I can do to replicate that error again? Between I noted that CNVKIt is not running in over HPC cluster as well.

[chapmanb]

Thanks for testing this and sorry about the confusion with the restart. Could you try removing checkpoints_parallel and re-running? What you're seeing is that the latest development version moves some aspects of CNV calling upstream to prior to variant calling as part of work to generalize CNV calling and support multiple callers. So it hasn't yet got to the strelka2 recalls (or ensemble) and isn't parallelizing because of the present of checkpoints from previous runs. Hopefully if you do this the CNV prep work will parallelize, finish, then it'll move on to strelka2. Thanks again.

[snashraf]

Hi Brad,

Now I am getting below error.

' returned non-zero exit status 1 [2017-10-18T23:50Z] hpcgenelite06.research.sidra.local: Uncaught exception occurred Traceback (most recent call last): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run _do_run(cmd, checks, log_stdout, env=env) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command 'set -o pipefail; zcat /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/coverage/0200135821/0200135821-target.regions.bed.gz | grep -v ^track | grep -v ^browser | grep -v ^# | sort -V -k1,1 -k2,2n | bedtools closest -g <(cut -f1,2 /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/inputs/ref/GRCh37-subset.fa.fai | sort -V -k1,1 -k2,2n) -d -t all -a - -b <(sort -V -k1,1 -k2,2n /gpfs/home/nsyed/bcbio/genomes/Hsapiens/GRCh37/rnaseq/ref-transcripts.bed) | awk -F$'\t' -v OFS='\t' '{if ($NF > 0) $9 = "."} {print}' | cut -f 1-10 | bedtools merge -i - -c 4,5,9 -o distinct,distinct,distinct -delim ',' -d -10 > /gpfs/projects/bioinfo/najeeb/withDavide/templateFinalPaired/work/bcbiotx/tmp7zt5oo/0200135821-target.regions-annotated.bed Error: requested chromosome GL000191.1 does not exist in the genome file /dev/fd/63. Exiting. sort: write failed: standard output: Broken pipe sort: write error sort: write failed: standard output: Broken pipe sort: write error

It seem this is because I have already subsetted FASTA file . Is their a way to avoid this?

[chapmanb]

Sorry about the issue, this is due to having a custom reference file that is out of sync with the provided transcript file. I pushed a fix to the latest development version which avoids this problem, if you update and re-run in place it should subset the initial BED to only contigs present in the reference genome and hopefully continue on to strelka2 so we can get back to diagnosing the initial issue.

Out of curiosity, why are you using a subset genome? The best way to avoid calls on on standard chromosomes is to align to the entire genome, then use a variant_regions file that specifies only the contigs you have interest in. Hope this helps as a better approach then building custom genomes for what you need.

[snashraf]

Hi Brad,

Here is error that I have getting with strelka.

ipyparallel.error.CompositeError: one or more exceptions from call to method: variantcall_sample

Since I started by analysis from BAM files and which was having an error due to Nonstandard Contigs so I just used BAM Clean step to remove that contigs and which seems to created subset genome.

[chapmanb]

Thanks much for passing along the outputs and for all the debugging help so far. This appears to be a case we hadn't seen and weren't handling in our custom conversion of strelka2 INFO tags into tumor and normal genotypes. The SGT specification is reverse complemented relative to the REF and ALT alleles. I pushed a fix which should handle this so if you update to the latest development version and re-run in place it should hopefully work cleanly for you. Please let us know if you run into any other issues.

[ctsa]

Hi @chapmanb -

Sorry for the confusion here. We should perhaps sync up on a separate issue about what you're trying to process out of the strelka output. There should not be any signatures in the SGT field which are rev-complemented relative to REF/ALT, in fact REF and ALT do not necessarily have to show up in SGT. What we're doing with SGT is trying as best we can to work around the limitations in the VCF format to express more complex somatic assertions. Let me go through some of the lines in the error message above and provide some context:

19:36223111

 ['19', '36223111'], 'C', ['A'], ['SOMATIC', 'QSS=1', 'TQSS=1', 'NT=hom', 'QSS_NT=1', 'TQSS_NT=1', 'SGT=GG->GG', 'DP=145', 'MQ=59.99', 'MQ0=0', 'ReadPosRankSum=-0.41', 'SNVSB=0.00', 'SomaticEVS=0.00'], 'GG')

In english, this is describing the following: At position 36223111, the reference is "C", and the germline genotype is "GG". We tested the assertion that the somatic allele "A" was also present at this site in the tumor sample, but rejected this because there was insufficient evidence.

1. In this case NT=hom indicates that we believe the germline is homozygous-alternative, more specifically that the genotype is "GG". Perhaps the first point of confusion here is that "G" is not listed in the ALT field, because we try to only list the somatic alt to avoid confusion.
2. "A" is listed as the ALT because this was the somatic allele evaluated, but the evidence for this being a somatic variant was too weak, thus the 'right' side of the SGT field is also "GG".

6:125252994

['6', '125252994'], 'C', ['T'], ['SOMATIC', 'QSS=2', 'TQSS=1', 'NT=hom', 'QSS_NT=2', 'TQSS_NT=1', 'SGT=AA->AA', 'DP=77', 'MQ=60.00', 'MQ0=0', 'ReadPosRankSum=0.63', 'SNVSB=0.00', 'SomaticEVS=0.00'], 'AA')

Similar to the above case: the germline genotype is AA, and we tested the assertion that the somatic allele "T" is present in the tumor sample, but rejected it due to insufficient evidence.

6:32634331

['6', '32634331'], 'G', ['A'], ['SOMATIC', 'QSS=1', 'TQSS=1', 'NT=hom', 'QSS_NT=1', 'TQSS_NT=1', 'SGT=CC->CC', 'DP=100', 'MQ=60.00', 'MQ0=0', 'ReadPosRankSum=-1.77', 'SNVSB=0.00', 'SomaticEVS=0.00'], 'CC')

Similar again: the germline genotype is CC, and we tested the assertion that the somatic allele "A" is present in the tumor sample, but rejected it due to insufficient evidence.

The root of the problem seems to be that Strelka provides call information in cases where the germline may be something other than homozygous-ref (there are calls were the germline is heterozygous as well NT=het, but these are less interesting because they are mostly triggered by LOH regions). We never PASS these calls by default but for some use cases these other somatic call types are processed and evaluated. We have some evidence that somatic calls at homozygous alt sites in the germline have a similar FP rate to conventional somatic calls, and may start passing these in the future, especially if we can find a reliable way to convey this in the VCF output.

-Chris

[snashraf]

Hi Brad,

Now I am getting this error finally.

[2017-10-20T17:59Z] nsnode38: Uncaught exception occurred Traceback (most recent call last): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run _do_run(cmd, checks, log_stdout, env=env) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 86, in _do_run line = s.stdout.readline() KeyboardInterrupt [2017-10-20T17:59Z] nsnode38: Uncaught exception occurred Traceback (most recent call last): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run _do_run(cmd, checks, log_stdout, env=env) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 86, in _do_run line = s.stdout.readline() KeyboardInterrupt [2017-10-20T17:59Z] nsnode38: Uncaught exception occurred Traceback (most recent call last): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run _do_run(cmd, checks, log_stdout, env=env) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 86, in _do_run line = s.stdout.readline() KeyboardInterrupt [2017-10-20T17:59Z] nsnode38: Uncaught exception occurred Traceback (most recent call last):

This error is happening when It was trying to merge VCF files. Can you please look into this.

Thanks Najeeb

[chapmanb]

Chris -- thanks so much for the details. I opened an issue on the strelka2 tracker to discuss this more and figure out the best way forward. Essentially we want to have standard genotypes for the somatic calls but maybe it would make sense to do more of it in strelka instead of these bcbio post-calling hacks. Thanks again for helping with this.

Najeeb -- sorry about the additional run problems. I can't provide much help from that output except that the jobs were killed. It reports KeyboardInterrupt which suggests someone hit ctrl-c to stop the job, but this could also come from a scheduler or the system stopping jobs which over-run memory or other resource constraints. How are you running bcbio? It is possible the step it got killed at used excessive memory or something else that we can debug? If you're using a scheduler you can often see this in the bcbio-engine* logs produced in the work directory from the scheduler job submissions. Hope this helps some for debugging and happy to provide more suggestions with additional details.

[snashraf]

bcbio-ipengine.bsub.%588777.txt bcbio-ipengine.bsub.%588911.txt bcbio-ipengine.bsub.%588914.txt bcbio-ipengine.bsub.%588919.txt

Hi Brad,

I was using LSF Scheduler to run the job. I am attaching logs file from scheduler as well. I looked into memory utilization and seems that its not even taking memory then few MBs so doest look like killing due to extra memory usage . Can you please suggest on this ?

[chapmanb]

Najeeb; Thanks for the additional log files. It's still not clear to me why it's getting killed but it's very useful to know where it's happening. Could you try updating to the latest bcbio development version (bcbio_nextgen.py upgrade -u development)? We've reworked this section of the code to be less resource intensive, so I'm hoping that it will avoid the issue you're seeing. If you still run into problems seeing the updated outputs would be helpful to try and track down further. Hope this helps get things running for you.

[snashraf]

Hi Brad,

Now I am getting below error. can you please check this now. ESC[0;31mCalledProcessErrorESC[0m: Command 'set -o pipefail; picard FixVcfHeader HEADER=/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bcbiotx/tmpy1vGsB/batch23-1_0_15491016-fixheader-header.vcf INPUT=/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/scalpel/1/batch23-1_0_15491016.vcf.gz OUTPUT=/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/scalpel/batch23-1_0_15491016-fixheader.vcf.gz 10:53:52.564 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gpfs/home/nsyed/bcbio/anaconda/share/picard-2.14-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so [Wed Nov 01 10:53:52 AST 2017] FixVcfHeader INPUT=/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/scalpel/1/batch23-1_0_15491016.vcf.gz OUTPUT=/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/scalpel/batch23-1_0_15491016-fixheader.vcf.gz HEADER=/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bcbiotx/tmpy1vGsB/batch23-1_0_15491016-fixheader-header.vcf CHECK_FIRST_N_RECORDS=-1 ENFORCE_SAME_SAMPLES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false [Wed Nov 01 10:53:52 AST 2017] Executing as nsyed@nsnode40 on Linux 3.10.0-229.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_121-b15; Deflater: Intel; Inflater: Intel; Picard version: 2.14-SNAPSHOT INFO 2017-11-01 10:53:52 FixVcfHeader FORMAT line found in HEADER will be added to OUTPUT: GT INFO 2017-11-01 10:53:52 FixVcfHeader FORMAT line found in HEADER will be added to OUTPUT: DP INFO 2017-11-01 10:53:52 FixVcfHeader FORMAT line found in HEADER will be added to OUTPUT: AD [Wed Nov 01 10:53:52 AST 2017] picard.vcf.FixVcfHeader done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=514850816 To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp Exception in thread "main" htsjdk.tribble.TribbleException: Contig 1 does not have a length field. at htsjdk.variant.vcf.VCFContigHeaderLine.getSAMSequenceRecord(VCFContigHeaderLine.java:80) at htsjdk.variant.vcf.VCFHeader.getSequenceDictionary(VCFHeader.java:206) at picard.vcf.FixVcfHeader.doWork(FixVcfHeader.java:188) at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:268) at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98) at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)

[chapmanb]

Najeeb; Thanks for the report and sorry for the additional problems. The traceback looks like you're using scalpel as a variantcaller in your analysis. Is that right? That's not a typical usage of scalpel, which is primarily used to add indels to mutect SNP-only calling, so you might want to just avoid using it here. I also pushed a fix which resolves the problem by adding contigs to the scalpel VCFs but to fix you'l have to re-run the scalpel calling (rm -rf scalpel) before re-running. Hope this fixes it for you.

[snashraf]

Hi Brad,

Now I am stuck at this steps. Sees its stuck at HLA typing steps. Below are logs data and I am copying my config.yaml files as well. Can you please suggest on this? ==> bcbio-nextgen-debug.log <== 95%: 498717.85 99%: 1668411.53 max: 2239111

[2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: ipython: calculate_sv_bins [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: ipython: calculate_sv_coverage [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: Timing: hla typing [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: ipython: call_hla [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: Resource requests: freebayes, gatk, mutect, mutect2, picard, scalpel, strelka2, vardict, varscan; memory: 16.00, 10.00, 16.00, 16.00, 16.00, 16.00, 16.00, 16.00; cores: 8, 1, 1, 8, 8, 8, 8, 8, 8 [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: Configuring 982 jobs to run, using 1 cores each with 16.00g of memory reserved for each job

==> bcbio-nextgen.log <== median: 11546.5 75%: 43531.25 95%: 498717.85 99%: 1668411.53 max: 2239111

[2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: ipython: calculate_sv_bins [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: ipython: calculate_sv_coverage [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: Timing: hla typing [2017-11-02T08:07Z] hpcgenomicn19.research.sidra.local: ipython: call_hla

-- details:

• analysis: variant2 genome_build: GRCh37 metadata: batch: your-batch-name phenotype: tumor # or "normal" algorithm: mark_duplicates: true recalibrate: false realign: false remove_lcr: true bam_clean: remove_extracontigs variantcaller: [mutect, freebayes, vardict, varscan, mutect2, strelka2, scalpel] indelcaller: false ensemble: numpass: 2 variant_regions: /gpfs/projects/bioinfo/najeeb/refData/exomeTargetedBedfile/S07604514/S07604514_Covered_withoutChr.bed svcaller: [cnvkit, lumpy, delly, manta]

[chapmanb]

Najeeb; Sorry about the issue. I don't see any reason based on your configuration it'll be stalling in call_hla. Since it's build 37, this doesn't do any HLA calling. Would it be possible to run this with a single core (-n 1) and see if it replicates the behavior of hanging there? If so, ctrl-C-ing out of the process would let us know what it's doing that causes it to wait and give some clue about how to fix.

Sorry for not knowing what is going on but hope this helps.

[snashraf]

yes, Its working when I am using (-n 1) or submitting on the terminal. I am guessing that since each Job is taking 1 core and 18 GB of memory and node on which jobs are going doesn't have enough memory!

On Fri, Nov 3, 2017 at 1:19 PM, Brad Chapman notifications@github.com wrote:

Najeeb; Sorry about the issue. I don't see any reason based on your configuration it'll be stalling in call_hla. Since it's build 37, this doesn't do any HLA calling. Would it be possible to run this with a single core (-n 1) and see if it replicates the behavior of hanging there? If so, ctrl-C-ing out of the process would let us know what it's doing that causes it to wait and give some clue about how to fix.

Sorry for not knowing what is going on but hope this helps.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chapmanb/bcbio-nextgen/issues/2112#issuecomment-341665269, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJUN0gTOiaDYxx4FRLxglU0boWo_LL2ks5syujNgaJpZM4P8uOk .

-- Syed Najeeb Ashraf

[chapmanb]

Thanks for following up. Do you have a sense of what process uses this much memory and we could try to debug further? I'd be happy to try and improve memory usage to get this working better with an understanding of what is causing issues on your dataset. Thanks for the help debugging.

[snashraf]

Hi Brad,

Scapel was taking so much memory.

I am getting error with MANTA now.

[2017-11-07T15:00Z] [2017-11-07T15:00:05.124862]
[ngslogin1.research.sidra.local] [12657_1]
[TaskRunner:makeLocusGraph_chromId_005_6_0008] Task initiated on local node
[2017-11-07T15:00Z] [2017-11-07T15:00:05.935812]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR] Failed to
complete command task: 'makeLocusGraph_chromId_005_6_0008' launched from
master workflow, error code: 1, command:
'/gpfs/home/nsyed/bcbio/anaconda/share/manta-1.2.1-0/libexec/EstimateSVLoci
--output-file
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bcbiotx/tmpP2hqRC/manta/workspace/svLocusGraph.bin.tmpdir/svLocusGraph.chromId_005_6_0008.bin
--align-stats
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bcbiotx/tmpP2hqRC/manta/workspace/alignmentStats.xml
--region 6:91261371-102669041 --min-candidate-sv-size 8
--min-edge-observations 3 --ref
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/inputs/ref/GRCh37-subset.fa
--align-file
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bamclean/0200135731/0200135731-noextras-dedup.bam
--tumor-align-file
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bamclean/0200135681/0200135681-noextras-dedup.bam'
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936179]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[makeLocusGraph_chromId_005_6_0008] Error Message:
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[makeLocusGraph_chromId_005_6_0008] Last 12 stderr lines from task (of 12
total lines):
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.618511] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] ERROR: Exception caught while
processing single read record:
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.621452] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008]
 ST-E00301:165:HKNYLALXX:7:2202:9841:23636/1 tid:pos:strand 5:91619108:+
cigar: 116H30M4H templSize: 0 sa: GL000229.1,5112,+,6M2I72M4D50M20S,60,6;
issupp mate_tid:pos:strand -1:2176234:-
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.843737] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] FATAL_ERROR: EstimateSVLoci EXCEPTION:
2017-Nov-07 18:00:05: Operation not permitted:
/builder/src/c++/lib/manta/SVLocusScanner.cpp(343): Throw in function void
parseSACandidatesFromRead(const ReadScannerOptions&, const bam_record&,
const chromMap_t&, std::vector<SimpleAlignment>&)
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.846543] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] Dynamic exception type:
N5boost16exception_detail10clone_implIN8illumina6common14LogicExceptionEEE
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.848932] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] std::exception::what: ERROR: Split
alignment segment maps to an unknown chromosome: 'GL000229.1'
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.851410] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008]
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.853475] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008]
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.880831] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] ...caught in program.run()
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.882964] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] cmdline:
/gpfs/home/nsyed/bcbio/anaconda/share/manta-1.2.1-0/libexec/EstimateSVLoci
--output-file
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bcbiotx/tmpP2hqRC/manta/workspace/svLocusGraph.bin.tmpdir/svLocusGraph.chromId_005_6_0008.bin
--align-stats
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bcbiotx/tmpP2hqRC/manta/workspace/alignmentStats.xml
--region 6:91261371-102669041 --min-candidate-sv-size 8
--min-edge-observations 3 --ref
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/inputs/ref/GRCh37-subset.fa
--align-file
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bamclean/0200135731/0200135731-noextras-dedup.bam
--tumor-align-file
/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/templateFinalPaired/work/bamclean/0200135681/0200135681-noextras-dedup.bam
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.885745] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] version:  1.2.1
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.888684] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] buildTime:
2017-10-09T20:54:15.697482Z
[2017-11-07T15:00Z] [2017-11-07T15:00:05.936238]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR]
[2017-11-07T15:00:05.891241] [ngslogin1.research.sidra.local] [12657_1]
[makeLocusGraph_chromId_005_6_0008] compiler: g++-4.9.3
[2017-11-07T15:00Z] [2017-11-07T15:00:05.937486]
[ngslogin1.research.sidra.local] [12657_1] [TaskManager] [ERROR] Shutting
down task submission. Waiting for remaining tasks to complete.
[2017-11-07T15:00Z] [2017-11-07T15:00:16.863172]
[ngslogin1.research.sidra.local] [12657_1] [WorkflowRunner] [ERROR] Worklow
terminated due to the following task errors:
[2017-11-07T15:00Z] [2017-11-07T15:00:16.863477]
[ngslogin1.research.sidra.local] [12657_1] [WorkflowRunner] [ERROR] Failed
to complete command task: 'makeLocusGraph_chromId_005_6_0008' launched f:

On Sun, Nov 5, 2017 at 4:21 AM, Brad Chapman notifications@github.com wrote:

Thanks for following up. Do you have a sense of what process uses this much memory and we could try to debug further? I'd be happy to try and improve memory usage to get this working better with an understanding of what is causing issues on your dataset. Thanks for the help debugging.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chapmanb/bcbio-nextgen/issues/2112#issuecomment-341942091, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJUN9TuCKDDVorC2bb7fFJSpC8dDvVRks5szQ2ZgaJpZM4P8uOk .

-- Syed Najeeb Ashraf

[chapmanb]

Sorry about the issue, I'm not exactly sure what is happening with manta but the error message looks like this issue (https://github.com/Illumina/manta/issues/85). How are you preparing the input alignments? Are these done with bcbio or using pre-aligned data? Is it possible the inputs have extra contigs not present in the reference genome? If so, you could try using bam_clean: remove_extracontigs which subsets to chr1-22,X,Y to avoid this:

https://bcbio-nextgen.readthedocs.io/en/latest/contents/configuration.html#alignment

Hope this helps explain what is going on.

[snashraf]

I have done this step already for one of other tools so I assume that my bams are already without these contigs.

Thanks Najeeb

On Wed, Nov 8, 2017 at 7:17 PM, Brad Chapman notifications@github.com wrote:

Sorry about the issue, I'm not exactly sure what is happening with manta but the error message looks like this issue (Illumina/manta#85 https://github.com/Illumina/manta/issues/85). How are you preparing the input alignments? Are these done with bcbio or using pre-aligned data? Is it possible the inputs have extra contigs not present in the reference genome? If so, you could try using bam_clean: remove_extracontigs which subsets to chr1-22,X,Y to avoid this:

https://bcbio-nextgen.readthedocs.io/en/latest/ contents/configuration.html#alignment

Hope this helps explain what is going on.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chapmanb/bcbio-nextgen/issues/2112#issuecomment-342868604, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJUN-bWsUCk3ecmPoSBDrvi2ZsteZgrks5s0dQBgaJpZM4P8uOk .

-- Syed Najeeb Ashraf

[chapmanb]

Najeeb -- I don't really understand exactly what you mean so don't have a good picture of how you've processed your inputs and they related to the bcbio genome you're using. I know manta is fairly stringent about how it deals with BAMs and seems to be running into something in your input it doesn't like. It might be worth excluding manta if you don't have any flexibility in processing, allow bcbio to align your BAM if you do, or you'll need to dig more into what is calling the issue in your input BAM. Hope this helps.

[snashraf]

As suggested I disable Manta and I tried to run again and now I am getting error with CNVKit .

Calling copy number with thresholds: -1.1 => 0, -0.25 => 1, 0.2 => 2, 0.7 => 3 Applying filter 'cn' Traceback (most recent call last): File "/gpfs/home/nsyed/bcbio/anaconda/bin/cnvkit.py", line 13, in args.func(args) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/cnvlib/commands.py", line 697, in _cmd_call args.thresholds) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/cnvlib/call.py", line 71, in do_call outarr = getattr(segfilters, filt)(outarr) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/cnvlib/segfilters.py", line 30, in wrapped_f result = func(segarr) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/cnvlib/segfilters.py", line 138, in cn return squash_by_groups(segarr, segarr['cn']) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/cnvlib/segfilters.py", line 53, in squash_by_groups data = data.groupby(groupkey, sort=False).apply(squash_region) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/pandas/core/groupby.py", line 716, in apply return self._python_apply_general(f) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/pandas/core/groupby.py", line 720, in _python_apply_general self.axis) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/pandas/core/groupby.py", line 1802, in apply res = f(group) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/cnvlib/segfilters.py", line 75, in squash_region out['log2'] = np.average(cnarr['log2'], weights=cnarr['weight']) File "/gpfs/home/nsyed/bcbio/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 959, in average "Weights sum to zero, can't be normalized") ZeroDivisionError: Weights sum to zero, can't be normalized ' returned non-zero exit status 1 [2017-11-19T12:56Z] nsnode20: Unexpected error Traceback (most recent call last): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 51, in _setup_logging yield config File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 417, in detect_sv return ipython.zip_args(apply(structural.detect_sv, *args)) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/structural/init.py", line 195, in detect_sv for svdata in caller_fn(items): File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 43, in run return _cnvkit_by_type(items, background) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 55, in _cnvkit_by_type return _run_cnvkit_cancer(items, background) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 104, in _run_cnvkit_cancer tumor_data = _associate_cnvkit_out(ckouts, [paired.tumor_data], is_somatic=True) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 71, in _associate_cnvkit_out ckout = _add_variantcalls_to_output(ckout, data, items, is_somatic) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/structural/cnvkit.py", line 470, in _add_variantcalls_to_output do.run(cmd, "CNVkit call ploidy") File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run _do_run(cmd, checks, log_stdout, env=env) File "/gpfs/home/nsyed/.local/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command '/gpfs/home/nsyed/bcbio/anaconda/bin/cnvkit.py call --filter cn --ploidy 2 -o /gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/analysisData/work/bcbiotx/tmpzAtM0O/0200135591-noextras-dedup-batch17-call.cns /gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/analysisData/work/structural/0200135591/cnvkit/raw/0200135591-noextras-dedup-batch17.cns --vcf /gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/analysisData/work/freebayes/batch17-effects-annotated-filter.vcf.gz --sample-id 0200135591 Selected test sample 0200135591 and control sample 0200135641 Loaded 44587 records; skipped: 215 somatic, 15907 depth Skipping 259 additional somatic record based on T/N genotypes Kept 28442 heterozygous of 44328 VCF records Calling copy number with thresholds: -1.1 => 0, -0.25 => 1, 0.2 => 2, 0.7 => 3

Can you please help me on this now ?

Thanks Najeeb

On Wed, Nov 8, 2017 at 7:44 PM, Brad Chapman notifications@github.com wrote:

Najeeb -- I don't really understand exactly what you mean so don't have a good picture of how you've processed your inputs and they related to the bcbio genome you're using. I know manta is fairly stringent about how it deals with BAMs and seems to be running into something in your input it doesn't like. It might be worth excluding manta if you don't have any flexibility in processing, allow bcbio to align your BAM if you do, or you'll need to dig more into what is calling the issue in your input BAM. Hope this helps.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chapmanb/bcbio-nextgen/issues/2112#issuecomment-342877888, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJUN3Yufv0XOSqgQKgL5duzVlVk60jsks5s0dqJgaJpZM4P8uOk .

-- Syed Najeeb Ashraf

[chapmanb]

Najeeb; Sorry about the issue. I'm suspecting something is problematic about some of these samples that are triggering all these edge cases during structural variant calling. This error message seems to indicate everything might be zerod out in your input file. Would you be able to share the input cns file:

/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/analysisData/work/structural/0200135591/cnvkit/raw/0200135591-noextras-dedup-batch17.cns

So we could see what it looks like? Either by e-mail or as a gist (https://gist.github.com/) would work great. I might be able to identify what is wrong with the file and skip processing it.

Practically, if this is blocking your analysis I'd suggest also dropping cnvkit since I don't think it's going to provide useful calls, but I'd also like to cleanly avoid the problem in the future. Thanks again for the patience debugging.

[snashraf]

Hi Brad,

Please find attached file .

On Mon, Nov 20, 2017 at 9:15 AM, Brad Chapman notifications@github.com wrote:

Najeeb; Sorry about the issue. I'm suspecting something is problematic about some of these samples that are triggering all these edge cases during structural variant calling. This error message seems to indicate everything might be zerod out in your input file. Would you be able to share the input cns file:

/gpfs/projects/bioinfo/najeeb/withDavide/inflammatory_breast_cancer/analysisData/work/structural/0200135591/cnvkit/raw/0200135591-noextras-dedup-batch17.cns

So we could see what it looks like? Either by e-mail or as a gist ( https://gist.github.com/) would work great. I might be able to identify what is wrong with the file and skip processing it.

Practically, if this is blocking your analysis I'd suggest also dropping cnvkit since I don't think it's going to provide useful calls, but I'd also like to cleanly avoid the problem in the future. Thanks again for the patience debugging.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chapmanb/bcbio-nextgen/issues/2112#issuecomment-345601397, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJUNxy7sGNz1wfHTyzUu96oYVSUSy3Xks5s4RkBgaJpZM4P8uOk .

-- Syed Najeeb Ashraf

[chapmanb]

Najeeb; Unfortunately you can't attach files via GitHub e-mails, they get removed. You can either post a gist or send it to me directly (https://github.com/chapmanb). Thanks much.

[chapmanb]

Najeeb; Thanks much for passing along the file. This looks like an edge case where CNVkit tries to merge regions without a weight, leading to the normalization error. I sent a fix upstream to CNVkit and pushed a new version that includes it. If you do:

bcbio_conda install -y -c conda-forge -c bioconda cnvkit

and re-run in place it should hopefully pick up where it left off and proceed past this point. Thanks again for the help debugging.