Open stsmall opened 9 years ago
Scott; Sorry about the continued problems. If you could make the FreeBayes file available that would be a big help to help diagnose further. This error indicates there is something wrong with one of the VCF lines in the file since it does not have the INFO column but I'm not sure why that would happen from just the traceback. If the file is not obviously truncated or otherwise problematic I can dig more to see what is happening. Thanks again for all the help debugging.
Hi Brad, I am working on posting the file. Could you please comment on my question above: "If I was only going to use SNPs and not indels is it OK to remove indels/MNPs from the input files of (HaplotypeCaller, UnifiedGenotyper, Freebayes) prior to running ensemble or will this cause discordance errors? This set of files w/ SNPs only runs to completion." thanks, scott
Scott;
Sorry about forgetting to respond to that. That's fine with the cavaet that MNPs also contain SNPs, just compressed together in haplotypes, so you'll miss those. If you've previously pre-processed the files with bcbio or manually using vcfallelicprimitives
, this shouldn't be a problem as they'll already be split into individual SNPs.
input vcf causing errors dropbox link: https://www.dropbox.com/s/2bhhh3bim2oyhtb/WbPNGL3-ALL.freebayesv0.9.20.out.vcf.gz?dl=0
Scott; I've been fruitlessly trying to dig for an error in this file and haven't been able to find anything yet after running through a few different validation tools. So apologies, I'm still not sure what is going on. To reproduce it in the contect of bcbio.variation I'd also need the reference file, since it looks like a custom assembly you might have locally. You can send it to me off list via Dropbox if it's not too large.
Practically, these inputs do need to be run through some post-processing to normalize and split MNPs into individual SNPs/indels. Here are the post-calling commands we use in bcbio for this:
Doing that might help resolve your issue if it's somehow related to MNPs, but if not happy to dig more with the reference file.
thanks Brad, I will give this a try. One quick question, how do I use the {fix_ambig} pipe? I see it defined as "vcfutils" and imported into python, but couldnt find the module.
On 1/7/15 9:17 AM, Brad Chapman wrote:
Scott; I've been fruitlessly trying to dig for an error in this file and haven't been able to find anything yet after running through a few different validation tools. So apologies, I'm still not sure what is going on. To reproduce it in the contect of bcbio.variation I'd also need the reference file, since it looks like a custom assembly you might have locally. You can send it to me off list via Dropbox if it's not too large.
Practically, these inputs do need to be run through some post-processing to normalize and split MNPs into individual SNPs/indels. Here are the post-calling commands we use in bcbio for this:
Doing that might help resolve your issue if it's somehow related to MNPs, but if not happy to dig more with the reference file.
— Reply to this email directly or view it on GitHub https://github.com/chapmanb/bcbio.variation/issues/22#issuecomment-69026876.
Every gun that is made, every warship launched, every rocket fired signifies,in the final sense, a theft from those who hunger and are not fed, those who are cold and are not clothed. This world in arms is not spending money alone. It is spending the sweat of its laborers, genius of its scientists, the hopes of its children. --Dwight D. Eisenhower
Scott; I don't think that step is essential, unless your reference genome has non-N ambiguous bases, so you could skip it. If you want to include it, it's only a big nasty awk command defined here:
Hope this helps fix the underlying issues.
Hi Brad, I reran the same files with 0.2.1 and received a different set of errors. It again halted during the processing/normalization of the freebayes joint calling file. If I was only going to use SNPs and not indels is it OK to remove indels/MNPs from the input files of (HaplotypeCaller, UnifiedGenotyper, Freebayes) prior to running ensemble? This set of file w/ SNPs only runs to completion. Since you closed the other issue I opened this as a new issue. Let me know if the offending input file is needed, it is 150mb but I can upload it to dropbox and provide a link. thanks, scott
Progress.log: 2015-01-05T00:16:50 :: State :begin :: {:desc "Starting variation analysis"} 2015-01-05T00:16:50 :: State :clean :: {:desc "Cleaning input VCF: combo"} 2015-01-05T02:03:45 :: State :merge :: {:desc "Merging multiple input files: combo"} 2015-01-05T02:03:45 :: State :prep :: {:desc "Prepare VCF, resorting to genome build: combo"} 2015-01-05T02:10:14 :: State :normalize :: {:desc "Normalize MNP and indel variants: combo"} 2015-01-05T02:10:14 :: State :clean :: {:desc "Cleaning input VCF: gatk-hc"} 2015-01-05T03:51:01 :: State :merge :: {:desc "Merging multiple input files: gatk-hc"} 2015-01-05T03:51:01 :: State :prep :: {:desc "Prepare VCF, resorting to genome build: gatk-hc"} 2015-01-05T03:56:43 :: State :normalize :: {:desc "Normalize MNP and indel variants: gatk-hc"} 2015-01-05T03:56:43 :: State :clean :: {:desc "Cleaning input VCF: gatk-ug"} 2015-01-05T06:05:23 :: State :merge :: {:desc "Merging multiple input files: gatk-ug"} 2015-01-05T06:05:23 :: State :prep :: {:desc "Prepare VCF, resorting to genome build: gatk-ug"} 2015-01-05T06:11:33 :: State :normalize :: {:desc "Normalize MNP and indel variants: gatk-ug"} 2015-01-05T06:11:33 :: State :clean :: {:desc "Cleaning input VCF: freebayes"} 2015-01-05T09:06:19 :: State :merge :: {:desc "Merging multiple input files: freebayes"} 2015-01-05T09:06:19 :: State :prep :: {:desc "Prepare VCF, resorting to genome build: freebayes"}
Stack: Exception in thread "main" java.lang.IndexOutOfBoundsException at clojure.lang.PersistentVector.arrayFor(PersistentVector.java:107) at clojure.lang.PersistentVector.nth(PersistentVector.java:111) at clojure.lang.RT.nth(RT.java:763) at bcbio.variation.normalize$fix_vcf_line$fix_info651.invoke(normalize.clj:273) at bcbio.variation.normalize$fix_vcf_line.invoke(normalize.clj:280) at bcbio.variation.normalize$vcf_by_chrom$write_by_chrom662.invoke(normalize.clj:292) at bcbio.variation.normalize$vcf_by_chrom$fn665.invoke(normalize.clj:306) at bcbio.variation.normalize$vcf_by_chrom.invoke(normalize.clj:300) at bcbio.variation.normalize$write_prepped_vcf.invoke(normalize.clj:345) at bcbio.variation.normalize$prep_vcf.doInvoke(normalize.clj:369) at clojure.lang.RestFn.invoke(RestFn.java:969) at bcbio.variation.combine$dirty_prep_work.invoke(combine.clj:147) at bcbio.variation.combine$gatk_normalize.invoke(combine.clj:187) at bcbio.variation.compare$prepare_vcf_calls$fn7526.invoke(compare.clj:120) at clojure.core$map$fn4207.invoke(core.clj:2487) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:60) at clojure.lang.Cons.next(Cons.java:39) at clojure.lang.PersistentVector.create(PersistentVector.java:51) at clojure.lang.LazilyPersistentVector.create(LazilyPersistentVector.java:31) at clojure.core$vec.invoke(core.clj:354) at bcbio.variation.compare$prepare_vcf_calls.invoke(compare.clj:121) at bcbio.variation.compare$variant_comparison_from_config$iter75827586$fn__7587.invoke(compare.clj:255) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:60) at clojure.lang.RT.seq(RT.java:484) at clojure.core$seq.invoke(core.clj:133) at clojure.core$tree_seq$walk4647$fn__4648.invoke(core.clj:4475) at clojure.lang.LazySeq.sval(LazySeq.java:42) at clojure.lang.LazySeq.seq(LazySeq.java:60) at clojure.lang.LazySeq.more(LazySeq.java:96) at clojure.lang.RT.more(RT.java:607) at clojure.core$rest.invoke(core.clj:73) at clojure.core$flatten.invoke(core.clj:6478) at bcbio.variation.compare$variant_comparison_from_config.invoke(compare.clj:254) at bcbio.variation.ensemble$consensus_calls.invoke(ensemble.clj:113) at bcbio.variation.ensemble$_main.doInvoke(ensemble.clj:133) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invoke(core.clj:617) at bcbio.variation.core$_main.doInvoke(core.clj:35) at clojure.lang.RestFn.applyTo(RestFn.java:137)