pkrusche
commented
8 years ago The reason that there are very short confident blocks in PG is that hom-ref calls and calls with an alt allele are validated in a different way:
- hom-ref calls come from an intersection of hom-ref call regions across the pedigree -- everything that is called 0/0 across the pedigree in at least 2/3 aligners becomes "confident hom-ref"
- variant calls are validated against the pedigree phasing information. If we can phase them without getting any GT conflicts with our inferred haplotypes in the pedigree, they become confident variant calls
The very short confident regions will mostly come from places where we were able to make a confident variant call, but where we failed to call the surrounding region hom-ref potentially due to low depth in one sample / alignment noise / ... Since the input into PG comes from unfiltered variant calls, we are likely to see this type of scenario quite frequently. Ideally, we should also validate the reference bases around this variant call (we are working on this).
I think the majority of these cases will be SNPs or small indels which end up with a clear TP/FP/FN classification just based on the genotype. If the genotype is wrong here (0/1 instead of 1/1, ...), then the call should be counted as a kind of FP, not an UNK.
Things get trickier in places where we can only make part of a complex call in a pedigree-consistent fashion, and have surrounding calls as truth-rejects -- here, I think a stronger argument could be made for leaving such query calls as UNK. Maybe one way to address this would be to have "uncertain" regions to flag places where we have rejected truth variants, but weren't quite sure. However, this type of region can make the comparison problem much harder.
RebeccaTruty
jzook
@jzook said in #4:
And @pkrusche also said:
This is quite interesting, and I've not previously thought a great deal about the confident regions, and had assumed that they were relatively few in number so that these edge effects would be fairly minor. I was surprised to find that the confident regions BED that comes with Illumina PG 8.0.1 truth set (what I have handy) contains 5.8 million regions (and this for only 4.5 million variants in the corresponding truth VCF).
The confident regions are being used to say: "Within the regions, we are confident that the truth set contains all the variants." The implication is that the genomic sequence outside of the regions could not be confidently determined (and so may be variant or non-variant). What are some of the reasons why regions are excluded?
When doing comparisons near the edges of these regions, some of the cases we need to be careful about (there are more, but let's just worry about these):
a) A call made outside the confident region could not be matched to a truth variant (thus assigned match status UNK). (These may or may not match truth-rejects.)
b) A truth variant can be matched by a call made outside the confident region. (This should be a TP, but which pre-filtering calls to the regions would incorrectly result in an FN)
c) A call made inside the confident region could not be matched to a truth variant. (Seems like this should be a FP, but hang on, what if the call would be equivalent to a truth-reject -- it should be UNK, right? What if it was "nearly equivalent" such as a zygosity error -- should also be UNK since we're not confident in the truth-reject correctness?)
Looking at the distribution of region lengths, the Illumina PG 8 dataset has over 600k confident regions that are a single base long and another 340k that are only two bases long. Why are they so short? How can you be confident at a single base and yet not be confident at the bases on either side? These locations seem like they would be vulnerable to pre-filtering FN (case b above).
Similarly, this dataset has 3.6 million cases where the gap between consecutive confident regions is only a single base long (and another 400k where the gap is 2 bases long). This suggests the position is being excised due to call inconsistency during truth set construction. These cases seem like they would be very vulnerable to generating FP during comparison time for any callers that don't happen to use the same variant representation as went into the PG construction (whereas a caller using the same positional representation would get UNK instead) (case c above). This is regardless of whether region filtering is pre or post comparison, with the exception that it turns out that PG 8 has 200k "silver" variants that are outside the confident regions (which will alleviate things for alternate representations that do match these decoys, but it seems like a drop in the bucket).
It would be interesting to look further at the magnitude of these effects. Just going by the numbers above, case c should be four times larger than case b. Case b is the easier to evaluate, just compare pre and post filtering results, although as mentioned above I'm quite sceptical of those confident regions that are only one or two bases long in the first place. Case c effects, is harder. It would also be interesting to give truth variants a score which is the distance from the variants to their nearest confident region boundary, and plot a "ROC curve" with this score - I would assume that those with a larger distance are more likely to be TP in a typical query set.