In the course of writing the ticket below, I can to the conclusion that it is actually not useful to distinguish between a criterion being unmet due to insufficient data versus the criterion being definitive refuted - at least not from the perspective of using the refuted criterion as meaningful evidence for a target VariantInterpretation.
Feel free to skip to the punchline in the last paragraph - but I recorded my complete thoughts and path toward changing my tune here, as it may prove useful for others.
There has been recent debate about the practicality and utility of distinguishing between the notions of a criterion being unmet-insufficient (when there is not sufficient information to determine if the criterion is met), versus unmet-refuted (when there is sufficient information to definitively say that the criterion is not met).
One argument against making this distinction was that the ACMG calculator does not use info about unmet criteria so it is not necessary to capture in this context. My counter to this is that in a broader context, outside of this framework, this distinction provides very useful information that is relevant to evaluating the validity of a target VariantInterpretation. Capturing it presents a fuller picture of the evidence for a given VariantInterpretation, that can be used by other types of evaluation frameworks outside the ACMG - e.g. those being developed by Monarch/SEPIO.
The value of this distinction of course is that an unmet-refuted outcome means that the CrterionAssessment can be used as meaningful as evidence to evaluate the target VariantInterpretation represents - i.e. the outcome is strong enough to impact the probability of accepting the proposition put forth in the VariantInterpretation. For example, consider a VariantInterpretation asserting a variant to be pathogenic, and a PM2 (absent in control pops) CriterionAssessment that is unmet (because the variant was present at significant frequency in general populations). If we know that it is unmet definitively (i.e. unmet-refuted, rather than unmet because of insufficient information), then this becomes meaningful as evidence that disputes to some degree the validity of the pathogenic VariantInterpretation. If we only know that it is unmet, then we are unable to use this information as evidence in evaluating the VariantInterpretation.
Now of course, many criterion such as the PM2 example above have counterparts with the opposite default interpretation/direction. For example, PM2 (absent in control pops) has BA1 (present above 5% in population) as counterpart. Saying that PS2 is refuted is roughly the same as saying that BA1 is met - so recording both is duplicative. In such cases, it is less important to know that a P criterion is refuted because there will presumably being a corresponding B criterion that is met and gives us the same information. Other examples of paired counterparts are PP1 and BS4 (does/does not segregate with disease), and PS3 and BS3 (damaging/not damaging effect demonstrated by functional studies).
But many criterion don't have counterparts - such that we miss meaningful information not capturing that a criterion is definitively unmet (i.e. refuted).
OR DO WE?? . . . Below I evaluate each criteria that is not paired with a counterpart, do determine if refution of these criteria provides meaningful evidence against the criterion's default interpretation (i realize that 'refution' is not a word, but using it anyway :)
Criteria with no counterpart:
PVS1 - Null Variant in known LOF genes: refution based on not being predicted as 'null' would presumably be complemented with a B criteria that describes non-null impact. If refuted because not in a known LOF gene, this provides no definitive evidence because the gene may become linked to a disease in the future
PS1 - same as previous pathogenic amino acid change: refution provides no meaningful evidence against a pathogenic interpretation (just because the affected aa not currently known to be pathogenic, doesn’t mean if wont one day be so)
PS2/PM6 - confirmed de novo / assumed de novo: refution provides no meaningful evidence against a pathogenic interpretation . . . the fact that a variant is not de novo doesn’t provide evidence against pathogenicity
PM1 - Located in hot spot w/out benign variation: refution provides no meaningful evidence against a pathogenic interpretation (just because the affected region is not currently a known hotspot, doesn’t mean it isn't one. and even if it is not, this provides no meaningful evidence against pathogenicity (affecting a hotspot is not necessary condition for pathogenicity)
PM5 - Located in hot spot w/out benign variation: refution provides no meaningful evidence against a pathogenic interpretation (affecting a known pathogenic aa is not a necessary condition for pathogenicity - so no evidence provided by this being refuted)
PP4 - disease specific phenotype and family history: refution provides no meaningful evidence against a pathogenic interpretation (patient exhibiting disease specific phenotypes is not a necessary condition for a variant's pathogenicity, so no evidence provided by this being refuted)
BP5 - alternate cause of disease: refution provides no meaningful evidence against a pathogenic interpretation (finding an known alternate cause of the disease is not a necessary condition for benign-ness, so no evidence provided by this being refuted . . . i.e. the fact that no alternate cause was found does not meaningfully argue against the benign-ness of the variant)
BP7 - silent variant no splicing impact and low conservation: refution based on variant having such predicted impacts would presumably be complemented with a P criteria that describes such predicted impacts
Punchline: The take home from my analysis above of the ACMG criteria not paired with a counterpart in opposite direction (P vs B) is that the ACMG criteria are smart/complete enough that if the creators thought that that refution of a criteria signified evidence in the other direction (i.e. refuition of a P criteria provided meaningful evidence against pathogenicity), then a counterpart criteria was created to capture this. From my analysis - all criteria for which there was no counterpart are cases where refution of the criteria provides no meaningful evidence against the default interpretation (P vs B). If this is not the case, perhaps the solution is to create a new criteria. rather than force capture of refution.
In the course of writing the ticket below, I can to the conclusion that it is actually not useful to distinguish between a criterion being unmet due to insufficient data versus the criterion being definitive refuted - at least not from the perspective of using the refuted criterion as meaningful evidence for a target VariantInterpretation.
Feel free to skip to the punchline in the last paragraph - but I recorded my complete thoughts and path toward changing my tune here, as it may prove useful for others.
There has been recent debate about the practicality and utility of distinguishing between the notions of a criterion being
unmet-insufficient(when there is not sufficient information to determine if the criterion is met), versusunmet-refuted(when there is sufficient information to definitively say that the criterion is not met).One argument against making this distinction was that the ACMG calculator does not use info about unmet criteria so it is not necessary to capture in this context. My counter to this is that in a broader context, outside of this framework, this distinction provides very useful information that is relevant to evaluating the validity of a target VariantInterpretation. Capturing it presents a fuller picture of the evidence for a given VariantInterpretation, that can be used by other types of evaluation frameworks outside the ACMG - e.g. those being developed by Monarch/SEPIO.
The value of this distinction of course is that an
unmet-refutedoutcome means that the CrterionAssessment can be used as meaningful as evidence to evaluate the target VariantInterpretation represents - i.e. the outcome is strong enough to impact the probability of accepting the proposition put forth in the VariantInterpretation. For example, consider a VariantInterpretation asserting a variant to be pathogenic, and a PM2 (absent in control pops) CriterionAssessment that is unmet (because the variant was present at significant frequency in general populations). If we know that it is unmet definitively (i.e. unmet-refuted, rather than unmet because of insufficient information), then this becomes meaningful as evidence that disputes to some degree the validity of the pathogenic VariantInterpretation. If we only know that it is unmet, then we are unable to use this information as evidence in evaluating the VariantInterpretation.Now of course, many criterion such as the PM2 example above have counterparts with the opposite default interpretation/direction. For example, PM2 (absent in control pops) has BA1 (present above 5% in population) as counterpart. Saying that PS2 is refuted is roughly the same as saying that BA1 is met - so recording both is duplicative. In such cases, it is less important to know that a P criterion is refuted because there will presumably being a corresponding B criterion that is met and gives us the same information. Other examples of paired counterparts are PP1 and BS4 (does/does not segregate with disease), and PS3 and BS3 (damaging/not damaging effect demonstrated by functional studies).
But many criterion don't have counterparts - such that we miss meaningful information not capturing that a criterion is definitively unmet (i.e. refuted).
OR DO WE?? . . . Below I evaluate each criteria that is not paired with a counterpart, do determine if refution of these criteria provides meaningful evidence against the criterion's default interpretation (i realize that 'refution' is not a word, but using it anyway :)
Criteria with no counterpart:
PVS1 - Null Variant in known LOF genes: refution based on not being predicted as 'null' would presumably be complemented with a B criteria that describes non-null impact. If refuted because not in a known LOF gene, this provides no definitive evidence because the gene may become linked to a disease in the future
PS1 - same as previous pathogenic amino acid change: refution provides no meaningful evidence against a pathogenic interpretation (just because the affected aa not currently known to be pathogenic, doesn’t mean if wont one day be so)
PS2/PM6 - confirmed de novo / assumed de novo: refution provides no meaningful evidence against a pathogenic interpretation . . . the fact that a variant is not de novo doesn’t provide evidence against pathogenicity
PM1 - Located in hot spot w/out benign variation: refution provides no meaningful evidence against a pathogenic interpretation (just because the affected region is not currently a known hotspot, doesn’t mean it isn't one. and even if it is not, this provides no meaningful evidence against pathogenicity (affecting a hotspot is not necessary condition for pathogenicity)
PM5 - Located in hot spot w/out benign variation: refution provides no meaningful evidence against a pathogenic interpretation (affecting a known pathogenic aa is not a necessary condition for pathogenicity - so no evidence provided by this being refuted)
PP4 - disease specific phenotype and family history: refution provides no meaningful evidence against a pathogenic interpretation (patient exhibiting disease specific phenotypes is not a necessary condition for a variant's pathogenicity, so no evidence provided by this being refuted)
BP5 - alternate cause of disease: refution provides no meaningful evidence against a pathogenic interpretation (finding an known alternate cause of the disease is not a necessary condition for benign-ness, so no evidence provided by this being refuted . . . i.e. the fact that no alternate cause was found does not meaningfully argue against the benign-ness of the variant)
BP7 - silent variant no splicing impact and low conservation: refution based on variant having such predicted impacts would presumably be complemented with a P criteria that describes such predicted impacts
Punchline: The take home from my analysis above of the ACMG criteria not paired with a counterpart in opposite direction (P vs B) is that the ACMG criteria are smart/complete enough that if the creators thought that that refution of a criteria signified evidence in the other direction (i.e. refuition of a P criteria provided meaningful evidence against pathogenicity), then a counterpart criteria was created to capture this. From my analysis - all criteria for which there was no counterpart are cases where refution of the criteria provides no meaningful evidence against the default interpretation (P vs B). If this is not the case, perhaps the solution is to create a new criteria. rather than force capture of refution.