Predicted Functional Impact Annotation Definition and Scope

[mbrush]

We will initially proceed with our initial decision to split 'Predicted' (#21) from 'Experimental' (#34) Functional impact annotations - and model these as separate VA types. Our rationale was that:

• they have very different provenances and evidence types (computational vs experimentally validated)
• they generally make statements about functionality at different levels of granularity
• there is a fairly clean separation in terms of the processes, tools, and sources for these VA types
• there is a clear understanding in the community w.r.t. how they are different and how they can be used

The proposals/notes below are derived from the initial requirements work for this VA type here.

---

Definition: A statement generated by a computational algorithm that predicts the impact a variant has on the functionality or behavior of a gene product (e.g. 'deleterious', 'damaging', 'tolerated').

Scope/Comments:

• These statements are always in silico predictions of expected impact generated by computational algorithms, and are not based on direct experimental evidence.
• Prediction algorithms calculate a score, and typically also assign a category to the variant - based on pre-defined score thresholds.
• There are very many algorithms and tools that perform these predictions, and use different scoring systems, thresholds, and categories labels.
• These computational predictions typically describe impact in general terms (e.g. 'deleterious', 'tolerated') - and possibly the extent (e.g. a distinction between 'damaging' and 'deleterious') or likelihood (e.g. 'possibly damaging') of damage.
• This contrasts with most experimentally derived impact statements that make more pointed assertions about things like a variant being neomorphic, dominant negative, or gain of function, or about impact on specific functions of a gene product (e.g. increases kinase activity, alters localization).
• However, among the list of algorithms here are some that make more specific kinds of predictions than simply 'damaging' vs 'tolerated'.
  • e.g. "AutoMute" and "CUPSAT" predict stability changes, and "BeAtMuSiC" predicts changes in protein-protein binding affinity. And some frame impact in terms of pathogenicity (implying some knowledge that the LoF of the affected gene/protein is causal for a disease). e.g. the LRT tool (likelihood ratio test) classifies as 'Disease-Causing' or 'Polymorphism'. . . consider how this might change our definition/description of this VA type.
• The key feature distinguishing these from 'Experimental Functional Impact" annotations is their generation by predictive computational algorithms. This distinction is important because consumers of functional impact annotations trust and apply them differently than experimentally derived annotations.

Sources of more info:

• https://omictools.com/functional-predictions-category
• https://genomeinterpretation.org/impact
• https://link.springer.com/article/10.1186/s40246-017-0104-8 - see table 1
• https://onlinelibrary.wiley.com/doi/full/10.1002/humu.22768 - see table 1
• http://varianttools.sourceforge.net/Annotation/DbNSFP
• http://www.mutationtaster.org/info/documentation.html#output
• The ENIGMA paper "Towards controlled terminology for reporting germline cancer susceptibility variants: an ENIGMA report" has some useful text on this topic (submitted as of March 5 2019).

[mbrush]

Questions for Discussion:

• Shall we maintain the split between Predicted and Experimental Functional Impact annotations? See #34 to compare against examples/scope of 'Experimental' Impact annotations.
• Do we like the name? I like 'Predicted' vs 'Experimental' Functional Impact because it highlights what we saw as the critical distinguishing criteria. Other names to consider? 'Variant Effect Prediction' vs 'Experimental Functional Impact Statement'
• What should be allowed as the variant subject of these annotations? The impact is usually about the protein level alteration, but often made on variants at the genomic or transcript level. In all of these cases, however, I believe that it is a PreciseVariation that is annotated (as opposed to a BucketVariation).
• The type of prediction/algorithm used is an element that is captured by ClinGen. This generally falls into two categories - missense effect and splicing effect. Should this be called out explicitly? If so, how? Chris B noted previously that 'the outcome value sets are going to be disjoint for these two cases'.
• The fact that categorical classifications (e.g. 'deleterious') are assigned by some tools, while others provide just a quantitative score, poses an interesting modeling challenge to consider - as does defining the relationship between a category and the score that supports it. ClinGen's model treated the category as a separate statement than the score - and frames the score as evidence for the categorical assertion. I suspect this may be too complex for our use cases.

[mbrush]

Outcomes/Actions from March 20 VA Call:

• We will initially maintain a split between Predicted and Experimental Impact annotation types.
• We need to settle on a definition - see two proposed above.
• Revisit name of this VA type. There was general agreement that "Computational" or In Silico" Functional Impact may be better than "Predictive" - as this gets more directly at the key distinction from "Experimental", and avoids confusion with use of "predictive" in somatic interpretation space.
• There are some in silico tools that make more specific/pointed types of predictions (beyond a variant simply being 'damaging' or 'tolerated'). For example, predicted expression level changes, predictions that the region hit by the variant may be a regulatory region, predictions that a variant impacts protein stability or binding interactions.
• Action Item: @cibzon will add a list of examples to the ticket so we can discuss what is in scope here, and how this impacts the definition of this VA type and our model - see #35.
• Regarding the subjects of this VA type, tools always assess 'precise' variants. But an agent may come along and look at predictions on many precise variants and decide that an assertion about a 'bucket' variation is warranted (e.g. "all missense mutations in the last exon of gene Y are deleterious". These, however, are not in scope here as they are not purely computational in nature.
• Regarding capturing the algorithm type - we should review the landscape of tools and flesh out the list of two general types ClinGen uses (missense effect and splicing effect) - as there are likely others. Our model can recommend capturing the algorithm type, and provide a list of recommended categories, but initially may not constrain to a specific value set here.

[mbrush]

Elements to capture in a statement model: (based on notes from initial requirements work here)

• variant - typically a precise variant instance, can be at genomic, transcript, or protein level
• categorical prediction - the impact term assigned by the tool based on score threshold/cutoffs
• impact score - the quantitative score calculated by the tool (used as evidence for the prediction?)
• affected gene product - use in cases where a genomic-level variant is specified, and the impact applies to a particular transcript or protein isoform.

---

This is the first VA type where I feel that aligning with the ACM-based approach (casting the elements above into subject, predicate, descriptor and qualifier slots to precisely represent statement semantics) is a bit complicated. The challenge posed is rooted in the fact that an ACM-based model scopes an annotation to contain a single, primary statement with a single descriptor - but there are two elements above that represent descriptors of the variation (the impact score and the categorical prediction) - and sometimes only one or the other is provided. There is no compact way to capture this in a single annotation using the ACM slots (S, P, O Q). And treating the score as evidence for a categorical prediction creates an issue when only a score is given.

_We drafted an initial proposal for an ACM based model in the google doc here. Comments can be added to the doc and we will move the final proposal to the ticket here once it is hardened a bit._

[mbrush]

Listing some high-level evidence and provenance modeling requirements that emerged from review of the competency questions here - as for this particular VA type, I feel like E/P-related information may influence how we scope and structure the primary statement.

• Computational agent/tool making the prediction
• The algorithm a computational tool implements (need to better understand the distinction/relationship between the two)
• The version of tool and algorithm generating a prediction
• The computed score (evidence) from which a categorical prediction is derived
• Key parameters defined in the algorithm implemented/executed by the agent/tool (e.g. threshold distinguishing categories, transcript version the prediction was calculated for)

These requirements focus overwhelmingly on provenance, and minimally on evidence (the score underlying a prediction being the only evidence of import).
A separate ticket will be opened to discuss/document development of the E/P model for this VA type.

[mbrush]

Modeling here is essentially done, with exception of small issue with uncovered in pre-testing with BRCA Exchange data - which revealed an overlooked requirement from our initial analysis.

Our proposed model uses a Computational Impact Data Set as the descriptor (see here). This object is used to bundle the two types of 'data' typically reported in a CFI statement - a categoricalImpact, and an impactScore. But we have no structured way to represent the what type of scores these are, to help users understand their meaning and significance. This is important, given that there are a myriad of different types of computational impact algorithms that use different methods to derive scores describing different aspects of gene product function.

There are different ways the model might capture this important aspect of CFI statements. The red 'impact type' attribute in the proposal here represents one approach (an additional attribute to capture the 'impact type'). Alternatively we might represent the impact score itself as an object (as opposed to a literal) where we could hang this type information. We need to evaluate the adequacy of the proposed and alternate approaches.

[mbrush]

Example of the proposed model used to represent a 'prior probability of pathogenicity' computational impact prediction reported on the BRCA Exchange website here: https://brcaexchange.org/variant/287750.

 - id: ex:Statement001
   type: va:ComputationalFunctionalImpactStatement
   subject: brcaexchange:287750 # BRCA1 NM_007294.3:c.2864C>G
   descriptor: 
      - id: ex:CFIData001
        type: va:ComputationalImpactStudyData
        impactType: 'in silico prior probability of pathogenicity (protein-level estimation)'
        impactScore: 0.99
    method: HCI Breast Cancer Genes Prior Probabilities Algorithm

Note here that we rely on the method being captured to allow user to find out more about the impact type . . .

[mbrush]

On the March 18 VA call, we discussed the possibility of changing the names for the attributes in the Computational Impact Data Set - essentially replacing 'impact' with 'prediction'.

An alternate naming scheme could be:

• impactScore -> predictionScore, or predictedImpactScore
• categoricalImpact -> categoricalPrediction or predictedCategory, or predictedImpactCategory
• impactType -> predictionType

This was motivated by the fact that some CFI statements use categorical terms that don't describe impact on gene function directly (at lease superficially). e.g. the BRCA Exchange example above is called a 'prior probability of pathogenicity' prediction (but under the hood the prediction is about impact on gene function, and this is expressed as a probability that this altered function will be pathogenic).

Using the more explicit/detailed of these labels in the BRCAExchange example above (and adding a fake categorical value to see how all three look together), we would get the following

 - id: ex:Statement001
   type: va:ComputationalFunctionalImpactStatement
   subject: brcaexchange:287750 # BRCA1 NM_007294.3:c.2864C>G
   descriptor: 
      - id: ex:CFIData001
        type: va:ComputationalImpactStudyData
        predictionType: 'in silico prior probability of pathogenicity'
        predictedImpactScore: 0.99
        predictedImpactCategory: 'Pathogenic'
    method: HCI Breast Cancer Genes Prior Probabilities Algorithm

[larrybabb]

Prediction: { impact score, impact category, description/interpretation }. I think it could get confusing to have attributes that end in “xxxType”, as it gets into the concept of “classifying” the predicted impact very specifically.


Isn’t it natural to conflate the “type” attribute with the “predictionType” attribute. Or is this more of a result that the “predicted impact” concept is flattened and thus needs to be qualified as “predictionType”?


the above example shows that the Type of the "descriptor" is "va:ComputationalImpactDataSet". So is that ComputationalImpactDataSet Descriptor holding a complex type called "prediction" that contains a impact category, impact score and a text or coded interpretation of the prediction.

[mbrush]

Thanks @larrybabb, I agree it is still a bit confusing. Will brainstorm more on this, and record alternate proposals here.

[mbrush]

The purpose of the predictionType/impactType field proposed above is to describe the aspect of gene product function or significance that the score is about. Depending on the algorithm generating the prediction, this may be: (1) something very general - e.g. the variant's "generic impact on gene product function" (deleterious vs tolerated); (2) something more pointed - e.g. its "impact on transcription factor activity", "impact on gene product stability"; or (3) translated into other terms - e.g. impact on overall function of a disease-related gene is often framed as "probability of pathogenicity".

Below is a proposal to capture this as the type of algorithm that generated the impact study data - as I think this concept fits cleanly in the context of a ComputationalImpactStudyData object. I also propose new names for the attributes holding the data itself - 'impactScore' and 'impactClassification' (instead of impactCategory).

 - id: ex:Statement001
   type: va:ComputationalFunctionalImpactStatement
   subject: brcaexchange:287750 # BRCA1 NM_007294.3:c.2864C>G
   descriptor: 
      - id: ex:CFIData001
        type: va:ComputationalImpactStudyData
        impactScore: 0.99
        impactClassification: 'Pathogenic'
        algorithmType: 'in silico prior probability of pathogenicity'
    authoredBy: 'HCI Breast Cancer Genes Prior Probabilities Predictor'