Closed cbizon closed 3 years ago
I'll summarize my thoughts from last week's call. I'd welcome input from those who see things differently from me.
There are two pieces of information regarding the outcome of a CriterionAssessment. Firstly, whether the criteria is satisfied or not. Secondly, if satisfied, then the "weight" which which the evaluated criteria would be used in evaluating pathogenicity in the overall framework. The latter is needed within the ACMG interpretation framework because the guidelines allow for an analyst to apply a criteria but give it a different weight than usual:
To provide critical flexibility to variant classification, some criteria listed as one weight can be moved to another weight using professional judgment, depending on the evidence collected. For example, rule PM3 could be upgraded to strong if there were multiple observations of detection of the variant in trans (on opposite chromosomes) with other pathogenic variants (see PM3 BP2 cis/trans Testing for further guidance).
@cbizon and I had a subsequent discussion off-line (without reaching a conclusion) about whether there should be some variability in representation of "not satisfied", for representing the two cases of:
So, my proposal would be the following:
CriterionAssessment.value
Where "Criterion not met" and "Unable to evaluate" are basically considered "identity" weights (that is, 0 if using an additive analogy, or 1 if using a multiplicative analogy).
Great write-up. I particularly liked your comments regarding interdependent attributes and “model smell”. We seem to have a lot of that in HL7 and it is sometimes hard to find the balance between model elegance and practicality.
It’s hard to argue against the approach you describe unless more information needs to be captured about why "Criterion not met" and "Unable to evaluate" were selected. Do we need to support Evidence for those options? You gave great examples:
• There is insufficient information to evaluate the criterion (e.g. there is no reasonable estimate of populate prevalence) • There is sufficient info to evaluate, and the criterion is clearly not met (e.g. the condition is very rare and the general population frequency of an allele is some crazy high number like >5%).
If this is the case, one way to approach the model without creating interdependent fields is to define the different types of possible results of a CriterionAssessment: met, not met, unable to evaluate. Those 3 types could be specializations of a more generalized CriterionAssessmentOutcome class, which is associated to CriterionAssessment and has a way of capturing supporting Evidence. The “met” class (only) could have a required association to a “result” class that contains an attribute for the PVS..Benign value set and an attribute for “weight”. Any other attributes that might be needed could be added to the respective classes.
I hope this makes sense – I tried to crank this out before my next meeting. Note that I’m not necessarily arguing for this approach, rather I’m offering it as an option if we wanted a bit more flexibility in the model (at the expense of complexity).
Thanks, Bob
From: Bradford Powell [mailto:notifications@github.com] Sent: Monday, May 30, 2016 8:01 PM To: clingen-data-model/clingen-data-model Subject: Re: [clingen-data-model/clingen-data-model] CodeableConcept for CriterionAssessment.outcome (#175)
I'll summarize my thoughts from last week's call. I'd welcome input from those who see things differently from me.
There are two pieces of information regarding the outcome of a CriterionAssessment. Firstly, whether the criteria is satisfied or not. Secondly, if satisfied, then the "weight" which which the evaluated criteria would be used in evaluating pathogenicity in the overall framework. The latter is needed within the ACMG interpretation framework because the guidelines allow for an analyst to apply a criteria but give it a different weight than usual:
To provide critical flexibility to variant classification, some criteria listed as one weight can be moved to another weight using professional judgment, depending on the evidence collected. For example, rule PM3 could be upgraded to strong if there were multiple observations of detection of the variant in trans (on opposite chromosomes) with other pathogenic variants (see PM3 BP2 cis/trans Testing for further guidance).
(Richards et al 2015http://www.nature.com/gim/journal/v17/n5/full/gim201530a.html)
@cbizonhttps://github.com/cbizon and I had a subsequent discussion off-line (without reaching a conclusion) about whether there should be some variability in representation of "not satisfied", for representing the two cases of:
So, my proposal would be the following:
Where "Criterion not met" and "Unable to evaluate" are basically considered "identity" weights (that is, 0 if using an additive analogy, or 1 if using a multiplicative analogy).
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/clingen-data-model/clingen-data-model/issues/175#issuecomment-222571582, or mute the threadhttps://github.com/notifications/unsubscribe/ALAhBkAuALH5EV88FSulnY71ZaHaqIehks5qG4hlgaJpZM4IifVh.
Sorry to jump into the discussion late but does anyone know if these values have associated LOINC codes? Free text values are good for visualization, but codes will be important as well.
I completely agree that coded values would be best.
As far as I am aware, these terms are not in LOINC. We could talk to LOINC about adding them, but LOINC operates more at the test interpretation/result level; the assessment of individual variants might be too low level for them. That said, they might be interested in creating a value set or two to support ClinGen. ☺ I’d be happy to mediate those interactions – I just did the same for CPIC (PGx interpretations).
From: Shawn Rynearson [mailto:notifications@github.com] Sent: Tuesday, May 31, 2016 12:04 PM To: clingen-data-model/clingen-data-model Cc: rrfreimuth; Comment Subject: Re: [clingen-data-model/clingen-data-model] CodeableConcept for CriterionAssessment.outcome (#175)
Sorry to jump into the discussion late but does anyone know if these values have associated LOINC codes? Free text values are good for visualization, but codes will be important as well.
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/clingen-data-model/clingen-data-model/issues/175#issuecomment-222753293, or mute the threadhttps://github.com/notifications/unsubscribe/ALAhBgROZdPns8aXimPdedBBOBS2j1W0ks5qHGn3gaJpZM4IifVh.
Thanks for the great discussion today. I think we’re in agreement about the concepts in the model (which is always helpful), regardless of whether we represent them one way or another. A slightly updated (and clarified?) diagram is attached (and checked into the VP server).
[cid:image001.png@01D1BC10.91BCC060]
Restating the question on the table: Should we capture the outcome of an assessed criterion in a single field (e.g., “pathogenic strong”) or in two fields (e.g., “pathogenic”, “strong”)? The way the information is stored and the way it is displayed could be different, of course.
Fundamentally, I think this is a question of terminology management. In general, pre-coordinating terms can result in headaches later as it makes maintenance and semantic inference more complex, but it can simplify small value sets in the short term. Does ACMG anticipate changes to the set of terms used to describe level of evidence? Precoordination would make mappings between “old” and “new” assessments more difficult.
I parsed paragraph 2 on page 7 of the guidelines https://www.acmg.net/docs/standards_guidelines_for_the_interpretation_of_sequence_variants.pdf
and got the following:
The ACMG criterion classifications are:
Pathogenic or Benign
Pathogenic is used for (pathogenic or likely pathogenic)
Benign is used for (benign or likely benign)
Each Pathogenic criterion is weighted as one of:
Very strong PVS1
Strong PS1-4
Moderate PM1-6
Supporting PP1-5
Each Benign criterion is weighted as one of:
Stand-alone BA1
Strong BS1-4
Supporting BP1-6
The numbering is simply for distinct label names not a reflection of weighting.
Scoring rules use the “satisfied” criterion from above to derive a classification from the five-tier system:
Pathogenic
Likely Pathogenic
Uncertain Significance
Likely Benign
Benign
I do agree that we need a “result” or “outcome” coded value as well as an indicator for both “unmet” (or unsatisfied) and “insufficient data to evaluate” (or unable to evaluate).
I am not convinced that it is worthwhile to model the coded values to parse out these special indicators and the Path v Benign etc…
In the end these “outcome” codes will be very specific to each CriteriaSet or Guideline that gets developed. This ValueSet will be the “ACMG Criteria Assessment Outcome Result” and they should be coded as a Nominal List of Values. I do not see the need for the complexity at this time, modeling it finer will distract adopters and raise complexity. It is not clear that the computational benefits will be worth the cost.
My opinion is for the list to be coded as
--- met outcomes ----
Pathogenic Very Strong
Pathogenic Strong
Pathogenic Moderate
Pathogenic Supporting
Benign Stand-alone
Benign Strong
Benign Supporting
---- not met outcome ----
Criterion Not Met
---- unevaluated outcome ----
Unable to Evaluate (or Assess?)
Comments (or reason/justification) for the final coded outcome will be provided, implementers may choose to require a reason for “unable to evaluate” as an example.
From: rrfreimuth notifications@github.com Reply-To: clingen-data-model/clingen-data-model reply@reply.github.com Date: Wednesday, June 1, 2016 at 3:25 PM To: clingen-data-model/clingen-data-model clingen-data-model@noreply.github.com Subject: Re: [clingen-data-model/clingen-data-model] CodeableConcept for CriterionAssessment.outcome (#175)
Thanks for the great discussion today. I think we’re in agreement about the concepts in the model (which is always helpful), regardless of whether we represent them one way or another. A slightly updated (and clarified?) diagram is attached (and checked into the VP server).
[cid:image001.png@01D1BC10.91BCC060]
Restating the question on the table: Should we capture the outcome of an assessed criterion in a single field (e.g., “pathogenic strong”) or in two fields (e.g., “pathogenic”, “strong”)? The way the information is stored and the way it is displayed could be different, of course.
Fundamentally, I think this is a question of terminology management. In general, pre-coordinating terms can result in headaches later as it makes maintenance and semantic inference more complex, but it can simplify small value sets in the short term. Does ACMG anticipate changes to the set of terms used to describe level of evidence? Precoordination would make mappings between “old” and “new” assessments more difficult.
One thing I would like to add here is that Uncertain Significance can arise in two ways:
1) Uncertain Significance - due to insufficient evidence 2) Uncertain Significance - due to conflicting evidence
I think it should be good idea to separate them to distinguish two types of uncertain significance.
I believe the way we’ve modeled the provenance it should be clear. The Uncertain Significance will be the final interpretation. We are proposing that we have an outcome on the individual assessment of criterion to be “unable to evaluate” and then allow the assessor to add a comment describing why he/she was unable to evaluate (i.e. due to insufficient evidence or due to conflicting evidence or any other reason). We have not decided to code the reasons for “unable to evaluate”.
From: ronakypatel notifications@github.com Reply-To: clingen-data-model/clingen-data-model reply@reply.github.com Date: Wednesday, June 1, 2016 at 5:00 PM To: clingen-data-model/clingen-data-model clingen-data-model@noreply.github.com Cc: "Babb, Larry" larry.babb@gmail.com, Comment comment@noreply.github.com Subject: Re: [clingen-data-model/clingen-data-model] CodeableConcept for CriterionAssessment.outcome (#175)
One thing I would like to add here is that Uncertain Significance can arise in two ways:
1) Uncertain Significance - due to insufficient evidence 2) Uncertain Significance - due to conflicting evidence
I think it should be good idea to separate them to distinguish two types of uncertain significance.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
What are the values for this?
Is it something like: Pathogenic Very Strong Pathogenic Strong ... Benign Strong Not Applicable?