Coverage calculation issue

[adongare]

Summary

There seems to be some issue with the overage calculation. This measure(bundles attached) is supposed to have 100% coverage but for some reason looks like a couple of clauses are not being included while calculating coverage which drops the coverage by one percentage.

External Tracking Ticket

https://oncprojectracking.healthit.gov/support/browse/BONNIEMAT-1617

Expected Behavior

100% coverage

Version or Commit

v1.3.1

Inputs (e.g. Measure Bundle, Patient Bundle, CQL Library)

Measure bundle: CMS108-measure.zip Patient bundles: CMS108-patients.zip

Relevant Calculation Options (e.g. Measurement Period, meta.profile Validation)

{
    includeClauseResults: false,
    trustMetaProfile: true,
    buildStatementLevelHTML: true,
    measurementPeriodStart: 1/1/2025,
    measurementPeriodEnd: 12/31/2025,
}

cql-to-elm Version Used for Measure Logic Translation (if known)

v2.11.0

Any Additional Info

[hossenlopp]

Hi @adongare!

Thanks for adding all the data we need for looking into this. On a quick look, this may be an issue with the coverage highlighting and perhaps a small lack in test data that isn't apparent in the highlighting. Our team is starting a new sprint this week and will be prioritizing this issue.

[hossenlopp]

Hi @adongare,

It looks like there are some visual coverage issues in this case that were hiding three uncovered clauses. The coverage percentage calculation is correct and the user should be able to get to 100% by adding test cases to cover the following cases:

One Clause in define "Intervention Comfort Measures":

The InterventionRequest.doNotPerform is not covered by the test cases.

define "Intervention Comfort Measures":
  ( ["ServiceRequest": code in "Comfort Measures"] InterventionRequest
      where InterventionRequest.intent in { 'order', 'original-order', 'reflex-order', 'filler-order', 'instance-order' }
      and InterventionRequest.status in { 'active', 'on-hold', 'completed'}
      and InterventionRequest.doNotPerform is not true
  )
    union ( ["Procedure": "Comfort Measures"] InterventionPerformed
        where InterventionPerformed.status in { 'completed', 'in-progress' }
    )

There is no test case that has a "Comfort Measure" ServiceRequest with doNotPerform set to true. Adding a test that has this field set to true would cover the InterventionRequest.doNotPerform clause.

Two Clauses in define "Encounter with Prior or Present Diagnosis of Atrial Fibrillation or Prior Diagnosis of VTE":

Two code comparisons are not being fulfilled in this define statement in the third part of the union that makes up the statement. VTEDiagnosis.clinicalStatus ~ QICoreCommon."remission" and VTEDiagnosis.clinicalStatus ~ QICoreCommon."resolved" clauses are not covered, specifically the (~) equivalence operators.

union ( VTE."Encounter with Age Range and without VTE Diagnosis or Obstetrical Conditions" QualifyingEncounter
    with ["Condition": "Venous Thromboembolism"] VTEDiagnosis
        such that ( VTEDiagnosis.clinicalStatus  ~ QICoreCommon."inactive"
            or VTEDiagnosis.clinicalStatus  ~ QICoreCommon."remission"
            or  VTEDiagnosis.clinicalStatus ~ QICoreCommon."resolved"
        )
        and VTEDiagnosis.verificationStatus is not null and VTEDiagnosis.verificationStatus ~ QICoreCommon."confirmed"
        and VTEDiagnosis.onset.toInterval() before start of QualifyingEncounter.period
)

The test cases only satisfy the VTEDiagnosis.clinicalStatus ~ QICoreCommon."inactive" case. The "remission" and "resolved" cases can be covered by adding test data that uses those codes for the clinicalStatus element.

Coverage Highlighting Issues

While the user can get to 100% with adding test cases, there are still issues with the coverage highlighting that makes it hard for them to have known where the uncovered clauses are. We are continuing to investigate and will give an update when we have a fix.

Best, Chris

[adongare]

Hi @hossenlopp, thank you so much for the thorough analysis. We'll go ahead and add the test cases for the above scenarios.

[adongare]

Hi @hossenlopp,

The user was able to get the coverage up by adding tests to cover the clauses that you mentioned above.

There is one more measure having a similar issue(measure and patient bundles attached below). After debugging a bit, I think the following clauses are not being covered but are highlighted incorrectly:

[
  {
    "raw": [],
    "statementName": "Pharmacological or Mechanical VTE Prophylaxis Received",
    "libraryName": "IntensiveCareUnitVenousThromboembolismProphylaxisFHIR",
    "localId": "298",
    "final": "UNHIT"
  },
  {
    "statementName": "Encounter with VTE Prophylaxis Received on Day of or Day After First ICU Stay or Procedure",
    "libraryName": "IntensiveCareUnitVenousThromboembolismProphylaxisFHIR",
    "localId": "349",
    "final": "UNHIT"
  },
  {
    "statementName": "Encounter with VTE Prophylaxis Received on Day of or Day After First ICU Stay or Procedure",
    "libraryName": "IntensiveCareUnitVenousThromboembolismProphylaxisFHIR",
    "localId": "350",
    "final": "UNHIT"
  },
  {
    "raw": [],
    "statementName": "No VTE Prophylaxis Medication Administered or Ordered",
    "libraryName": "IntensiveCareUnitVenousThromboembolismProphylaxisFHIR",
    "localId": "683",
    "final": "UNHIT"
  }
]

I couldn't relate them to CQL and ask the user to add test cases for them. I do not have measure names yet but I came to know that there are more such measures. I may attach them once I get them. Could you please take a look at the following measure? Thank you so much for your help on this

Measure and Test case bundles: CMS190-measure.zip CMS190-patients.zip

EDIT
CMS 71 CMS71-measure.zip CMS71-patients.zip

CMS 104 CMS104-measure.zip CMS104-patients.zip

[hossenlopp]

Hi @adongare!

Thank you for attaching the additional measures and tests cases. We are continuing to look into these issues.

[jkotanchik-SB]

Hi @hossenlopp,

Another group of users are reporting coverage issues. Would you prefer I attach the materials to this issue, or open new issues?

There are 7 measures mentioned in the report and are all some combination of <100% coverage with unhighlighted aliases, union operators, and/or attributes (that are covered in one or more test cases).

[hossenlopp]

Hi @jkotanchik-SB,

Please open new issues as this will help with organization of efforts and status reporting.

[hossenlopp]

Hi @adongare,

We have just released v1.3.2 which fixes an issue with visual coverage and should now properly show locations in the logic that are uncovered in many cases. This update should make it clear where those last few uncovered clauses are and aid in creating additional test cases. Note: This does not fix visual coverage issues with some alias, unions and library usages. These do not affect the coverage percentage calculation.

Please test with this release and open a new issue if there are additional problems with coverage or you are unable to get to 100% after satisfying the clauses that will now properly show as uncovered.