projecttacoma / fqm-execution

fqm-execution is a library that allows users to calculate FHIR-based electronic Clinical Quality Measures (eCQMs) and retrieve the results in a variety of formats
https://projecttacoma.github.io/fqm-execution/
Apache License 2.0
18 stars 6 forks source link

Add regression script for comparing previous calculation results #217

Closed mgramigna closed 1 year ago

mgramigna commented 1 year ago

Summary

Adds scripts for comparing calculation results across ecqm-content repos. See additions to contributing.md for details on what exactly is happening.

New behavior

None in source code, new script exists.

Code changes

Le script. Update docs

Testing guidance

I've simulated an error on the simulated-regression-error branch, where for a given calculation, there is a 25% chance that it will incorrectly set numer/denom to true when IPP is false. If you run the script and get no failures at all, go buy a lottery ticket! To test:

1) git checkout simulated-regression-error 2) Run the following:

./regression/run-regression.sh -b add-regression-script

This will run regression with this PR's branch as the base, and the error branch as the "one under test." Conceptually this is simulating a scenario where you might be reviewing the code on simulated-regression-error and want to make sure that it didn't change any functionality from its base branch (in this case add-regression-script). In general the base branch will be master, but right now it won't work against master since the regression script isn't checked in there yet.

You should get some errors (this won't be idempotent since errors are random):

This indicates that calculation results differ across the branches

  1. To get more info, use -v during the regression run which will print the diffs of the actual results (NOTE: the errors you get will be different than step 2 since it's random):
./regression/run-regression.sh -b add-regression-script -v

Diffs will be printed to the terminal in realtime:

image

github-actions[bot] commented 1 year ago

Coverage report

St.:grey_question:
Category Percentage Covered / Total
🟢 Statements 85.39% 2023/2369
🟡 Branches 73.97% 1802/2436
🟢 Functions 87.5% 357/408
🟢 Lines 85.69% 1952/2278

Test suite run success

379 tests passing in 29 suites.

Report generated by 🧪jest coverage report action from 0d309b7179ef7f0079b89b7d2856b4e927635ffd