wk3 oct_epic: ANDON Appointment Supply DAX Code in 3.0 Release of Team (AGG) Care not consistent w/Prior Releases

[dlkibbe]

1. Add description:

Team data in Aggregate Module for blue team needs review and validation. Appointment Supply numbers in Agg module are considerably lower than in the individual modules of Care Coordination (CC), Psychotherapy (Psy) or Medication Management (MM). The team suggested the Appointment Supply (AS) showing in Agg was less than actual. Screenshots for AS for CC, MM, Psy and Agg for comparison of AS numbers. The AS definitions from the "Descriptions" and "Details" section of the Data UI are pasted below for reference: DETAILS in the Data UI indicate: Appointment Supply (median and 75th percentile) – These estimates were calculated from the weekly visit counts by the patient cohort with the team over one year. The model uses the 75th percentile. DESCRIPTIONS in the Data UI indicate: Appointment Supply (median and 75th percentile) – An estimate of the median total weekly number of medication management appointments available with this team. The estimate is calculated using the volume of medication management visits completed with he team over one year. (appt/wk)

Use of data in SIM UI will begin October 17, 2022.

<< Paste screenshots here>>

1. When was the bug discovered?
   • 9-24-2022 during prep session with Klocek, Nappi, Askerlund, and Kibbe.
2. Is this an invidivual or shared world?
   • not in SIM UI, Data UI
3. What is the name of the experiment?
   • Not an experiment, Data UI only

[lzim]

@emilymetcalf114 & @lzim discussed at support_workflow wk4 sept_epic

Next Steps:

• [x] Evaluate other MTL 3.0 teams for similar discrepancies
• [x] Check the time units at https://mtl.how/demo
• [x] Check the time units at https://mtl.how/data
• [x] @lzim reviwed DataNotes w/Emily and doesn't see the Team (AGG) Care Service Arrays in the Data Notes 1 or 2 tab.
• [x] Identify documentation at https://github.com/lzim/research/tree/master/data_ui

User Persona

• [x] SD Modeler - SDM-DOC
• [x] NIH Program Officer - Open Science; Open Standard SDM-DOC
• [x] @emilymetcalf114 Can a typical Team PSD member find the answer to the time units (unit/wk) for appointment supply in this documentation?

[emilymetcalf114]

First step: @emilymetcalf114 looked at a comparison MTL blue site (Tuesdays at 11am PST) on the morning on 9/29 and witnessed the same effect, where Agg Care appt supply #'s are lower and do not add up to appt supplies of MM, CC, Psy, etc. Psy:

[emilymetcalf114]

@emilymetcalf114 compared with another MTL blue site (Thursdays at 11am PST) on 9/30 am and witnessed the same effect, where Agg Care appt supply #'s are lower and do not add up to appt supplies of MM, CC, Psy, etc.

Psy:

[emilymetcalf114]

Definitions for appt supply are different between CC, MM, etc. and in the Agg care model. Here are the definitions in the Data_UI: CC, MM & Psy:

Agg Care:

[emilymetcalf114]

Documentation from mtl.how/demo:

• CC (v36 2022_01_10) - time unit = weeks

  

• MM (v36 2022_01_11) - time unit = weeks

  

• Psy (v21 2022_02_11) - time unit = weeks

  

• Agg Care (v15 2021_01_20) - time unit = weeks

  

  (equation continued)

  

[lzim]

Thanks for reviewing docuementation @emilymetcalf114 💪🏽 It's very helpful to have it all in one place, and it confirms what I expected was happening. But, wanted to double-check our documentation to confirm, before confusing any learners/facilitators.

In MTL time for these variables will always be in weeks because the clinical decision-makers told us that was the most useful period of time for upgrading team care decisions.

But, for unit checking we have to review and confirm the validity of any unit conversions: unit/wk. For example, appt/wk is not hrs/wk.

From the sim

From your post above

So, the conclusion is that the documentation needs to be clearer that:

CC, MM, PSY are using appt/wk for appointment supply (75%ile, etc.) in any given week and Team (AGG) care is using hrs/wk for appointment supply (see equation from documentation above),

• This makes sense, because this enables experiments standardized to hrs in Team Care given the services have different lengths.

So, the next step would be:

• [x] Confirm that the calculation is working as intended between appt/wk and hrs/wk starting with our Monday team.
• [x] Review our archive in Team PSD > quant_workflow of the highlighted Team (AGG) Care data array information below back in Data Notes in the Data UI (i.e., MTL 3.0). Clearly this documentation is not sufficient for facilitators to read it an understand what it is doing.

@emilymetcalf114 Does that make sense to you?

FYI: @dlkibbe Does this make sense to you?

[emilymetcalf114]

I pulled up the archived MTL 2.0 data_ui documentation for how Agg appointment supply was calculated, and the definition is identical to what we see in the current 3.0 documentation on the data_ui.

[lzim]

@epearman We talked about Story A2 R01 as a top research priority for oct_epic. Once we establish your access, reviewing this operations bug would be helpful!

We want to double-check the estimation procedures for appointment supply between (CC, MM, PSY) Care modules and the Team (AGG) Care module.

FYI: @emilymetcalf114

[ghost]

I see two different things going on.

1) CPT codes used to define the encounter type (CC, MM, and PSY) in the AGG model vs. the individual CC, MM and PSY models. See SQL code here:

• In the AGG model: CPT codes 90833 and 90836 are assigned to both encounter type PSY and MM using a CASE WHEN function (line 165). CASE WHEN functions stop evaluating after the first condition is met so all CPT codes 90833 and 90836 are assigned to encounter type PSY and NOT MM. Just want to check that this is what we want.

  • CASE WHEN cpt.CPTCode in ('90791','90792','99201','99202','99203','99204','99205') THEN 'Intake' WHEN cpt.CPTCode in ('90834','90837','90833','90836') THEN 'PSY' WHEN cpt.CPTCode in ('90853') THEN 'Group' WHEN cpt.CPTCode in ('99212','99213','99214','99215','90833','90836') THEN 'MM' WHEN cpt.CPTCode in ('t1016','90832','90839') THEN 'CC' ELSE 'Adjunct' END AS [EncounterType].

• The final cohorts (after filtering for CPT codes) used for the PSY, CC, and MM appointment supply parameters are different in the aggregate vs the individual models. Therefore, even if appointment supply was reported in appts/week in the AGG model, it would not match exactly appointment supply in the individual model.
  • PSY and CC Individual: CPT codes in (90834 & 90837)
    • PSY in AGG: CPT codes in (90834, 90837, 90833, 90836)
    • CC in AGG: CPT codes in (t1016, 90832, 90839)
  • MM Individual: CPT codes in (90833, 90836, 99212, 99213, 99214, 99215)
    • MM in AGG: CPT codes in (99212, 9913, 99214, 99215)

2. Formula to convert from appts./week to hrs./week in the AGG model. See DAX code here.

• All of the median appointment supply parameters in the AGG model (CC, EBPsy, Group, Intake, MM, and PSY) are calculated in line 38 in each of the "{AGG} [encounter type] Parameters.txt" files. The formula is the same across all files, it takes the median appointment supply in apts/week and multiplies it by 0.25 (15 min visits for each appointment?) to get hours/week. This contradicts the documentation of Appointment Supply Median under "Team Care Details" of the UI.
  • Documentation in UI
  • DAX formula: VAR APPTSUPPLY = IF(COUNTROWS(WEEKLYUSAGE) > 0, IF(ACTUAL50THPERCENTILE <= 0, 0, PERCENTILEX.INC(WEEKLYUSAGE, [VisCount], ACTUAL50THPERCENTILE) * 0.25), 0)
    • This varies from the legacy code line 264, which follows the array documented: select 'Appointment Supply (median)' as Label, [Intake] as intake, [psy] as psy, [ebpsy] as ebpsy, [cc]0.5 as cc, [mm]0.5 as mm , [adjunct]0.25 as adjunct, ([group]1.0/6) as 'group'

Emily's Recommendation for next steps:

1. Review CPT codes used to categorize CC, MM, and PSY in both the individual and the AGG models in the SQL scripts.
2. Review the formula to convert median appointment supply from appts./week to hrs./week in the DAX scripts.

[dlkibbe]

@emilymetcalf114 @lzim Thank you @epearman for the information above. @jwklocek and I shared the information in our team discussion. Questions below from QIICs:

The following are CPT codes for Psychotherapy: 90832, 90834, 90837 (90832 is included in care coordination, although we have certainly seen this used for short session psychotherapy like follow-up CBT-I sessions) 90839 (psychotherapy for crisis - It looks like this is called care coordination as well)

• It is helpful for us to know about these categorization decisions so we can explain them to the site as we could certainly see some of the team wondering why 90832 and 90839 are considered care coordination.

The couples/ family psychotherapy codes are missing (90846, 90847). Maybe this was intentional, but it could impact extent to which data reflects what is happening given some sites do a fair amount of evidence based couples psychotherapy?

The following are CPT codes for Psychotherapy (add on) when it is conducted with a Medication Management Visit. It seems that if you include these with PSY, it will muddy the waters with Medication Management. It would be helpful to understand why a decision was made to include these in PSY.
90833 90836 90838

Carla tried clicking on the SQL code link and was sent to a page with an error message indicating page not found so she is unable to assess the CPT codes for Agg Visit Details.

Can we have explanation of why 99201-99205 are assigned to Intake? Related to intake, it does appear that Intake is capturing codes 90791 and 90792 so it seems strange that data UI TeamAgg for ALX PCT team is showing only 1.25 appt supply yet dataEnc shows them using this code.

[emilymetcalf114]

During Live MTL prep session today, @emilymetcalf114 and @epearman met with facilitators:

Why is intake true missed appt % = 0 for all teams? @epearman looked at dax/sql code and saw it was written to be set to 0 (not calculated). This is also inline with the legacy code, not sure the decision behind this.

Further Clarification: We do not upload the missed appointment rate intake parameters into the SIM UI. So this variable is set to 0 on purpose.

[lzim]

@lzim discussed briefly with @emilymetcalf114 @epearman wk3 oct_epic support_workflow

• [x] Several of the decisions here have been carefully considered by the team in the past.
• [x] DECISION this impacts Thurs MTL meeting s7 10/20/22 led by @lzim

DECISION: wk4 oct_epic will track functional requirements on #2753 so this can be closed

• [x] DECISION this is a feature request: Cross-ref #2753 documentation issue task card where we will improve the documentation of these definitions. e.g.,
• [x] Describe Missed Appointment for Intake to 0 and provide rationale. #crossref #2773
• [x] ... [list task "functional requirements"] on documentation card #crossref #2753

• [x] However, the discrepancy translating legacy code to Power BI for Appointment Supply in Team (AGG) Care does need to be addressed as a BUG

  DAX formula: VAR APPTSUPPLY = IF(COUNTROWS(WEEKLYUSAGE) > 0, IF(ACTUAL50THPERCENTILE <= 0, 0, PERCENTILEX.INC(WEEKLYUSAGE, [VisCount], ACTUAL50THPERCENTILE) 0.25), 0) This varies from the legacy code line 264, which follows the array documented: select 'Appointment Supply (median)' as Label, [Intake] as intake, [psy] as psy, [ebpsy] as ebpsy, [cc]0.5 as cc, [mm]0.5 as mm , [adjunct]0.25 as adjunct, ([group]*1.0/6) as 'group'

[ghost]

Scoping of Bug:

• [x] Get Emily P Added as Admin to Power BI Workspace(s)
• [x] Get Emily P Added as Admin to Power BI Deployment Pipeline
• [x] Emily P. needs data to populate her data UI? (may be able to develop without this).
• [x] Emily P needs to implement the changes.

[ghost]

I have made the appropriate DAX code changes on a new personal version of Data UI - Master on power BI desktop.

Next Steps:

• [x] Testing - use these directions #crossref #2764
• [x] Publish to production (VISN priority order decided at hq_workflow once tested. crossref #455
• [x] Merge updated DAX code to GitHub - create a new card dec_epic #crossref #2774
• [x] Submit YourIt request for DAX code editor & create task card #crossref #2762

[ghost]

I cannot connect my test reports to the PTSD OHMO database through the Gateway connection in the testing pipeline. Gateway access to PTSD_OMHO must be granted Per-user through LEAF (add user to gateway).

• [x] Emily Pearman to request access to the PTSD_OHMO Gateweay

[ghost]

The testing Pipeline is currently ready for Review. Crossref #2764 QA Test Card.

• [x] @emilymetcalf114 review test reports and export data sets for SIM UI
• [x] @lzim review impact of new measures in the SIM UI

[emilymetcalf114]

1st test: Monday Blue MTL live team: NEW RESULTS using NEW code in TEST Pipeline HERE (followed by updated results in PROD using OLD code below):

For easy comparison, here are the updated results in PROD using OLD code

File for sim attached: Mon_Blue_MTL_data for sim_TEST.xlsx

[emilymetcalf114]

FYI @epearman Test #2: Thurs Blue MTL live team: NEW RESULTS using NEW code in TEST pipeline HERE (Followed by updated results in PROD using OLD code below): Appointment Supply increased across all services

For easy comparison, here are the updated results in PROD using OLD code:

@lzim File for sim attached: Thurs_Blue_MTL_data for sim_TEST.xlsx

[emilymetcalf114]

FYI @epearman I updated the clinic selections for the second test (above). The numbers match how we would expect them to now.

For documentation:

• [x] It was not adequate to compare the new code in the test environment to our previous screen shots of Agg Care data from over a month ago (see beginning of this issue card), because the cohort has been refreshed
• [x] To test the new code in the test environment, it required both aligning clinic selections in the test and prod environments for both live teams, and producing new screen shots of Agg care in both TEST and PROD environments.

Next Steps:

• [x] upload data files to sim shared environments for further testing.

[emilymetcalf114]

@lzim The new test data file for Thursday's MTL live session prep tomorrow has been uploaded to the sim for you.

[lzim]

@epearman Can you check off any functional requirement or task list boxes above that you believe are done from all the updates above? Or is this current?

It looks like we might be ready to move on to #2753 and #2509 💥

[ghost]

Crossref #2764 QA Test Card.

[ghost]

• @lzim are you done with testing the new data in the SIM UI?
• @lzim @emilymetcalf114 Should I push the minimal viable product to production? (i.e. the new data UIs with updated appointment supply calculations for Monday Blue MTL live team and Thurs Blue MTL live team)? Will these sites be needing the Data UI next week?

[lzim]

@epearman Testing in the Sim UI now and will update #2764 QA Test Card when done.

[ghost]

This issue has been closed:

• Any remaining functional requirements related to this ANDON are listed in the Training card crossref #2775.
• Production role of these changes across all VISN is tracked in crossref 455.