9/20 - FRI Team 2 Meeting PSY Sankey: Basecase removed the majority of patients

[lzim]

1. Add description:

@jamesmrollins We were prepping for Team 2 today and saw this below for a basecase run. Not sure what's causing this. Need to resolve and prep before Team 2's meeting Friday at 9:30AM Pacific.

**NOTE: I was unable to upload the screenshot of the issue and console, so I also sent an email with the same name to your attention with @ritahitching and @staceypark

***NOTE: @jamesmrollins added screen capture for the record. @lzim was also using 640a)_team2_2019_08_01.xlsx team data file.

2. Click on the right: a. Projects - assign to issue_tracker. b. Assignee - assign relevant team members. c. Labels- review and select all potential MTL dependencies. d. Milestones - select the dependent milestone.

---

Workgroups leads Identify Constraints (check all that apply):

• [ ] Capacity
• [ ] Cost
• [ ] Schedule
• [ ] Interference
• [ ] Capability
• [ ] Other (describe)

Add estimated person-hours to complete: 3 hrs.

Edit the due date (if necessary)

[lzim]

Adding @TomRust

• Just in case this is something that needs his attention on the PSY model side.

[jamesmrollins]

@lzim FYSA

@TomRust I am not sure it is a model problem at this point. I will work with DEV tonight to verify that hypothesis. We have been having some issues with Forio APIs, so it may be related. I will advise more tomorrow.

@hirenp-waferwire please take a look at the way Sankey related variables are (not) loading. As you can see from comparing the two pictures below, the Vensim client version is returning, for example, EXP-A / ~240. But in the sim it returns no value [ ]. I have verified that we are running the same modelparameters file in both simulations. In the Model Directory in PROD it is called 640a0_team2_2019_08_01.xlsx - Thanks, James

Vensim

Sim UI

Sankey Diagram (I would like to have a variable key for this diagram please)

[lzim]

Thanks @jamesmrollins and @hirenp-waferwire! 🕵🏼‍♂️

@jwklocek and I will stay posted for an update 👀

[ritahitching]

@lzim @jamesmrollins will develop a map that will be added to the master to explain what variables drive which variables in the chart.

[TomRust]

I talked with James in our Support Leads meeting, and he has enough info now to work with the Dev Team to get this sorted. Should be squashed by the tomorrow morning (9/20).

[jamesmrollins]

@TomRust @lzim @staceypark I have tested this out in TEST and PROD this morning by comparing outputs to client Vensim model run with same ModelParameters.xlsx. The results of the SIM UI outputs and the Vensim model outputs are consistent and accurate. I promoted directly to PROD due to the urgency of the issue. Please report any issues with results to me right away. Thanks James

[lzim]

@TomRust @lzim @staceypark I have tested this out in TEST and PROD this morning by comparing outputs to client Vensim model run with same ModelParameters.xlsx. The results of the SIM UI outputs and the Vensim model outputs are consistent and accurate. I promoted directly to PROD due to the urgency of the issue. Please report any issues with results to me right away. Thanks James

Awesome, @jwklocek and I can test it out today, to make sure we’re ready for “Team 2” tomorrow 🦸🏻‍♀️🦸🏻‍♂️

[lzim]

@jwklocek

I tested the Team 2 PSY Basecase in my individual world, and it is as expected where the Sankey diagram for the Basecase, and the “Experiment” run correspond.

The issue we observed is also fixed in the team world, and their Q text from 8/30/2019: “How can we get more pts through a full course of quality PSY?” is displayed.

[staceypark]

@jamesmrollins @lzim is in the sim and we see Low/Medium/High for Engagement Duration levels. She recalls changing it to 25%/Median/75% at some point previously. Could you weigh in on what happened with these variable names?

[lzim]

@jamesmrollins

We were experimenting with changing Engagement Duration for patients engaged in care >3 mo (aka "cedtm") in the image below.

• The slider range on this is -75% to 200%, but the save values range from 0-100.

[lzim]

@jamesmrollins @lzim is in the sim and we see Low/Medium/High for Engagement Duration levels. She recalls changing it to 25%/Median/75% at some point previously. Could you weigh in on what happened with these variable names?

More specifically, I could have sworn that when we were working on this last week, we saw "median" and "percentiles" for one variable (RVI or engagement duration) and "medium" and "low/high" for the other (RVI or engagement duration)

• NOTE: This applies to both engagement duration and RVI

If this is not related to work on issue #709, then this question and my question above, likely should be separate issue cards that should go in the needs_triage column of the issue_tracker.

[lzim]

@jamesmrollins and @staceypark

• I believe we standardized the "after 3 months" language, isn't that right?
• I see "after 3 months" almost everywhere.
• But, in two places for RVI in the experiment slider and model diagram I see "patient's past 3 months" for RVI.

Here:

And here:

• But, it is correct here 🎊

🙄 These things are so annoying. Like the questions above, let me know how we should handle them: 1) in the context of the "closed" issue #709 in which they came up, or 2) in a new issue?

Thanks!

[jamesmrollins]

@lzim, let me check it out and I will get back with some recommendations. I'm sorry for the annoyance and will work hard to resolve.

[lzim]

Hey @jamesmrollins and @TomRust FYI: @staceypark

Is there a reason RVI wouldn't be in the outputs?

• I was testing out my understanding of the 75% reduction sliders for RVI and engagement duration in PSY.
• I adjusted the slider for RVI to -75% and then wanted to know what that impact would be.
• I didn't see that as a output chart option in the drill down:

😍 SEARCH BOX that is such an upgrade! ❤️

[jamesmrollins]

@hirenp-waferwire please make the following changes in PSY:

Change slider name.

Change the slider icon name:

Evaluate the variable scale values in the save function and revise:

[jamesmrollins]

@lzim @staceypark @holbrooa @TomRust I investigated the Engagement Duration label issue, see https://github.com/lzim/teampsd/issues/451#issue-466013260, which is the PSY Team Data Table revision. There are no instructions changing "Low/Medium/High" to "25%/Median/75%." I also kind of rummaged around in my brain and didn't find anything either, but that isn't saying much 🤦‍♂️ . In any case, do you folks want me to change it?

[jamesmrollins]

@lzim @TomRust please see below. I looked over the model and found that the RVI is just a constant set at "0." I discuss the relationships in the diagram below. It appears not to be "reportable."

[lzim]

@jamesmrollins @TomRust and @staceypark

Hi 👋

I found a mix-up in two slider in PSY

• See in the screenshot how the two switches that are supposed to sum to 100 switch the stock they reference Starters plus Initiators on the left and Initiators plus Starters on the right.

Based on the output when I used the switch on the right, I believe the model is doing what is says.

• It is setting the Initiators who Complete to 50% AND Starters who Return Later to 50%
• What I need is Initiators who Complete to 50% AND Initiators who Return Later to 50%.

@lzim @staceypark I investigated the Engagement Duration label issue, see #451 (comment), which is the PSY Team Data Table revision. There are no instructions changing "Low/Medium/High" to "25%/Median/75%." I also kind of rummaged around in my brain and didn't find anything either, but that isn't saying much 🤦‍♂️

• Okay, I believe this is something that @holbrooa and I clarified, but I thought it was carried forward throughout.
• In fact, I believe we needed to keep an eye on this to solve a problem of some kind, not just provide clarity.
• I note in the Vensim you've pulled up, RVI and Engagement Duration labels state: 1st quartile, mean and 4th quartile.
• @holbrooa Will you please check the disseminate_scientists_va column session 3 card and Column D of the Team Data Table and let me know what these values actually refer to?
• @staceypark We need to get this right in our documentation and i information.
• @jamesmrollins In terms of the Sim_UI let's get this clarified first.

@lzim @TomRust please see below. I looked over the model and found that the RVI is just a constant set at "0." I discuss the relationships in the diagram below. It appears not to be "reportable."

• Okay, thanks for looking into this @jamesmrollins.
• @TomRust It may be tricky to explain what is happening when we make the RVI reduction by 75% (for example) if all we have to think about it with is the Sankey.

[lzim]

@lzim, let me check it out and I will get back with some recommendations. I'm sorry for the annoyance and will work hard to resolve.

...This stuff is a collective eye roll... 🤦‍♀ 🤦‍♂

...I'm so grateful for all you do to stay on top of this!

[jamesmrollins]

Hi @lzim @TomRust I am confused about your last entry, as I can't find "Initiators who Return Later %" neither in the Crosswalk nor in the model. However, I do find "Intiators who will Return." But it is measured in "pts" not "dmnl." Please clarify.

[TomRust]

@lzim The 9 RVI estimates for the cohorts in care >3 months aren't displayed in the model diagram, so we decided that they didn't need to be in the Expanded Outputs chart list either. Also, the RVIs are inputs to an experiment (and aren't affected by any other experiment), so I think it would be confusing to see any of them in an outputs chart. There wouldn't be anything interesting happening in those chart, anyway -- just a step change at week zero, as described in the i-text. As for outputs, we do have charts for the direct effect of these RVI changes = changes in the supply used by the different sub-cohorts of patients in care >3 months.

[TomRust]

@jamesmrollins -- that RVI variable (the one you describe as "hanging out") is only there so we can have i-text for a generic RVI variable in the model diagram. The model uses way more RVIs that we wanted to actually show in the diagram.

[lzim]

@jamesmrollins @TomRust

Thanks @jamesmrollins for this clarifying question. Yes, how these are supposed to be set up, is a bit confusing to me right now, too... I'm hoping @TomRust can help us, especially given your inability to find a dmnl "Initiators who Return Later %" as I would expect given the "i" text 🤓

There are the three outflows from the initiators stock and the starters stock

• "quit early %"
• or flow on to next state in first 12 weeks, i.e., "starters who initiate %" and "initiators who complete."
• "return later %"

Based on what I'm reading the "i" information and believe we need this pair:

• "Starters who Initiate %" with "Starters who Return Later %"

Then we need this pair

• **Initiators who Complete %" with "Initiators who Return later"

"i" text says these two pairs, should not sum to more than 100%, otherwise patients will be pursuing multiple engagement pathways simultaneously. (pct)

**This makes sense to me, but unfortunately, I can't help with your Vensim "missing" or "mismatched units" problem." Hopefully @TomRust can 💯

[lzim]

Thanks @TomRust

• Yes, re: outputs

  "we do have charts for the direct effect of these RVI changes = changes in the supply used by the different sub-cohorts of patients in care >3 months."

• I get this and work with it in training.

• It is the re: inputs - that I'm trying to make clearer 😆 when setting up experiments, to clarify how the 9 RVI estimates that we review will be effected by the 75% reduction In other words, what the ~ RVI will be, throughout the two-year experiment timeline.

just a step change at week zero, as described in the i-text.

Okay this clarifies: I wondered because the "i" information calls this a "relative change," which commonly refers to differences in values observed over time. I want to be clear if someone asks about applying a typical "relative change" formula in this way related to the experiment timeline:

• ((RVI at end of 2 year run - RVI today)/RVI Today) x 100

🧮 Training Team 2 example:

• Across each RVI Distribution (low/medium/high or TBD 25%ile, median 75%ile ?? see above) for starters, initiators and completers who return after 3 months, have > 3 Mo RVIs of:

  • 9 wk RVI (starters)
  • 4 wk RVI (initiators)
  • 2 wk RVI (completers) We could sum them, take their average, then subtract 75% of that value, to estimate what the likely RVI for > 3mos will be when we hit "Run" in this experiment.

• Step 1: (9 + 4 + 2)/3 = average RVI of 5 weeks

• Step 2: 5 * 0.75 = 3.75 is 75% of this average

• Step 3: 5 - 3.75 = RVI 1.25

• We would be experimenting with an RVI for patients in care > 3mo that is ~1.25 at time 0 And, it would stay fixed throughout the two year run.

• Sound right? And, thanks!

• CONTEXT: From the [team_tracker prototype](https://github.com/lzim/teampsd/issues/582): Team 2 asked a question about bringing RVI up to weekly for everyone...

[TomRust]

@jamesmrollins -- yup. There is no separate "Initiators who Return %" variable. It is calculated from the other two relevant %s (i.e., the other two possible care pathways - quitting or continuing on in their first 3 months), and included directly in the "Initiation Rate who will Return." I can pop that part of the rate equation out into a new "Initiators who Return %" variable, if needed.

[TomRust]

@lzim -- RE: calculating the impact of a 75% reduction in >3month RVI for "Team 2"

Yup! That all makes perfect sense to me. Two quibbles: 1) I'd be a bit wary about assuming the average RVI is 5 weeks, though, as those three groups are probably not the same size. 2) Changing the RVI happens immediately at time zero...but only for appointments not yet booked! We aren't assuming that they re-book all the appointments already on the calendar to meet this new clinical goal, but that they apply it to all booking decisions going forward from time zero. This will delay the impact of this change (by the old RVI for each sub-cohort). You should see a staggered change in the Supply Used >3 months chart (e.g., first the completes will start being booked at the new RVI, as they only have 2 weeks worth of appointments on the calendar to get through, then the supply used by Initiators will increase after 4 weeks, etc.). At least until they run out of supply for this vastly increased demand for slots! :)

[lzim]

@tomrust

I'd be a bit wary about assuming the average RVI is 5 weeks, though, as those three groups are probably not the same size.

• Yes, it’s “back of the envelope” for sure, and not weighted by the proportion of patients in each group to get a better estimate.
• Also, TBD: Would I be averaging the median or the mean?

...only for appointments not already booked...

Very helpful clarification. Thanks!

[lzim]

Hey @jamesmrollins and @TomRust

Did these two switches get cleared up today? I do not see any updates about this since @jamesmrollins post yesterday afternoon saying he did not find the expected **dmnl “initiators who return later %” variable in Vensim.

I was busy all day with the Director job talk, so I’m just following up on it now.

Thanks, in advance, for the update!

Lindsey

@jamesmrollins @TomRust and @staceypark

Hi 👋

I found a mix-up in two slider in PSY

• See in the screenshot how the two switches that are supposed to sum to 100 switch the stock they reference Starters plus Initiators on the left and Initiators plus Starters on the right.

Based on the output when I used the switch on the right, I believe the model is doing what is says.

• It is setting the Initiators who Complete to 50% AND Starters who Return Later to 50%
• What I need is Initiators who Complete to 50% AND Initiators who Return Later to 50%.

[jamesmrollins]

Hi @lzim , I hope you had a good director's talk today. We took care of the switch labeling issue regarding "after 3 Months" and the -75 to 200 drop down range issue last night. I was focused on the PsychTool and Chris today, so didn't follow up with @TomRust regarding the switch mix-up . I will reach out to him in the morning. Thanks, James

[lzim]

Great, thanks for the update @jamesmrollins!

I’ll be facilitating a training w/PSY at 9:30AM Pacific.

[jamesmrollins]

Hi @lzim . I just spoke with @TomRust and he indicates that indeed, the variable you seek does not exist; however, maintains that it was not essential to the story, since it was not something the team has direct control over. In any case, he will be at the Leads meeting tomorrow morning to discuss further. James

[lzim]

@TomRust and @jamesmrollins

Then, if these switches are meant to work the way they do, can we get the “i” information to match the labels of the switches, so the “story” and variables that actually exist to match then, please?

Based on what I'm reading the "i" information and believe we need this pair:

• "Starters who Initiate %" with "Starters who Return Later %"

Then we need this pair

• **Initiators who Complete %" with "Initiators who Return later %"

"i" text says these two pairs, should not sum to more than 100%, otherwise patients will be pursuing multiple engagement pathways simultaneously. (pct)

**This makes sense to me, but unfortunately, I can't help with your Vensim "missing" > or "mismatched units" problem." Hopefully @TomRust can 💯

I’m available via phone now, if you two are. 🤳🏼

[lzim]

Morning @jamesmrollins and @TomRust

• Tom and I just got off the ☎️

• So the sliders are working individually as they should, but they aren't aligned correctly

@jamesmrollins it should be:

• "Starters who Initiate %" with "Starters who Return Later %"
• Initiators who Complete %" with "Initiators who Quit Early %" (not Initiators who Return %).

@TomRust is going to update the "i" information .xlsx on OSF and upload to the Sim UI.

• It will provide a short clarification about what is happening to the 3rd outflow (or remainder) if they do select to have the outflows of the two paired sliders sum to 100%.

NOTE: We will NOT be experimenting with this at today's training, but could be soon 📆

Thanks!

[jamesmrollins]

@lzim @TomRust I will issue a change order tonight to DEV and get this corrected by tomorrow. James

[jamesmrollins]

@lzim @TomRust please verify this is what you want. Then I will release. Thanks James

[TomRust]

@jamesmrollins , @lzim -- LZ and I talked this morning, and we decided that the itext was insufficient. I added this text (below) to the itext for these four slider variables in model pop-up table in OSF.

For the Starter sliders: Note, if these two percentages sum to 100%, that implies that no (zero percent) Starters are quitting early (i.e., the team has no "One-and-Dones").

For the Initiator sliders: Note, if these two percentages sum to 100%, that implies that no (zero percent) Initiators are returning later (i.e., completing there next appointment more than 3 months after they initiate).

[lzim]

Thanks to you both!

@jamesmrollins

• Yes, the right slider set looks good.

• But, for the right slider, please keep "Initiators who Complete %" as the top slider (where it is now), and put the "Initiators who Quit Early%" on the bottom right slider set.

• This keeps the slider sets parallel with flow in the diagram, where the desirable flow is on the top (starters -> initiate, and initiate -> complete) for each slider set, and the undesirable flow is on the bottom.

Make sense?

Thanks!

[jamesmrollins]

@hirenp-waferwire please make adjustments as indicated above.

[lzim]

@holbrooa

Among other things discussed, there’s a callout to you about use of low/medium/high for RVI and engagement duration in the thread above.

• The PSY Vensim model says this refers to 25th%ile/mean/75th%
• @staceypark said the Team Data Table Definitions clarify whether that is the way these parameters are operationalized.
• I am also remembering some discussion of the use of the mean versus the median, solving a problem for us somewhere, but cannot recall if that was for PSY or not.

@holbrooa Can you help us to confirm?

[holbrooa]

Yes, the team data table clarifies this. It says:

"After the patients have been categorized by patterns of engagement, those who return for additional visits after three months are further categorized into low, medium, and high groups based on duration of engagement. The medium group comprises the middle 50%, with low and high comprising 25% each. The duration values are the 25th percentile, 50th percentile, and 75th percentile values taken from the entire group, and then the visit frequencies (RVI) are calculated from the median visit counts of each group."

The discussion about mean making something easier was something else.

[jamesmrollins]

@holbrooa and @lzim - so what is the call?

1. Low/Medium/High or
2. 25%/50%/75% or
3. Low (25%ile)/Medium (50%ile)/High (75%ile).

BTW I looked up how to abbreviate percentile, and that is what Bing came back with - so it must be right 😉

[lzim]

@staceypark

Can you explain why @holbrooa found this answer in the Team Data but you did not?

• We need to look for this in the Team Data and keep it clear/consistent across resources.

@TomRust - If we send in the Vensim models with PSY manuscripts, we also need to make sure inaccuracies are caught (i.e., Vensim says “mean” when it’s the “middle 50 %”). We said we were post-poning some under the hood Vensim changes, previously. But, we may need to make sure we are current now.

@holbrooa

• So for engagement_duration We could use:

• low (25%ile), medium (50%ile), high (75%ile)

• and for RVI We could use:

• low (median), medium (median), high (median)

[staceypark]

@lzim not sure what you mean. That is what I said

Can you explain why @holbrooa found this answer in the Team Data but you did not?

@staceypark said the Team Data Table Definitions clarify whether that is the way these parameters are operationalized.

[lzim]

Apologies @staceypark I was remembering a meeting where I believe you weren’t clear about this answer, no?

• I thought that was why we had to loop in @holbrooa

If it’s all clear between team data and sim ui than we just need to get Vensim current and make sure what I proposed above for duration and rvi works for @holbrooa

[lzim]

@staceypark

Need to make sure these concepts are clear across dependent columns in the document_tracker

• For example, for RVI, the summary I'm checking with @holbrooa above really means this:

"low duration patients (median RVI in weeks), medium duration patients (median), high duration patients (median)"

[holbrooa]

Seems fine to me.

[jamesmrollins]

@lzim @holbrooa @staceypark Does this get the job done?

[lzim]

Yes, this looks good @jamesmrollins

What about the RVI and Engagement Duration charts?

Also, I want to keep this on @TomRust’s radar in terms of the Vensim models

[jamesmrollins]

@lzim - Thank you, I will review and add instructions.

@hirenp-waferwire - See changes below, thank you .

[lzim]

@jamesmrollins FYI: @staceypark

James, can you double-check that Team Data Table “i” information for “Engagement Duration” and “RVI” because it’s in the PSY module, but sounds like the two edits you have in the red call outs describe Care Coordination.