R21 Pilot Site World Buildout

[staceypark]

@TomRust @dlounsbu - It seems like working on this asynchronously makes more sense right now. We can meet once we have more of this fleshed out. As a reminder, I've de-identified the team names and used their team numbers instead. Please continue to do so on this issue.

Team6 AGG Basecase **Question: What trends or patterns of service delivery describe our Team's current efforts if we make no changes? **Hypothesis: We will see differences in terms of the number of patients served across serves. We will see high work pressure, too. Findings: Currently we serve more MM patients than any other service (about 1000 persons). PSY our second highest number of patients (about 500, currently). Patients receiving group service have been decreasing over time. We are providing very little EB-PSY. It appears that the number of patients waiting for services is between 8 and 10, over time. Decisions: After review of the base case, we will explore what happens if we increase the proportion of appointment supply for EBPsy.

Experiment 1: as_ebpsy_4_cc_4_asps_ebpsy_6_cc_10 Question: Does increasing the proportion of appointment supply for EBPsy increase wait-times for care coordination? What should the proportion of appointment supply be for EBPsy vs intakes vs care coordination? Hypothesis: If we increase the proportion of our total "appointment supply dedicated to EBPsy appointments" then access is reduced because "patients waiting to start service" will increase for care coordination. We expect we will do 5 EBPsy per week (not the 2 EBPsy per week based on our 2 year estimate). If we reallocate appts and service proportions so that we can double the amount of EBpsy (from 3% to 6%) and reduce CC from (22% to 10%, about half), we will not see appreciably high CC wait times. Findings: Our hypothesis is correct. If we make changes to both service proportions and appt supply, we start to provide more EBpsy while continuing to provide CC, but without generating appreciable wait times for CC. Decisions: Wait times actually improve, over time, for CC, when we adjust the appt supply and the service proportion accordingly. We now want to further increase of provision of EBPsy.

Experiment 2: as_psy_12_ebpsy_16_asps_psy_12_ebpsy_12 Question: What if we further expand our delivery of EBPsy, decreasing PSY proportion and appt supply? Hypothesis: We can reallocate patients from PSY to EBpsy over time by decreasing the proportion of PSY and the number of PSY appts for EBpsy. Findings: We see that patients in service for EBpsy goes from 0 to 1 to about 9 persons! CC and PSY patient loads decrease over time. Overall, patients waiting to start is about 24 persons on a given week. Decisions: Try to reduce patients waiting to start by service by changing the return interval rate for those in PSY. It is too high now (22 weeks!).

Experiment 3:rvi_psy_3 Question: To improve upon our current situation, what if we try to reduce patients waiting to start by service by changing the return interval rate for those in PSY? Hypothesis: We can reduce the number of patients waiting to start services in our team by adjusting the return interval rate for PSY from 22 wks to, say, 3 wks. Findings: We have eliminated the 24 persons-waiting backlog from our team! And we have much more room for PSY appts now. Decisions: Careful examination of the return interval rates for core services, like PSY, can make a big difference in managing patient load. Our decision is to keep return interval time for PSY to about 3 to 4 weeks, on average, for our patients.

@TomRust Can you add the Q/H/F/D and experimental values changed that you worked on for Team3 AGG? I didn't see any experiments saved in their team world.

Thanks @dlounsbu for working on the remaining one - Team5 PSY. You can add them here as well once they're fleshed out.

[staceypark]

@TomRust Thanks for working on Team3 AGG q/h/f/d! I've added it below. Basecase Question: What happens if we make no new decisions? Hypothesis: We expect our historical trends to continue for at least the next two years. Findings: It looks like we have more patients coming into Adjunctive, Psych, and EBPsych than we have slots for, AND too many slots set aside for the usual demand for Group (and, this is a little bit true for CC as well).
Decisions: Since we are expecting a surge of PSY referrals as staffing changes out in the CBOCs, we want our next experiment to help us see the impact on productivity (# of patients seen, time to treat, etc.).

Experiment 1: sptd_psy_40 Question: What's the impact on our productivity (# of patients seen, time to treatment; appropriateness of referral - increase in referrals that may not be appropriate due to loss of staff, increase in ATS and PCT referrals ) in relation to CBOC staff departures (psychiatry, social workers doing therapy)and on the telehealth team (full-time social worker, will return in August) ? Hypothesis: If we start receiving more referrals for patients needing psychotherapy (the service proportion increases from 26.1% to 40%), then we expect wait times for those patients to increase, along with our team's stress levels.
Findings: As predicted, patients waiting for PSY quintuples and wait time jumps to 4 weeks! However, even if we start with this higher % of patients trying to get into PSY, the high wait time causes us to refer fewer and fewer. After a serious drop and rebound, the % of patients we refer to PSY falls back down to the original value. We didn't see any spill over stress from this backlog of patients affecting patients in other services, as we didn't end up taking on more PSY patients, even though the demand went up. If we want to start more patients, then we have to cut the RVI, allocate more hours to PSY, or cut the Missed Appointment %. Decisions: We also expect to lose staff (hours to MM to zero) in the near future, so we want to see what impact that will have on wait times for our team.

Experiment 2: as_psy_0_ebpsy_20_cc_0_group_0_sptd_psy_0_ebpsy_50_cc_0_group_0 Question: Given the high wait times for PSY if we get a lot more PSY referrals, is it possible to meet the demand if we switch over to providing EBPsy instead? Hypothesis: If we allocate our PSY, CC, and Group hours to EBPsy, the we expect to be able to meet the increased demand for psychotherapy services, as patients will complete their treatment faster than they do currently. This will also be a more accurate picture of what is possible, as currently we believe that our hours aren't being read into the model the right way (specifically, estimates for CC and Group). Findings: We can almost meet the current psychotherapy demand if we switch to providing exclusively evidence-based psychotherapy, assuming we can commit 20 staff hours/week to EBPsy (starting 1.04 ppw, instead of 1.24 ppw). We definitely can't meet the hypothetical increased demand we predicted in Experiment 1 of 2.66 ppw,. Decisions: We know we have staff loses coming up (team lead's leave), so we'd like to run a scenario with that loss, to see what happens to wait times and patient care.

Experiment 3: as_psy_17_cc_0_mm_0_group_0_sptd_psy_40_tma_psy_10 Question: Is cutting the Missed Appointment % in half is enough to address a hypothetical increased demand for psychotherapy (from experiment #1)? We also expect to lose staff (hours to MM to zero) in the near future, so we want to see what impact that will have on wait times for our team.
Hypothesis: If we expect increased demand for PSY (service proportion grows to 40%, as in experiment #1), then more accurate estimates of our hours (ie, no hours for CC and Group, so those estimates are instead added to PSY), and reducing the Missed Appointment % by half should be enough to allow us to meet the higher demand. Losing our MM hours (from 3.5 hours per week to zero), we expect wait times to increase to start that service. Findings: Unfortunately, adding those two mis-allocated hours to PSY and reducing the Missed Appointment % from 20 to 10 was not enough to meet the expected increased demand for psychotherapy. We were able to start 1.5 ppw instead of 1.25 ppw, but no where near the 2.6 ppw that we expected (from exp #1). Missed appointment % doesn't appear to be an important factor in access to care, ie patient start rate.
Decisions: We need to dig into the details of where our referrals are coming from, so we can work with those clinics to reduce demand as we lose team hours: Look at specific CBOCs and clinics in the data UI tool and make adjustments in the sliders to experiment with "clinic-specific" impacts of the change.

[dlounsbu]

I have completed the re-build for Team5

The notes and the prior runs suggested that, across 5 MLT sessions, Team 5examined CC, MM and PSY. The later sessions where experiments appear to have been attempted were for PSY. So, I re-build basecase, exp 1, 2, and 3 using PSY. Note that each run included the parameter settings for the prior one.

The quality of the notes available was poor, but I think that I was able to rebuild runs that supported the discussion in the notes.

Thanks for your patience @staceypark . I guess the next step is to see how the REPORTS for the teams look.

I am in Wash DC for a conference ALL of next week (March 4 to 8), so my availability is limited. Hope to touch base with @staceypark and @TomRust , though, in the coming days.

[staceypark]

@dlounsbu Thanks for building out Team5 (*Note: please use the deidentified team numbers). When you say that the runs included the parameter settings for prior ones, should the name of Experiment 2 be this instead: "cwg_60_iwc_50_cetppm_75_cwg_80"?

Basecase Question: What is our team's current pattern of delivery in psychotherapy? Hypothesis: We are see a lot of people initiate, but they do not complete or graduate. Our team data suggests that they quit after initiating, and then return later. They stay 'engaged' in care for an extended period, well past the 12 to 16 weeks that would be sufficient to provide an evidence-based course of psychotherapy. Findings: As we expected, we see that a lot of people initiate, but they do not complete or graduate. Nearly all patients drop out within 2 to 7 visits of PSY, across all types of patients. Decisions: We will look to find ways to increase the number of PSY patients to complete and graduate.

Experiment 1: cwg_60_iwc_50 Question: We will look to find ways to increase the number of PSY patients to complete and graduate. Hypothesis: By reducing the proportion of patients who initiate by quit early to just 15%, and by increasing the proportion of patients who initiate and complete to 50%, we can manage to support a graduation rate of about 60% among our stock of PSY completers! Findings: These changes allowed us to increase the number of completer who graduated from 3 to 17, but we did not have a dramatic effect on initiators who do not graduate by return (BC=69 and Alt 1= 65). Decisions: We need to consider reducing our Return Visit Interval and/or our Engagement over 3 months time.

Experiment 2: cetppm_75_cwg_80 Question: Can we increase the number of patients who receive an adequate dose of PSY by reducing the problem of patients staying engaged with the team well after 3 months? Hypothesis: Building on Alt 1, we will now look to reduce the engagement time for patients who are in care past 3 months by 75% and, also, we will look to increase completers who graduate from 60% to, now, 80%. This will translate into more effective team work, as we are using our appoint supply more efficiently. Findings: The number of completers who graduated jumped from just 24 to 124! And we have more capacity to start new patients, which went from less than on pt per week to nearly 2 pts per week, more than doubling our new patients start rate. Decisions: We need to continue to assess the feasibility of our these change. We could work to reduce the number of completers who return. Despite the changes we have made we there are about 25 people who return even after they have completed. Maybe they are returning for a second course of PSY?

Experiment 3: apaet_50 Question: Can we further improve upon the number of patients who are receiving an evidence-based course of PSY, over and above our Alt 2 run? Hypothesis: Building on Alt 2, if we increase EB PSY template care to 50%, we will further increase the number of completers who graduate. This will translate into even more effective team work, as we are optimizing our RVI. Findings: We did not see an improvement in overall service delivery for PSY by increasing use of the EB PSY template. The number of completers who graduated remained at about 124. Decisions: The change we made by directing use of the EP PSY template did not change our service delivery patterns because our our prior return visit interval (RVI) was already close to the EB PSY standard (about once per week). We will look to implement the results of Alt 2, as this scenario appears to be close to optimal for managing our team's PSY patients.

[staceypark]

@dlounsbu @TomRust @lzim Please review the Q/H/F/D buildout (using Sim UI TEST: https://forio.com/app/va/va-psd-test/login.html) by next Wed, 3/13. If we need in-depth discussion time, I can setup a meeting.

Otherwise, we can just ask questions here. and once they're good to go, confirm by checking them off in the list below.

David

• [ ] Team 6 AGG
• [ ] Team 3 AGG
• [ ] Team 5 PSY

Tom

• [x] Team 6 AGG
• [ ] Team 3 AGG
• [x] Team 5 PSY

Lindsey

• [ ] Team 6 AGG
• [ ] Team 3 AGG
• [ ] Team 5 PSY

[TomRust]

@staceypark -- Is it okay if all the runs are in PROD, not TEST? I saved mine in PROD.

[lzim]

@tomrust and @staceypark we ultimately need them in PROD not TEST anyway.

But... @jamesrollins ... what it likely to happen when we release all updated MTL 1.8 models in PROD will the runs be saved? Has that been fixed?

Obviously, it would be much better in the long run if new releases don’t wipe out the old runs from prior releases. Otherwise, we’ll have this exact form of rework again in the future...

[dlounsbu]

@dlounsbu Thanks for building out Team5 (*Note: please use the deidentified team numbers). When you say that the runs included the parameter settings for prior ones, should the name of Experiment 2 be this instead: "cwg_60_iwc_50_cetppm_75_cwg_80"?

Basecase Question: What is our team's current pattern of delivery in psychotherapy? Hypothesis: We are see a lot of people initiate, but they do not complete or graduate. Our team data suggests that they quit after initiating, and then return later. They stay 'engaged' in care for an extended period, well past the 12 to 16 weeks that would be sufficient to provide an evidence-based course of psychotherapy. Findings: As we expected, we see that a lot of people initiate, but they do not complete or graduate. Nearly all patients drop out within 2 to 7 visits of PSY, across all types of patients. Decisions: We will look to find ways to increase the number of PSY patients to complete and graduate.

Experiment 1: cwg_60_iwc_50 Question: We will look to find ways to increase the number of PSY patients to complete and graduate. Hypothesis: By reducing the proportion of patients who initiate but quit early to just 15%, and by increasing the proportion of patients who initiate and complete to 50%, we can manage to support a graduation rate of about 60% among our stock of PSY completers! Findings: These changes allowed us to increase the number of completer who graduated from 3 to 17, but we did not have a dramatic effect on initiators who do not graduate by return (BC=69 and Alt 1= 65). Decisions: We need to consider reducing our Return Visit Interval and/or our Engagement over 3 months time.

Experiment 2: cetppm_75_cwg_80 Question: Can we increase the number of patients who receive an adequate dose of PSY by reducing the problem of patients staying engaged with the team well after 3 months? Hypothesis: Building on Alt 1, we will now look to reduce the engagement time for patients who are in care past 3 months by 75% and, also, we will look to increase completers who graduate from 60% to, now, 80%. This will translate into more effective team work, as we are using our appoint supply more efficiently. Findings: The number of completers who graduated jumped from just 24 to 124! And we have more capacity to start new patients, which went from less than on pt per week to nearly 2 pts per week, more than doubling our new patients start rate. Decisions: We need to continue to assess the feasibility of our these change. We could work to reduce the number of completers who return. Despite the changes we have made we there are about 25 people who return even after they have completed. Maybe they are returning for a second course of PSY?

Experiment 3: apaet_50 Question: Can we further improve upon the number of patients who are receiving an evidence-based course of PSY, over and above our Alt 2 run? Hypothesis: Building on Alt 2, if we increase EB PSY template care to 50%, we will further increase the number of completers who graduate. This will translate into even more effective team work, as we are optimizing our RVI. Findings: We did not see an improvement in overall service delivery for PSY by increasing use of the EB PSY template. The number of completers who graduated remained at about 124. Decisions: The change we made by directing use of the EP PSY template did not change our service delivery patterns because our our prior return visit interval (RVI) was already close to the EB PSY standard (about once per week). We will look to implement the results of Alt 2, as this scenario appears to be close to optimal for managing our team's PSY patients.

Hi @staceypark : So, it looks like you have successfully extracted the text from the runs I set up for my build out. This is good to go, in my opinion. Now, I do not recall the name of the data file I used for the team. When I called up what I thought might be the data file, there was no text in the boxes. Not sure exactly what you need me to find and verify, at this point.

[staceypark]

@TomRust Are these runs good to go from your opinion as well?

[TomRust]

Edits suggested for Team 6: BC-Findings -- change "...receiving group service have been decreasing..." to "receiving group service will decrease..." as the findings are a prediction not data.

Exp1-Findings -- I would add, "This experiment has clarified the trade-off between access (i.e., new patient start rates) for patients needing EBPsy and those needing CC, given our current resources."

Exp2-Hypothesis -- change "...reallocate patients from PSY..." to "...reallocate supply from PSY ..." We can't move the patients already in care to other services. We just change the hours and the service proportions, so the start rates change. With the 1.8 update, users have more abilities to move patients around, as they can change how long patients are in a service. If they wanted to clear the current patients out, then they could reduce the engagement time in q1 to be 1 week, which would clear all the current patients out, then put it back to normal in q2.

Exp2-Findings -- I'd report the start rates, too, as that is a clearer measure of access than wait times.

Exp3 -- Findings -- Needs a new explanation: Decreasing RVI for PSY will increase the hours needed for current patients, making even fewer hours (probably zero) available for new patients. The RVI change in the experiment creates a 7-fold increase in hours needed for current PSY patients (bringing patients in every 3 weeks vs only every 22 weeks). Wait times decline because of the feedback loops between poor quality (ie, much longer RVI than desired) and service proportion -- if you change the desired RVI to something completely impossible without changing the hours available, then providers will now see that current patients aren't getting the new standard of care (actual RVI is nowhere near the desired RVI of 3 weeks), and will choose to refer fewer patients to PSY (ie, the service proportion will drop to near zero).

[TomRust]

Changes suggested for Team 5

Exp1 -- Hypothesis -- Did you change the "patients who initiate by quit early to just 15%"? I don't see that in the run name. It's fine if you didn't, as drastically reducing that proportion in the real world would be very difficult, anyway. I'll bet most providers don't know why patients leave mid-treatment, let alone what they can do about it. Well, we are working in a virtual world. If I were a clinician, I would be concerned about this, and I would have more than one hypothesis about why my patients don't come back. I like this test, too, because it illustrates how the model works. Exp1 - Findings -- I'd also report on the start rate here, like you do in the other experiments. Getting more people to graduate is a good goal, but we don't want the team forgetting about this other win. If Stacy can help me get into this world, I will look into it again. I do not remember what the start rate was.

Exp3 - Changing the % of patients who get a template doesn't change their care pathways, or how many patients go down which path (i.e., all the %s and RVIs don't change). To the model, using templates is just icing on the cake -- it should be 100%, but even if it's zero, it doesn't change the underlying cake. We built the model this way because we found many instances of patients getting a timely complete dose without the provider completing all the desired templates. This is a fine experiment to run, to illustrate how disconnected the patients actual quality of care is from whether or not the providers are using templates...if that is what the team was interested in. The one thing I'd change, though, is that the base case value is 50%, so setting the slider to 50% didn't change anything... not that this slider can change any of the flows anyway, but still. Looks like we should rerun this experiment again, then. But I need help getting back into this work. But I need TA from @staceypark to get back into this. world.

[staceypark]

@dlounsbu Can you look at @TomRust's comments and update your q/h/f/d based on his comments (while still keeping it in "learner voice" - how a learner would typically phrase their q/h/f/d).

[dlounsbu]

@dlounsbu Thanks for building out Team5 (*Note: please use the deidentified team numbers). When you say that the runs included the parameter settings for prior ones, should the name of Experiment 2 be this instead: "cwg_60_iwc_50_cetppm_75_cwg_80"?

@staceypark All I meant, here, is that Exp 1 uses all the settings for basecase, plus the Exp 1 change, Exp 2 uses all the settings for Exp 1, plus the change for Exp 2, and so on.

[dlounsbu]

@dlounsbu @TomRust @lzim Please review the Q/H/F/D buildout (using Sim UI TEST: https://forio.com/app/va/va-psd-test/login.html) by next Wed, 3/13. If we need in-depth discussion time, I can setup a meeting.

Otherwise, we can just ask questions here. and once they're good to go, confirm by checking them off in the list below.

Hi @staceypark : I cannot log in to the Sim UI TEST. I am probably doing something wrong. Will try to reach you for special assistance tomorrow, Friday May 3. I am so curious to see the reports! Best and thanks, David

[lzim]

Hi @dlounsbu and @staceypark

Is this to still be done in TEST, or is this PROD now?

Lindsey

[dlounsbu]

Hi @dlounsbu and @staceypark

Is this to still be done in TEST, or is this PROD now?

Lindsey

@lzim : I am not sure. If the summary reports for the teams have been compiled, can you send a PDF copy? I am not having luck logging in to the SIM to find the QHFD text for this rebuild. Can @staceypark help me with this tomorrow?

[staceypark]

Changes suggested for Team 5

Exp1 -- Hypothesis -- Did you change the "patients who initiate by quit early to just 15%"? I don't see that in the run name. It's fine if you didn't, as drastically reducing that proportion in the real world would be very difficult, anyway. I'll bet most providers don't know why patients leave mid-treatment, let alone what they can do about it. Well, we are working in a virtual world. If I were a clinician, I would be concerned about this, and I would have more than one hypothesis about why my patients don't come back. I like this test, too, because it illustrates how the model works. Exp1 - Findings -- I'd also report on the start rate here, like you do in the other experiments. Getting more people to graduate is a good goal, but we don't want the team forgetting about this other win. If Stacy can help me get into this world, I will look into it again. I do not remember what the start rate was.

Exp3 - Changing the % of patients who get a template doesn't change their care pathways, or how many patients go down which path (i.e., all the %s and RVIs don't change). To the model, using templates is just icing on the cake -- it should be 100%, but even if it's zero, it doesn't change the underlying cake. We built the model this way because we found many instances of patients getting a timely complete dose without the provider completing all the desired templates. This is a fine experiment to run, to illustrate how disconnected the patients actual quality of care is from whether or not the providers are using templates...if that is what the team was interested in. The one thing I'd change, though, is that the base case value is 50%, so setting the slider to 50% didn't change anything... not that this slider can change any of the flows anyway, but still. Looks like we should rerun this experiment again, then. But I need help getting back into this work. But I need TA from @staceypark to get back into this. world.

@TomRust are you suggesting that we need a new experiment 3? Maybe we can briefly touch base on it tomorrow, 5/10?

[lzim]

@dlounsbu are you clear about what is needed?

The edits from Tom are not in team language.

FYI: @staceypark

[staceypark]

@dlounsbu we briefly discussed this last Friday - were you able to make any headway on it?

[dlounsbu]

No, I was not able to turn back to this yet.

[TomRust]

@staceypark -- yes, I think we need a new experiment 3. Sorry for not making that clearer! :)

[dlounsbu]

@lzim @staceypark @TomRust : I have rebuilt the runs for Team 5 PSY. I used the 640a0*.xlsx data file. My experiments are not identical to what I ran before, as the data for the team seemed different. Hopefully this rebuild will be sufficient. Or if we need to change any of my new experiments, they are properly stored in the SUI. Best and thanks.

DWL Psychotherapy | Team 5 | data=640a0 … 16jul2018.xlsx

Team 5 PSY Basecase

Question: What is our team's current pattern of delivery in psychotherapy?

Hypothesis: We see a large proportion of patients who initiate (85%), but they do not complete and graduate (only 12%). Our team data also suggests that many patients who completed a full course of psychotherapy come back, with a return rate of 88%. They stay in care for an extended period, well past the 12 to 16 weeks that would be sufficient to provide an evidence-based course of psychotherapy.

Findings: As we expected, we see that a lot of people initiate (77), but they do not complete or graduate (3). Nearly all patients drop out within 2 to 7 visits of PSY, across all types of patients (AUD, DEP, OUD, and PTSD).

Decisions: We will look to find ways to increase the number of PSY patients who complete and graduate.

Team 5 PSY Experiment 1

Question: What would increase the number of PSY patients who complete and graduate?

Hypothesis: By reducing the proportion of patients who initiate but quit early to from 16% to just 5%, and by increasing the proportion of patients who initiate and complete to from 34% to 50%, we will generate a higher graduation rate of patients who complete PSY. This is a relatively dramatic change and may not be plausible in the real world, but since we are in a virtual world and there is no harm done, it would be useful to see what the impact for our team would be.

Findings: We had a marginal effect on the number of patients would completed (26 to 32). Similarly, our graduation rate impact was even smaller (3 to 4 patients!). Notably, the supply of appointments used by completers who continue is going up (from 5 to about 6.3).

Decisions: We need to explore strategies that would decrease duration time of patient who are in psychotherapy past 3 months.

Team 5 PSY Experiment 2

Question: What if we cut the duration time of patients who are in psychotherapy past 3 months by 50%?

Hypothesis: Building on Exp 1, we will now look to reduce the engagement time for patients who are in care past 3 months by 50%. This will translate into more effective team work, as we are using our appoint supply more efficiently, allowing us to increase our EB PSY reach.

Findings: We have significantly increased the number of PSY patients who are completers (from 26 to 51). We are also using our appointment supply more efficiently, with patients who are in care for less than three months accounting for increased use of supply and those in care for longer than three months accounting for less use of appt supply. But the number of people returning after three months has gone up. Not good! And our graduation rate has gotten worse (18 now, compared to basecase of 21!).

Decisions: We need to experiment with return visit interval.

Team 5 PSY Experiment 3

Question: What is the effect of shortened return visit interval on completers and patients who graduate?

Hypothesis: Building on Exp 2, we will shorten the return visit interval (RVI) by 50% to see if we can increase the number of patients who receive an evidence-based dose of psychotherapy (EB PSY, which is currently at 50%). This should also further increase our number of completers who graduate, and this will translate into even more effective team work, as we are optimizing our RVI.

Findings: Interestingly, this experiment degraded the progress we made to increase the number of PSY patients who are completers. We had increased it from 26 to 51 in Exp 2, but now we dropped to 39. Also, the changed in appointment supply use seemed short term. The RVI reduction only changed patterns of appointment supply use over the first year, then matched our Exp 1 pattern of results thereafter. Finally, we worsened the matter of completers who return (from 66 to 77), we did not increase our graduation rate (down to 13 from 21!).

Decisions: It appears that we need to be more targeted in working with particular subgroups of patients (OUD, DEP, PTSD, AUD). For example, PTSD patients account for more than 80% of our appoint supply use, and they also account for 50% of all initiators who return later. Future experiments should focus on these patients.

REVISED Findings and Decisions for Team 5 PSY Exp 3 (2019-05-31):

**Findings (revised): Interestingly, this experiment degraded the progress we made to increase the number of PSY patients who are completers. We had increased it from 26 to 51 in Exp 2, but now we dropped to 39. Also, the changed in appointment supply use seemed short term. The RVI reduction only changed patterns of appointment supply use over the first year, then matched our Exp 1 pattern of results thereafter. Finally, we worsened the matter of completers who return (from 66 to 77), we did not increase our graduation rate (down to 13 from 21!). Considering these results, it is now clear that reducing the RVI means that patients are seen more frequently (less time between visits), thus the same number of patients are now taking up more slots in the team's schedules than before, leaving fewer slots open for new patients!

Decisions (revised): It appears that we need further explore how best to change RVI. Also, we may need to be more targeted in working with particular subgroups of patients (OUD, DEP, PTSD, AUD). For example, PTSD patients account for more than 80% of our appoint supply use, and they also account for 50% of all initiators who return later. Future experiments should focus on these patients.**

[staceypark]

@dlounsbu 😄 🎉 this is great! THANK YOU for working on this. We really want to wrap these teams and offer them insights soon! @TomRust can we get a second set of eyes on this again?

[TomRust]

@staceypark -- David's Team 5 experiments look great. I'd maybe add one sentence to the Findings of experiment 3, explaining why decreasing the RVI made things worse: "Reducing the RVI means that patients are seen more frequently (less time between visits), thus the same number of patients are now taking up more slots in the team's schedules than before, leaving fewer slots open for new patients."

[dlounsbu]

Great and thanks @staceypark and @TomRust

I logged in and edited experiment 3.

Findings (revised): Interestingly, this experiment degraded the progress we made to increase the number of PSY patients who are completers. We had increased it from 26 to 51 in Exp 2, but now we dropped to 39. Also, the changed in appointment supply use seemed short term. The RVI reduction only changed patterns of appointment supply use over the first year, then matched our Exp 1 pattern of results thereafter. Finally, we worsened the matter of completers who return (from 66 to 77), we did not increase our graduation rate (down to 13 from 21!). Considering these results, it is now clear that reducing the RVI means that patients are seen more frequently (less time between visits), thus the same number of patients are now taking up more slots in the team's schedules than before, leaving fewer slots open for new patients!

Decisions (revised): It appears that we need further explore how best to change RVI. Also, we may need to be more targeted in working with particular subgroups of patients (OUD, DEP, PTSD, AUD). For example, PTSD patients account for more than 80% of our appoint supply use, and they also account for 50% of all initiators who return later. Future experiments should focus on these patients.

[dlounsbu]

@lzim I think we can close this issue!