9/16 EBPsy and EBPharm course data

[holbrooa]

ORIGINATOR Please follow the steps below to open an issue for the MTL Workgroups to address. Be sure to provide a descriptive title for the issue when saving.

Step 1 - Please assign this issue to the issue_tracker, under the “Projects” menu on the right side of your GitHub page.

Step 2 - Please answer the questions below by placing a lowercase "x" between the applicable brackets below and providing brief narratives and screenshots where appropriate:

1. Is this issue a problem, such as a malfunction in the simulation, an error in documentation or similar concern, or is this a new idea we should consider?

   • [ ] Problem
   • [x ] New idea

2. What team are you on (exg., Menlo Park, Team Blue)?

3. How urgent is this issue?

   • [x] Urgent - should be addressed right away!
   • [ ] Non-urgent - can be prioritized within normal workstreams.

4. In the space provided below, please describe the information or behavior you expected from the product or otherwise describe what you expected to see or hear.

Need to add course/dose data to comparator for 6 HF templates. See table of EBP completer defs for ACT, PE, CPT, IBCT, IPT, CBT-D (it's different for each).

And also use scripts and tables from Jodie's team for EBPharm dose.

5. Please provide a screenshot of the bug, if possible. Make as many screenshots as you like that would be helpful to illustrate the issue.

   << Paste screenshots here>>

6. Within which product did you notice the problem? Check all that apply

Data Products

• [ ] SharePoint
• [x ] Data/Queries
• [ ] Team Data
• [ ] R-reports

Instructional Products

• [ ] See Guide
• [ ] Say Guide
• [ ] Videos
• [ ] Post tests
• [ ] Fidelity Checklist
• [ ] Cheatsheets
• [ ] Other? Please describe here >>

Simulations

Home

• [ ] Login
• [ ] Session
• [ ] Experiment Maintenance Tile
• [ ] Team Data Menu Maintenance
• [ ] Chat
• [ ] Help

• [ ] Facilitator Dashboard

  Care Coordination

• [ ] Model Diagram
• [ ] Outputs and Text
• [ ] Experiments
• [ ] Export function
• [ ] Save function
• [ ] Run/reset function
• [ ] "i" Information

• [ ] Team Data Table

  Medication Management

• [ ] Model Diagram
• [ ] Outputs and Text
• [ ] Experiments
• [ ] Export function
• [ ] Save function
• [ ] Run/reset function
• [ ] "i" Information

• [ ] Team Data Table

  Psychotherapy

• [ ] Model Diagram
• [ ] Outputs and Text
• [ ] Experiments
• [ ] Export function
• [ ] Save function
• [ ] Run/reset function
• [ ] "i" Information

• [ ] Team Data Table

  Aggregate

• [ ] Model Diagram
• [ ] Outputs and Text
• [ ] Experiments
• [ ] Export function
• [ ] Save function
• [ ] Run/reset function
• [ ] "i" Information

• [ ] Team Data Table

  Suicide Prevention

• [ ] Model Diagram
• [ ] Outputs and Text
• [ ] Experiments
• [ ] Export function
• [ ] Save function
• [ ] Run/reset function
• [ ] "i" Information
• [ ] Team Data Table

7. What kind of device are you using?

   • [ ] Tablet
   • [ ] Smartphone
   • [ ] PC

8. What browser were you using?

   • [ ] Chrome What version?
   • [ ] Edge What version?
   • [ ] Safari What version?

<< If you are highlighting a new idea, please continue >>

9. Please describe your idea below. Feel free to provide a scan or screen capture of a sketch of your idea.

10. Why is this idea important, in terms of your work?

11. How is this idea related to a problem you face in your daily practice?

12. Is there a deadline or other scheduled constraint driving your request that we should be aware of? This will help us to prioritize this request.

THANK YOU!!

---

STOP HERE. INFORMATION BELOW IS TO BE FILLED OUT BY WORKGROUPS AS PART OF TRIAGE AND WORK BREAKDOWN PROCESS

Executive Summary Need to add course/dose data to comparator for 6 HF templates. See table of EBP completer defs for ACT, PE, CPT, IBCT, IPT, CBT-D (it's different for each). Estimated person-hours to complete: 40 Estimated date for completion: 8/2/19 Lead team for effort: quant Key people (use @ assignment for people whose input will be necessary): @holbrooa @saveth @lzim

Step 1 - Complete the Interdependencies Table

Workgroup Leads review information provided by originator and collaborate to identify interdependencies. Then they describe modifications that are needed in the appropriate column and estimate hours needed to make changes:

● Workgroup Products - This column identifies the workgroups and the products for which they are responsible. This column can be modified by a workgroup lead on an ad-hoc basis to add one-time products or can be modified to include new and enduring products.

● Dependency? - This column is used by the workgroup leads during collaboration to identify the existence of an interdependency.

● Description of Change or Action Needed - Workgroup leads briefly describe the change required.

● Hours - Workgroup leads estimate the person-hours required to complete the task.

Workgroup Products	Dependency?	Description of Change Needed	Hours Estimate

Documentation
Brochures | | | MTL See Guide | | | MTL Say Guide | | | MTL Videos | | | Post Test | | | Investigators | | | Presentation | | | Quarterly Report | | | Annual Report | | | Qualitative | | | MTL Fidelity Checklists | | | Systems Thinking Codebook | | | CFIR Codebooks | | | Quantitative | | | Data UI | | | Data Queries | | | SharePoint | | | Excel Outputs | | | ModelParameters | | | tt Reports | | | Model CC | | | MM | | | PSY | | | AGG | | | SP | | | Sim User Interface | | | Model Diagram | | | "i"Information | | | Experiments | | | Outputs/Charts | | | Exports | | | Master Crosswalk Table | | |

Step 2 - Identify Constraints that may affect the team’s ability to complete the necessary actions.

Describe the constraint in terms of capability, capacity or affect on grant research goals. If there are no constraints identified, then leave the table blank.

● Type - The type of constraint identifies one or more resource areas that are affected.

● Workgroup - this is the workgroup responsible for developing a new or modifying an existing product.

● Discussion/recommendation - A brief narrative that expands the information regardingthe constraint and ideally a recommendation for overcoming the constraint.

Type Constraint (check all that apply) / Workgroup(s) Affected / Discussion/Recommendation

• [ ] Capacity
• [ ] Cost
• [ ] Schedule
• [ ] Interference
• [ ] Capability
• [ ] Other (describe)

Step 3 - Identify supported milestones. Affix the milestone at the issue level using available GitHub milestones.

Step 4 - Indicate the due date for completing the actions identified above, assuming constraints are removed.

Step 5 - Identify the lead workgroup for the action (check one):

• [ ] Documentation
• [ ] Investigators
• [ ] Quantitative
• [ ] Qualitative
• [ ] Model
• [ ] Sim UI

Step 6 - Go to the top of this section and complete Executive Summary information

Note to Workgroups: Use the issue thread to document further discussion, alternatives development and estimates. However, be sure to update this form if dates, team responsibility or other information changes.

[holbrooa]

I changed this issue to also apply to EBPharm dose, with the help of Jodie's resources.

[lzim]

@holbrooa and @saveth

Do we need to discuss where this fits in our overall priority list? On Monday, I know we prioritized documentation very heavily. Let me know if we need meeting time to cover this.

Thanks!

[holbrooa]

Code is written for this, but I'm still squashing bugs. I'm also doing my best to document more thoroughly as I go, so when the code is fully done and ready to be handed off, there will be documentation already available at the same time. That's slowing me down, obviously, so I'm not quite hitting the deadline of having this done today. I'm going to leave the due date as is, but just know that this work is "late" by virtue of the documentation-focused slowing down that we talked about, so perhaps that's not really a bad thing.

I went ahead and moved it to testing this morning. Which I guess I'm sort of using as "testing, bugfixing, and documenting" really.

[staceypark]

@holbrooa I moved the due date to the beginning following our convention of Title = due date + description.

Like we discussed at the Workgroup Leads meeting yesterday, #462, #615, and #424 are our top priority as we need them done to submit our protocols for the grants prior to launch.

You & @saveth will be at the AI conference starting 9/18 (and you have to travel out here) so we need these done by latest next Monday 9/16 as our top priority.

[saveth]

@holbrooa would you be able to provide any information on what the data structure or the column names will be like for the file you'll be providing to me? I would like to get a sense of how I should be thinking of the coding for #615. Thanks.

FYI: @lzim @staceypark

[holbrooa]

@saveth I'll end up with patient-level information tracking who hits completer status when. So what structure do we want for your analysis? Same as the initiate data, where I aggregate by sta6a and month for cy2018? That would make sense to me if it's part of the same analysis.

[saveth]

@holbrooa Since @lzim wants the EBP Dose measure to be in line with the EBP Initiate Calculation, then we do want it to be aggregated to the Sta6a level. Will this be similar to the EBPsy Initiate sheet or more like the BaseAndRx sheet? I'm assuming that the denominator will be the same, we're using the denominator from the BaseAndRx sheet that you've already provided?

[holbrooa]

I'm picturing just another column, or six columns if we want one per type of completer, on the EBPsy sheet. But will that work for what you need to do?

[saveth]

@holbrooa If you're planning on just updating the comparator data file, then a column of EBPDosePatients would be appropriate. With regards to the EBPharm Dose:

• Depression: I'm assuming this would be the sail43h and sail 47h numerators? Is this correct?
• AUD and OUD: are these the same people as those in the Initiate or are they a separate count, therefore new variables?

[holbrooa]

@saveth I think that's correct about the depression patients. I'm not sure yet about the AUD and OUD dose (the scripts we got are formatted very differently) but I'll let you know.

[lzim]

@holbrooa and @

Very glad to see some action on this “issue” and I hope that #615 and #462 are being tracked well, despite being separated FYI: @staceypark

AUD and OUD: are these the same people as those in the Initiate or are they a separate count, therefore new variables?

• I’m not sure I understand the operational definition of the phrase: “...are these the same people as those in Initiate...” Seemingly what is important according to how the tests of specific aims were written is what 1) EBP Reach (Initiate) and 2) EBP Reach (Course) are for a sta6a with monthly observations
• So, for me the clarity that I’m looking for is how we will assess (using the updated codes you have) this with a monthly observation.
• I welcome ongoing details here, and can track and try to support your work on this while I’m traveling. 🛫

[saveth]

@lzim @holbrooa As indicated in https://osf.io/z7y8m/wiki/home/ , the definition of AUD and OUD Initiate in the grant and how we operationalize these variables are counting anyone who has a prescription for AUD or OUD.

Given that the grant define course for EBPharm as anyone who has enough refills for a guideline-recommended trial for each medication, I do not know if this mean if they are counting the same people for Dose as Initiate or are different people? Not sure if the code @holbrooa got from Jodie help clarifies this or how this grant definition is being operationalize in the data query.

[holbrooa]

@lzim One bit of logic to double-check here, since you're monitoring:

In general, the main way EBP completers get captured is "X visits (with a template) in Y days" type of logic. For the purpose of these completers, I'm not using the concept of graduation, i.e. if you hit the threshold but continue getting more visits, that's ok, you still get credit. So then I have to "throw out" those subsequent visits, because we don't want to count a patient as having completed again in the following month. So in the CPT completer script where Jodie's team is giving credit if you get 8 visits in 180 days, I find the first time a patient hits that mark, i.e. the 8th visit in a span of 180 days or less, then throw out the visits after that so that you get counted once in the month where you "complete" but then you don't count again in the following month. E.g. a patient satisfies the above completion criterion for their visits 2 through 9, they get counted as a completer in the month that that 9th visit happened, and then I don't allow them to count again for visits 3 through 10, even though they technically satisfy the 8 visits in 180 days logic there. I'm fairly sure this is right, but if you have thoughts, this is a good time, before I hit the button and run queries that take hours.

tl;dr: I'm capturing completers regardless of "graduation" - so patients can continue getting templates after they've completed and I will count them once in the month where they first hit the completion criterion.

[lzim]

Right @saveth these are exactly the types of data structure and other details, that I am seeking clarity about too.

Except that, given that the unit of observation and analysis in the grant is sta6a each month...

1. it should be less critical whether a monthly sta6a observation for EBP Reach Initiate and EBP Reach Course are the same people for any given month.

   • Albeit, at an under specified conceptual level, it would seem that the numerator for EBP Reach Initiate at a given time point, should become an EBP Reach Course denominator at a later observation.

2. The key information being: Given how OMHSP typically handles an EBP course numerator and denominator, how does our monthly omnibus EBP Reach Course (measurement & EBPsy Course and EBPharm Course, respectively) jive with the typical quarterly observations used in typical operations.

Thanks @holbrooa and @saveth! 🤙🏽 FYI: @staceypark

[lzim]

@holbrooa

Our messages crossed in the ether...reading your post now on the runway...and will try to get a reply to you before I lose internet in the air ✈️

[lzim]

@holbrooa

• Your logic is making sense to me regarding not giving double-credit for a “completer”

• However, for the EBPsy templates, do you mean 8 visits in 180 days or 8 templates?

• I’m not sure whether there is any penalty to the site for removing them in subsequent months (to avoid double-counting), but it doesn’t seem like there would be.

• They’re also not really penalized (or at least, we’re not capturing in EBP Reach course) for not “graduating” folks.

• Just to track: OMHSP no longer uses an OR logic for either a. 8+ templates, or b. Checking a “complete” box on the template?

I’ll be in the ground in about ~2.5 hours, & I can help again then. I’ll be working this afternoon once I arrive.

Yes! Let’s get clear before you run very long queries ⌚️

[saveth]

@lzim Conceptually, I get that the numerator for Initiate eventually becomes the denominator for Dose as the proportion of those who complete treatments out of those who starts treatment.

However, all three grants clearly stipulate that: "We define reach as the proportion of patients diagnosed with OUD, AUD, PTSD, or depression (ICD-10 codes) who meet EBPsy and EBPharm 1a) initiation and 1b) course measures (numerator) divided by the total number of patients with these diagnoses (denominator) at that clinic station."

Therefore, when I get the data from @holbrooa, I will be operationalizing for #615 as:

1. EBP Reach Dose = (those who complete treatment)/(total number of patients with diagnosis) at each sta6a for each month

2. Then we we take the average monthly reach for 2018 for our baseline EBP Reach Dose.

Please clarify if this is not the correct procedure.

[lzim]

@saveth and @holbrooa

"We define reach as the proportion of patients diagnosed with OUD, AUD, PTSD, or depression (ICD-10 codes) who meet EBPsy and EBPharm 1a) initiation and 1b) course measures (numerator) divided by the total number of patients with these diagnoses (denominator) at that clinic station."

• Right. I believe your recap is accurate, and drawn from the grant.

• But, it is just underspecified (meaning, also underspecified in the grant) for the role of time in the EBP Course numerator/denominator and the issues that are discussed above by @holbrooa and I.

• Naturally, the maximum denominator is the eligible EBP diagnostic cohort for a sta6a, and the numerator is receipt of an EBP (simpler for EBP Reach Initiate).

• I believe we are on track working to resolve the differences with EBP Course tho, right?

• I will look to follow up on this with you both as I land! It’s a relatively short flight to Seattle

[holbrooa]

@lzim

• Roger

• Technically, neither. I mean this exact idea from the first sentence above: X visits (with a template) in Y days We know they often work on multiple templates in the same visit, so we can't just count templates. And in fact that's why the policy definition specifies templates completed on different days. So I filter to visits with a template, then I count those visits.

• There's no penalty per se, but it helps to explain some potentially funny data. Like, "wait, they're doing all these templates...but they're neither starting nor completing with very many patients." So the explanation is that they're either doing templates too slowly (probably the obvious explanation) or they're continuing treatment with their completers (the less obvious one that arises from this logic we're talking about). Or both.

• That's right. No penalty for not graduating completers. If we really wanted to, we could calculate a sort of "EBP efficiency" by comparing completers to templates or something.

• They are. And I am too. This graduation concept is contained in the "8+ visits" part, before we get to the early termination part. We do this part first, presumably because of how costly it is to go looking for those test scores (that complete box must be backed up by the relevant test score, per jodie's scripts).

[lzim]

@holbrooa

This all sounds very on track to me.

• good re: how handling templates and visits

• Right

• no need for this now

• great 👍🏾

If you have enough to proceed with this, then go for it 😃

[saveth]

@holbrooa I'm glad to hear that you got what you need to pull EBPsy Dose. Do you have enough info to operationalize the query for EBPharm Dose? If so, can you clarify for me on how we are counting EBPharm AUD and OUD?

@lzim

[lzim]

For @saveth’s question, and given @holbrooa’s double-checking.

So, it sounds like removed EBP Course folks (to avoid double-counting in subsequent months) will be handled via removal from both numerator and denominator (in subsequent months), to ensure an accurate EBP Course proportion for any given sta6a month.

[lzim]

BTW: Obviously still on the tarmac 😂

[saveth]

@holbrooa @lzim Based on the comments above, it sounds like I will be getting new numerators and new denominators variables for EBP Dose calculation, which is separate from the EBP Initiate calculation. Is this correct?

[holbrooa]

@saveth re: AUD and OUD: are these the same people as those in the Initiate or are they a separate count, therefore new variables? I'm not sure I'm following this question. If you mean new columns in the data table, then yeah. I will have new columns for each kind of completer. As for the same people, yes sort of. I mean, of course you have to get some of the drug (our initiate def) in order to get a therapeutic dose. But not necessarily in the same month. Or, looking at the depression one, definitely not in the same month.

[lzim]

@holbrooa @lzim Based on the comments above, it sounds like I will be getting new numerators and new denominators variables for EBP Dose calculation, which is separate from the EBP Initiate calculation. Is this correct?

@saveth Yes, I believe so. Based on the discussion above, but also conceptually both the numerator and denominator have to be different for EBP Course

[holbrooa]

@saveth I'm trying to get EBPsy correct and to you first. Today, if at all possible (but these queries will take a long time). So I might have to come back to some of these ebpharm questions later.

@lzim I'm not understanding the concept that causes the denominator to have to be different for EBP Course. Might have to follow up once you're back on the ground.

[saveth]

@saveth re: AUD and OUD: are these the same people as those in the Initiate or are they a separate count, therefore new variables? I'm not sure I'm following this question. If you mean new columns in the data table, then yeah. I will have new columns for each kind of completer. As for the same people, yes sort of. I mean, of course you have to get some of the drug (our initiate def) in order to get a therapeutic dose. But not necessarily in the same month. Or, looking at the depression one, definitely not in the same month.

@holbrooa that's what I was originally thinking you'll produce. New columns with the EBP Dose counts of patients and we're using the same denominators (diagnostic cohort) for our reach. However, @lzim seems to suggests that how EBP Dose should be calculated will be different from how EBP Initiate was calculated, therefore requiring a different set of numerator and denominator for each month.

[staceypark]

it sounds like we need to hold until @lzim provides more clarification so @holbrooa can give the correct pieces to @saveth ?

[holbrooa]

My scripts are running. Fingers crossed. Re-reading this thread, I see now where I missed what @lzim was saying about the new denominators. I hadn't thought of removing those patients from the denominator. I'm trying to wrap my head around how that would work.

[lzim]

@saveth @holbrook @staceypark

On the ground at SEA!

It’ll be a minute before I get to a place where I could talk to any of you about this.

But, where on a surface level (grant overview level), it makes sense to consider the exact same eligible diagnostic cohort denominator with either a...

1. EBP Reach Initiate numerator (to get a proportion/%) or a...
2. EBP Reach Course numerator (to get a proportion/%)...

I just want to think through kind of lag issues described above related to EBP Course, that could potentially lead to under-counting or double-counting, etc.

[holbrooa]

Ok. Sounds like maybe we should talk about it tomorrow. For now, I've put a new data file in the folder for @saveth and I will add the pharm columns either tonight or in the morning, depending on whether or not my queries for that finish.

[lzim]

@saveth and @holbrooa

Okay, sure, thanks for all the work to clarify this today. It would be great for you to discuss at the meeting and document your further considerations/decisions in this issue.

I can provide remote support💻, but only via online/text as I have SIRC responsibilities all day tomorrow.

This is the question I’m considering with you both:

• Given how the query is run, if the exact same diagnostic cohort denominator is used for EBP Reach initiate and EBP Reach Course, AND

These two things are true:

X visits (with a template) in Y days" type of logic

• BTW: Despite my confusing question, given all the units flying around (patients, visits, templates 🤓) I appreciated the extra detail you provided explaining how this was handled considering both visits and templates.

And this, I added bold below:

I find the first time a patient hits that mark, i.e. the 8th visit in a span of 180 days or less, then throw out the visits after that so that you get counted once in the month where you "complete" but then you don't count again in the following month. E.g. a patient satisfies the above completion criterion for their visits 2 through 9, they get counted as a completer in the month that that 9th visit happened, and then I don't allow them to count again for visits 3 through 10, even though they technically satisfy the 8 visits in 180 days logic there.

• Despite our discussion of capturing EBPsy completers the month they complete, avoiding EBP course “double-credit,” and the potential for “penalties” ...

• I am considering that if the eligible diagnostic cohort for a sta6a in any given month excluded prior completers only from the numerator (but NOT the denominator), then it would penalize a clinic in any given month when a previous “completer” continues in care (9th or more based on the quote above) and is removed from the numerator, but is still in the denominator based on diagnosis.

• This type of concern brings up the purpose of aim 1, which is to tell us both how many patients get touched with an EBP at all (initiate), versus how many patients complete an adequate therapeutic course/dose adequate to meet their need.

• The monthly observation is a key part of our analyses, but arbitrary for capturing the conceptual purpose of aim 1 immediately above.

• It would be clearer to explicitly translate the EBP Reach Initiate numerator to become the EBP Reach Course denominator, EXCEPT for the issue of “in any given month” and the lag between when EBP initiation and EBP Course occur and are captured.

• It was for these “lag” issues related to estimating an EBP Course proportion, in any given sta6a month, that would necessitate removing prior completers from both numerator and denominator, in order to avoid “penalizing” clinics for their prior completers.

[lzim]

Looping in @staceypark ☝🏽

[staceypark]

@lzim By "translate" do you mean we need another equation?

It would be clearer to explicitly translate the EBP Reach Initiate numerator to become the EBP Reach Course denominator, EXCEPT for the issue of “in any given month” and the lag between when EBP initiation and EBP Course are captured.

[lzim]

@staceypark

@lzim By "translate" do you mean we need another equation?

No, this statement (also note the EXCEPTION) should’t be considered in isolation of the overall thread from today.

Your quote from me is just one part of my multi-bullet attempt to describe what the potential issue is that I am attempting to clarify/resolve.

[staceypark]

@holbrooa @saveth

• Look to the table called "EBP Completers" in the comparator data set for dose definitions for EBPsy

[holbrooa]

Jodie's code for EBPharm doesn't seem to have dose/completion logic. It has some complicated things, but they're just alternate sources of the same information, like CPRS orders for the drugs, and non VA meds sources of the drugs. The result is still just counts people getting the rx, like we've done for our initiate columns. And since we're limiting ourselves to outpatient mh clinics, I don't think these other sources of meds are necessary.

@saveth @lzim I guess that means we need our own definition or another source of a therapeutic dose of AUD and OUD ebpharm. I'm not even sure what we're looking for. Some kind of continuity threshold like we are using for the depression meds?

[holbrooa]

@lzim we've been thinking through the denominator lag issue you brought up in the quant meeting. We have some ideas, but I think we need your input to make a final decision. The question is: when do we count the completion, i.e. which month does a completer get captured in? Theoretically, there are a three options. Clearest with an example:

A patient does CPT, needs to get 8 visits within a six month period to "complete" and they do so on their visits 10-17. We can count them as a completer in three different months:  the month of their first visit (I'll call this "true initiation"), the month of their tenth visit (I'll call this "dose initiation"), or the month of their 17th visits ("completion"). Currently we're counting them in the month of completion. This patient has already been counted as an initiator in the month of their true initiation. Options:

1. Status quo where we capture completion. This is pretty easy to explain, but when we go to do division, it'll give us some weird results, because we're not following a cohort of patients. But from a workload or care-delivery standpoint, it does make sense.
2. This option would capture the completers at their true initiation. This is a "cohort" model; we follow those initiators and see who completes, even if it's way in the future, and then count them back in that same true initiation month. This is very reasonable from a patient perspective for doing division, but weird from a care-delivery standpoint. Using the example above, the 10-17 visits might happen over March-May 2019, but they get counted as a completer way back in January 2018 because that's when they got that true initiation visit.
3. The other option is to capture completers at their dose initiation. This also makes sense from a temporal care delivery standpoint, but is weird for division because again we're not following certain patients.

Option 2 makes the most sense for using our initiation numerator as the denominator for dose, as you mentioned above. But there are clearly some reasons to prefer the other options as well.

[holbrooa]

Option 2 also makes our dataset much less static/stable, and introduces a bit of a bias toward underestimation, getting stronger as we look at later months. This is probably obvious from the March-May 2019 example above, but just to make it explicit.

[saveth]

@lzim @holbrooa @staceypark Option 2 makes the most sense in terms of calculation and being able to explain it to scientists. IF we don't use our initiation numerator as our denominator, then we are deviating from the grant's definition and we'll have to explain to the scientific audience that our "cohort" of initiators are not the people we're following and capturing in the analysis.

[holbrooa]

For options 1 and 3, we might want to consider a different denominator, e.g. the diagnostic groups, instead of the initiator count. Both options seem pretty reasonable.

[lzim]

@holbrooa and @saveth

I’m sorry about the delay responding to these questions...it was getting quite confusing to track with the level of issue_incoming I was fielding across the team yesterday.

And since we're limiting ourselves to outpatient mh clinics, I don't think these other sources of meds are necessary.

• Other sources of meds for OUD EBPharm may indeed be key if they are receiving methadone at a methadone clinic, but otherwise have visits managed in VA outpatient care. For methadone, this is common, as there just aren’t that many methadone clinics.
• However, I do not expect this to be as common (or happen at all, in fact) for OUD Bupenorphine or AUD EBPharm

need our own definition or another source of a therapeutic dose of AUD and OUD ebpharm. I'm not even sure what we're looking for. Some kind of continuity threshold like we are using for the depression meds?

• Right, we do need something similar and we do have to create it ourselves.
• I have all my notes from Jodie’s recommendations, but unfortunately they are in my office in Menlo Park. However, @staceypark may be able to photograph them and send them to me for re-assessment.
• I do believe Jodie understood what we needed, and I believe she thought she could get it from the code she sent.
• Since we do need to create our own sta6a/monthly definition, how can I help you propose/develop this ☺️

[lzim]

...working on thinking through the denominator issue now...

[lzim]

@holbrooa and @saveth

• Agreed that this is the key issue that we are working to clarify:

  The question is: when do we count the completion, i.e. which month does a completer get captured in?

• Our scientific audience reviewing the designs and results of a cluster randomized trial (CRT) are principally concerned with unbiased estimation of the difference between our two trial arms due to the intervention received in each arm (MTL vs. AF/uQI) - i.e., a phase III superiority CRT design. Therefore, considerations we have made for other parameter estimates (e.g., conserving staff time in models), are not primary here.

• Example of the expected description of tests of specific aim 1 for omnibus EBP Reach Initiate: “The average omnibus (i.e. depression, OUD and PTSD) EBP Reach Initiate was 4% among MTL clinics during the twelve months pre-intervention and 4% among AF/uQI clinics. The average proportion of omnibus EBP Reach Initiate during the 12 months post-MTL intervention was 9% and remained 4% in the AF/uQI clinics. As hypothesized, the pre/post change in EBP Reach Initiate is a difference in proportions of 5% across trial arms (95% CI XX-XX).”

• will post more once I switch to next session

[saveth]

@holbrooa @lzim I'm thinking through the EBPharm Depression. The denominator for sail43h and sail47h are the same people (depression patients newly treated with an antidepressant medication), so we only used the denominator for sail47h as our numerator for the EBPharm Depression initiation (so we're not double counting). The numerators for sail43h and sail47h are of those people who are treated with a new antidepressant medication who receive treatment for 84 days and 180 days, respectively. I'm thinking that I should be adding the sail43h and sail47h numerators to be used for our EBPharm Depression dose. Does this make sense? Is there a chance that we may be double counting people in the numerators?

[holbrooa]

@lzim @saveth The example you give for expected description of tests makes a lot of sense. And it seems to me that that means we can't use option 2.

[lzim]

Busy with the conference and just getting a chance to turn back to this:

We’re considering the role of the 8+ visits (with a template) in 180 days logic when we have either a 6-month or 12-month, monthly average, EBP reach observation period (for R21 & R01; 6-month for IIR)

• In specifying the Specific Aims analyses, I was trying to lift out the purpose of estimating a 6- or 12-month (drawing from the R21/R01) period EBP reach estimate for 1) pre/post intervention, and 2) initiate and course.
• The purpose of our R01/IIR trials is a clean, unbiased comparison of MTL vs AF/uQI interventions (or matched comparator sites for the R21)
• The importance of accurate monthly sta6a EBP reach observations is to enable the hypothesis tests according to specific aims, i.e., we do NOT want to use EBP reach measures that would make our trial arms incomparable because of when our EBP reach observation (Initiate or Course) is pulled relative to when trial interventions (MTL or comparator) occur in the clinics.

On the 3 “completer definitions” from @holbrooa’s example (I’ve added “EBP”)

1. the month of their first visit ("true EBP initiation") - our aim 1a “EBP Reach Initiate” measure
2. the month of their tenth visit ("EBP dose initiation") - aim 1b “EBP Reach Course” measure
3. the month of their 17th visits ("EBP completion") - still our aim 1b “EBP Reach Course” measure - Although this last visit seems closer to what we have operationalized as “graduation” in the PSY module of Modeling to Learn, due to averaging monthly EBP reach, they should still be included in the EBP Reach Course measure in any given month that they meet the criteria for the EBP within the 6-month (IIR) or 12-month (R21 & R01) observation window.

• Sounds like Option 3 to me: We will want to capture when they first “qualify” for the EBP Reach Course numerator (“dose initiation”) within the 6- or 12-month pre OR 6- or 12-month post observation window
• And, they need to count toward the EBP Reach Course numerator, in any given month, for as long as they are engaged in care (i.e., in the denominator) during the 6- or 12-month pre/post intervention period.

@lzim @saveth The example you give for expected description of tests makes a lot of sense. And it seems to me that that means we can't use option 2.

• I agree that there seem to be many problems with option 2.

Quoting @saveth but LZ added emphasis

depression patients newly treated with an antidepressant medication

• While at first the depression SAIL measures MDD43h (86 days) and MDD47h (180 days) may seem to really be EBP Course measures in terms of how the grant conceptualizes it, consistent with how SAIL is operationalized, and in terms of implementation science, the key idea is that the “depression evidence-based practice or ‘EBP’” is being evaluated for a therapeutic response after starting anti-depressants, not a simple anti-depressant script/fill/release. So, even though this is looking at care continuity, it is actually still an “EBP Initiate” measure.

• I’m not sure I’m understanding the rest of your post re: adding numerators over the same denominator? It seems it definitely would double-count them, unless I’m missing your proposal. - - Including in the numerator any patients who meet the EBP measure - in any given sta6a month - across the 12 month pre/post period of observation for any given sta6a month in which they are in the denominator_ is what we need.

• Although there are differences across the 3 grants (R21, R01 and IIR) in monthly EBP Reach observation windows, and the diagnostic cohorts, we want to work for parallel EBPsy & EBPharm Reach Course definitions in terms of timing because each study design intends to determine whether clinics that get MTL have superior improvements in pre/post EBP reach to clinics that do not (R21 comparator clinics; R01 AF clinics, and IIR clinics)

FYI: @staceypark

[saveth]

@holbrooa @lzim below is a clear explanation of the formulae I have in mind.

in Data file MDD43h = sail43h in Data file MDD47h = sail47h

For calculation of SAIL MDD formula is:

Na = number of patients who received 84 days of antidepressant medication Nb = number of patients who received 180 days of antidepressant medication Td = any patient diagnosed (dx) with depression (MDD) and newly treated with an antidepressant

MDD43h = Na/Td MDD47h =Nb/Td

THEN

EBPharm Reach for MDD formula is

Pd = in any given month is the number of patients diagnosed with MDD Td = any patient dx with MDD and newly treated with an antidepressant (same Td from sail measure above) Na = number of patients who received Rx for 84 day ( same Na from sail measure above) Nb = number of patients who received Rx for 180 days (same Nb from sail measure above)

EBPharm Reach Initiate = Td /Pd EBPharm Reach Dose = (Na + Nb) /Pd

@lzim Based on your comment: "even though this is looking at care continuity, it is actually still an “EBP Initiate” measure", means that we still need to figure out how to pull the EBPharm Dose numerator and denominator for AUD, OUD, and Depression.

Based on the explanation to Andrew's comments, it sounds like Andrew does need to pull a different set of denominators for EBPsy.

Based on my understanding for EBPsy: dx = patients who are diagnosed with PTSD, Depression or AUD Ts = True Initiation of EBPsy -- first visit T8s = Dose Initiation- the start of an EBPsy dose treatment within six or 12 months of treatment T8f = Dose Completion- completion of EBPsy treatment

EBPsy Reach Initiate = Ts/dx (which is what we already have and calculated) EBPsy Reach Dose = T8f/T8s (what Andrew still needs to pull)

and it sounds there needs to be two sets of T8s, one where dose initiation is within 12months and one where T8s is dose initiation is within 6months.

[lzim]

@lzim Based on your comment: "even though this is looking at care continuity, it is actually still an “EBP Initiate” measure", means that we still need to figure out how to pull the EBPharm Dose numerator and denominator for AUD, OUD, and Depression.

• @saveth absolutely we do (see my post above from this morning)
• NOTE: “EBPharm Reach Initiate for OUD and AUD do not have this same “extra” complication that depression has.