search

lzim / teampsd

Team PSD is using GitHub, R and RMarkdown as part of our free and open science workflow.
GNU General Public License v3.0
9 stars 23 forks source link

9/16 EBPsy and EBPharm course data #462

Closed holbrooa closed 4 years ago

holbrooa commented 5 years ago

ORIGINATOR Please follow the steps below to open an issue for the MTL Workgroups to address. Be sure to provide a descriptive title for the issue when saving.

Step 1 - Please assign this issue to the issue_tracker, under the “Projects” menu on the right side of your GitHub page.

Step 2 - Please answer the questions below by placing a lowercase "x" between the applicable brackets below and providing brief narratives and screenshots where appropriate:

  1. Is this issue a problem, such as a malfunction in the simulation, an error in documentation or similar concern, or is this a new idea we should consider?

    • [ ] Problem
    • [x ] New idea

  2. What team are you on (exg., Menlo Park, Team Blue)?

  3. How urgent is this issue?

    • [x] Urgent - should be addressed right away!
    • [ ] Non-urgent - can be prioritized within normal workstreams.

  4. In the space provided below, please describe the information or behavior you expected from the product or otherwise describe what you expected to see or hear.

Need to add course/dose data to comparator for 6 HF templates. See table of EBP completer defs for ACT, PE, CPT, IBCT, IPT, CBT-D (it's different for each).

And also use scripts and tables from Jodie's team for EBPharm dose.

  1. Please provide a screenshot of the bug, if possible. Make as many screenshots as you like that would be helpful to illustrate the issue.

    << Paste screenshots here>>

  2. Within which product did you notice the problem? Check all that apply

Data Products

Instructional Products

Simulations

Home

  1. What kind of device are you using?

    • [ ] Tablet
    • [ ] Smartphone
    • [ ] PC

  2. What browser were you using?

    • [ ] Chrome What version?
    • [ ] Edge What version?
    • [ ] Safari What version?

<< If you are highlighting a new idea, please continue >>

  1. Please describe your idea below. Feel free to provide a scan or screen capture of a sketch of your idea.

  2. Why is this idea important, in terms of your work?

  3. How is this idea related to a problem you face in your daily practice?

  4. Is there a deadline or other scheduled constraint driving your request that we should be aware of? This will help us to prioritize this request.

THANK YOU!!

STOP HERE. INFORMATION BELOW IS TO BE FILLED OUT BY WORKGROUPS AS PART OF TRIAGE AND WORK BREAKDOWN PROCESS

Executive Summary Need to add course/dose data to comparator for 6 HF templates. See table of EBP completer defs for ACT, PE, CPT, IBCT, IPT, CBT-D (it's different for each). Estimated person-hours to complete: 40 Estimated date for completion: 8/2/19 Lead team for effort: quant Key people (use @ assignment for people whose input will be necessary): @holbrooa @saveth @lzim

Step 1 - Complete the Interdependencies Table

Workgroup Leads review information provided by originator and collaborate to identify interdependencies. Then they describe modifications that are needed in the appropriate column and estimate hours needed to make changes:

● Workgroup Products - This column identifies the workgroups and the products for which they are responsible. This column can be modified by a workgroup lead on an ad-hoc basis to add one-time products or can be modified to include new and enduring products.

● Dependency? - This column is used by the workgroup leads during collaboration to identify the existence of an interdependency.

● Description of Change or Action Needed - Workgroup leads briefly describe the change required.

● Hours - Workgroup leads estimate the person-hours required to complete the task.

Workgroup Products Dependency? Description of Change Needed Hours Estimate

Documentation
Brochures | | | MTL See Guide | | | MTL Say Guide | | | MTL Videos | | | Post Test | | | Investigators | | | Presentation | | | Quarterly Report | | | Annual Report | | | Qualitative | | | MTL Fidelity Checklists | | | Systems Thinking Codebook | | | CFIR Codebooks | | | Quantitative | | | Data UI | | | Data Queries | | | SharePoint | | | Excel Outputs | | | ModelParameters | | | tt Reports | | | Model CC | | | MM | | | PSY | | | AGG | | | SP | | | Sim User Interface | | | Model Diagram | | | "i"Information | | | Experiments | | | Outputs/Charts | | | Exports | | | Master Crosswalk Table | | |

Step 2 - Identify Constraints that may affect the team’s ability to complete the necessary actions.

Describe the constraint in terms of capability, capacity or affect on grant research goals. If there are no constraints identified, then leave the table blank.

● Type - The type of constraint identifies one or more resource areas that are affected.

● Workgroup - this is the workgroup responsible for developing a new or modifying an existing product.

● Discussion/recommendation - A brief narrative that expands the information regardingthe constraint and ideally a recommendation for overcoming the constraint.

Type Constraint (check all that apply) / Workgroup(s) Affected / Discussion/Recommendation

Step 3 - Identify supported milestones. Affix the milestone at the issue level using available GitHub milestones.

Step 4 - Indicate the due date for completing the actions identified above, assuming constraints are removed.

Step 5 - Identify the lead workgroup for the action (check one):

Step 6 - Go to the top of this section and complete Executive Summary information

Note to Workgroups: Use the issue thread to document further discussion, alternatives development and estimates. However, be sure to update this form if dates, team responsibility or other information changes.

lzim commented 5 years ago

EBPharm Reach Initiate = Td /Pd]

Depression EBPharm Reach Initiate = (Given Td, Na)/Pd

lzim commented 5 years ago

So, based on my post above...

Depression EBPharm Reach Course = (Given Td, Nb/Pd)

FYI: @saveth @holbrooa and @staceypark

saveth commented 5 years ago

@lzim Based on your clarification:

Depression EBPharm Reach Initiate = (Given Td, Na)/Pd Depression EBPharm Reach Course = (Given Td, Nb/Pd)

  • Pd = in any given month is the number of patients diagnosed with MDD
  • Td = in any given month any patient dx with MDD and newly treated with an antidepressant (same Td from sail measure above)
  • Na = in any given month, the number of patients who received Rx for 84 day ( same Na from sail measure above)
  • Nb = in any given month, number of patients who received Rx for 180 days (same Nb from sail measure above)

@lzim What about the formulae I've outlined for EBPsy Dose?

lzim commented 5 years ago

@saveth I keep trying to return to this, but it is a VERY busy conference for me. 💃🏻 FYI @holbrooa and @staceypark

Given the updated formula, you've provided, this means I'll need to to revised all the calculations and analyses I've done for EBP Reach Initiate involving EBPharm Depression.

RETAIN for R01/IIR SITE SELECTION using CY2018: Current depression EBPharm Reach Initiate Calculation

EBPharm Reach Initiate = Td /Pd Pd = in any given month is the number of patients diagnosed with MDD Td = in any given month any patient dx with MDD and newly treated with an antidepressant (same Td from sail measure above)

CONSIDERING additional specificity for TESTS of SPECIFIC AIMS in 6- or 12-month pre/post observation window: Aim 1a) Depression EBPharm Reach Initiate = (Given Td, Na)/Pd

Pd = in any given month is the number of patients diagnosed with MDD Td = in any given month any patient dx with MDD and newly treated with an antidepressant > (same Td from sail measure above) Na = in any given month, the number of patients who received Rx for 84 day ( same Na from sail measure above)

Aim 1b) Depression EBPharm Reach Course = (Given Td, Nb/Pd)

Pd = in any given month is the number of patients diagnosed with MDD Td = in any given month any patient dx with MDD and newly treated with an antidepressant (same Td from sail measure above) Nb = in any given month, number of patients who received Rx for 180 days (same Nb from sail measure above)

saveth commented 5 years ago

@lzim 😌 Glad to hear we don't have to redo everything for site selection and that what we're discussing now is for analysis of the AIMS.

lzim commented 5 years ago

@saveth @holbrooa @staceypark

still TBD, but proposed/discussed above

and it sounds there needs to be two sets of T8s, one where dose initiation is within 12months > and one where T8s is dose initiation is within 6months.

  • No. Nothing like this at all in terms of defining when we say patients have met the criteria for EBP Course, and deciding which months to count patients in the EBP Course numerator once they have met this criteria.
  • I proposed use of Andrew’s “Option 3” and I proposed that after meeting this EBP Course threshold for being in the numerator, they should continue to count toward EBP Reach Course for any given month that they meet criteria to be in the monthly sta6a denominator.
  • But yes, if you’re tracking the key from my post above about the study design, in that we have both a pre/post 6-month, average monthly, EBP Reach observation window (IIR), and pre/post 12-month, average monthly, EBP reach observation window (R21 & R01) that are separate from the “EBPsy completion” logic Andrew is working with from Jodie/OMHSP.
holbrooa commented 4 years ago

In the meeting, we decided to use Andrew's option 1 from above, where we count EBPsy completers exactly once in the month where they first meet the completion criteria. This is how the current comparator dataset is, and the definitions are in the defs and EBPCompleters tabs.

saveth commented 4 years ago

In the meeting (https://meet.lucidmeetings.com/lucid/meeting/224690 ), we've decided how w're calculating EBPsy and EBPharm Initiate is fine. For course/dose:

EBPsy Course = EBPsy complete / EBPsy diagnostic cohort, where

EBPharm Course = (sail43hnum + oud84days + aud84days)/ EBPharm diagnostic cohort, where

lzim commented 4 years ago

DECISIONS MADE at https://meet.lucidmeetings.com/lucid/meeting/224690

1. We will use the "true initiation" or first eligible script or visit/template as our EBP Reach Initiate definition for any given sta6a month, as we have been.

(i.e., includes 3 unique criteria for the depression numerator not accounted for in OUD and AUD:

  1. not seen for depresion within last year

  2. a new anti-depressant scraip (i.e., has not had a prescription filled within last 105 days)

  3. must have had an encounter  within 120 days of the prescription fill.)

2. For EBPsy Reach Course we are counting only the month when a patient firsts meet the criteria worked out by OMHSP, for example X visits in Y days.

3. For EBPharm Reach Course we are paralleling SAIL MDD43h for AUD and OUD, without the special "new anti-depressant" rules applied (i.e., includes 3 unique criteria for the depression numerator not accounted for in OUD and AUD:

  1. not seen for depresion within last year

  2. a new anti-depressant scraip (i.e., has not had a prescription filled within last 105 days)

  3. must have had an encounter  within 120 days of the prescription fill.)

4. WE CONCLUDED HAVE TO USE ISSUE DATE TO TRACK REFILLS OF THE SAME PRESCRIPTION for the NUMERATOR

- RELEASE DATE: in the denominator, captures a patient who got a script for that diagnosis in a given sta6a month.

5. We decided NOT to work with MDD47h, but rather to work with MDD43h across all EBPharm.

NEXT STEPS

A.  Andrew will change the patcountDeprx, patcountAUDrx AND OUDRx to use Issue Date for the Numerator

1. Andrew will create the AUD and OUD EBP Course Numerators following MDD43h as a guide, but at the sta6a month level, and without applying the "depression exclusion."

  1. Will use Issue Date across the EBPharm Numerators

B. Savet will add near the Latex formulae the formula that correpond to the specific columns/variables used in the table below.

CONCLUSIONS

holbrooa commented 4 years ago

Okay. There is a new comparator dataset. @saveth , please go check it out. It has the new columns we talked about, and I've updated the definitions tab.

saveth commented 4 years ago

@holbrooa Thanks! Can you also update the definitions on OSF (https://osf.io/z7y8m/wiki/Data%20Definition/) for your part? I'll update my part where it references your variables as I work on the dataset.

saveth commented 4 years ago

@holbrooa @lzim There's something wrong with how the new OUD initiate is being pulled. I'll email the list of these sites.

holbrooa commented 4 years ago

@saveth I'm still looking into this. It probably has something to do with the new non-VA drugs part of the code. I'll let you know when I figure it out.

lzim commented 4 years ago

Thanks to you both for working on this! GitHub Trackers are a useful space for project breakdown, clarification of next steps, and conceptual discussion.

Reminder that any detailed discussion to resolve this must occur on our secure servers. 🔐

Thanks!

saveth commented 4 years ago

@holbrooa I pinged you on OSF (https://osf.io/z7y8m/wiki/Data%20Definition/) on how I'm calculating the reach and identified how the count of OUD initiate treated patient is higher than the OUD diagnostic cohort.

saveth commented 4 years ago

@holbrooa Thank you for looking into the fuzziness of the data. I agree with the solution of capping the average to 100% to deal with people being issued a prescription one month and picking it up the next month. This way we don’t have two sets of EBPharm denominator definitions. @lzim , do you agree?

With regards to how the average EBP measure is calculated, I’m trying to be consistent to how it has been stated in the granted on how we are doing the analysis. I believe the goal of looking at the average monthly EBP reach is to get a sense of, for a given month, how much of a reach is the site getting. Instead of saying for a given year, what is their overall reach. It is up to @lzim if she would prefer to do a yearly average EBP reach instead of the average of 12 monthly EBP Reach. @lzim, do you want me to update my calculation as Andrew is proposing or stay the course on how we have already defined and implement the calculation?

FYI: @ritahitching @staceypark

holbrooa commented 4 years ago

@lzim @saveth @staceypark

This question about the issue -> release and averaging computation is the last nagging issue before I can run the whole query for all the years to get @saveth what she needs. So then my part of https://github.com/lzim/teampsd/issues/615 and https://github.com/lzim/teampsd/issues/424 and https://github.com/lzim/teampsd/issues/476 will be done. (And https://github.com/lzim/teampsd/issues/672 if that's a legit issue.)

We added some documentation to osf about this, and of course all the different defs are there, but I guess we probably just need to briefly talk about it in the quant meeting? It shouldn't take a long time. I think we have a pretty clear options A, B, C. I could even try to type them out here if you want, instead of going and putting them in the meeting agenda.

saveth commented 4 years ago

@lzim We've answered your questions in OSF (https://osf.io/z7y8m/wiki/definition_details_needed/) during the quant meeting. We still need an answer to our question in this tread, which we've posted in OSF (https://osf.io/z7y8m/wiki/Data%20Definition/) with, hopefully, enough detail to help you make a decision.

FYI: @holbrooa @saveth @ritahitching @staceypark

lzim commented 4 years ago

Great thank you!!! @holbrooa @saveth @ritahitching @staceypark

To consider options A,B,C for OUD EBPharm (and likely AUD) we need definitions of Issue date: and Release date:

Please see my edits to clarify on OSF: https://osf.io/z7y8m/wiki/Data%20Definition/

lzim commented 4 years ago

Also FYI: @holbrooa @saveth @staceypark and @ritahitching

Separate from us trying get to get the Issue Date and Release Date and therefore, the validity of the EBPharm Reach Initiate measure definitions crystal clear here: https://osf.io/z7y8m/wiki/Data%20Definition/

@saveth is correct, for a variety of additional proposed analyses, we need average monthly EBP Reach observations for each clinic of cluster size ~800 or larger, and not annual EBP reach.

As we are doing, working together to think this all through carefully, clarifying distinctions between the average monthly period EBP reach and the annual period EBP reach is challenging for many reasons, including the need to account for both new and existing patients, different types of evidence-based care, care episodes of variable duration, differences in primary presenting concerns, as well as the possibility of continuing in care in ways that are no longer evidence-based. Even the differences between issue date and release date require close review.

We are working to assess the proportion of an eligible clinic population that is receiving EBPs at the same point in time across our arms. And, expect a more valid comparison, if our denominator and numerator are “refreshed” monthly as we have proposed.

I continue to welcome everyone’s best thinking about all of this, and very strong documentation. Hopefully, we can pin down the last remaining questions on OSF and wrap this up. Thanks for all your hard work and careful checking with each other and the team 🤗

lzim commented 4 years ago

@holbrooa @saveth @staceypark @ritahitching

Response to what I’m seeing on OSF for options A and B** information will hopefully still be forthcoming about the definition of “Issue Date” and “Release Date” to consider that Option C** denominator question you are asking about

I believe this is what is being described as option B: “Average the monthly percentages by summing the numerator and denominator, then dividing. We still would want to cap the result.”

Sum of total EBPs received over the period. over the Sum of total eligible patients over the period.

This first solution would be the population EBP reach (proportion). This would answer the question "What proportion of total eligible patients over the observation period received an EBP."

This is what is being described as option A on OSF: “Average the monthly percentages the way it is currently doing it and just cap the result at 100%. This is computing monthly average reach percentages the same as a quantity: sum the values and divide by 12.“

So, just to check, consider Option D: I believe that the approach we need is the mean clinic EBP reach (proportion). This answers the question "What was the average EBP reach (proportion) among eligible patients in each clinic month".

We find the average EBP reach (proportions) in each clinic (sta6a) month, in which each clinic has a different number of eligible patients in any given month, and therefore each EBP reach value estimated has a different denominator.

Yes, we want to sum the EBP reach proportions for each clinic, but divide by the number of clinic month as: ((EBP Received in Clinic 1 in month 1/Eligible population in Clinic 1 in month 1)...(EBP Received Clinic n in month n/Eligible population Clinic n in month n))/n clinic months.

Option B and D methods will produce different results. Both would be valid but have different interpretations.

Thanks! 👍

lzim commented 4 years ago

@holbrooa and @saveth FYI: @ritahitching and @staceypark

Due to my competing demands, I was unsatisfied with the amount of time I had to respond to this issue yesterday.

I've done more thinking about the differing averaging of proportions options, continuing to consider the role of the

  1. Denominators, especially any small clinics who say reach 1 out of 2 patients with EBPs.
  2. Numerators: The differences we observe with the summing of numerators, summing of denominators, and dividing, versus accounting for each unique clinic and month, by the number of clinic observations, especially with regard to the behavior of zeros in the numerator.
  3. The need to get the documentation down for all of @holbrooa's steps and @saveth's to get to the bottom of this, which applies quite broadly.

In fact, it made me return to Simpson's Paradox

Example 1            
  Jan Feb Mar Summed Num/Summed Denom EBP Reach Average EBP Reach for 3 Obs.  
Clinic 1 12.5% (25/200) 33.33% (30/90) 100% (50/50) 30.1% (25+30+50)/(200+90+50) 48.6% (0.125+0.333+1)/3  
Clinic 2 47.1% (40/85) 4% (12/300) 48% (60/125) 21.96% (40+12+60)/(85+300+125)*100 33% (0.471+0.04+0.48)/3  
Month Summed Num/Denom 22.8% (25+40)/(200+85) 10.8% (30+12)/(90+300) 62.9% (50+60)/(50+125) 25.5% (25+30+50+40+12+60)/(200+90+50+85+300+125) 40.8% (0.125+0.333+1+0.471+0.04+0.48)/6
Month Averaged 29.8% (0.125+0.471)/2 18.7% (0.333 +0.04)/2 74% (1 + 0.48)/2      
Example 2        
Clinic Type EBPsy EBPharm Omnibus EBP Reach Summing Omnibus EBP Reach Averaging
Clinic 1 (small CBOC) 0% (0 EBPsy/1 Pts)* 100% (1/1) (1 EBPharm/1 Pt) 50% (1/2) 50% (0 + 1)/2
Clinic 2 (larger Med Ctr) 1.3% (13/1000) 75% (75/100) 8% (88/1100) 38.2% (0.013 + 0.75)/2
Both Clinics 1.3% (13/1001) 75.2% (76/101) 8.1% (89/1101) 44.1% (0+1+0.013+0.75)/4
         

Notes. *We have this scenario often. We have not required them to have EBPsy or EBPharm in each month, but have required an encounter or EBPharm release.

Summary of Observations:

  1. The summed approach is consistently lower than than the averaged approach, inappropriately ignores any clinics who have 0 EBP in the numerator, and inappropriately ignores the monthly variation we need.
  2. For all downstream analyses, we will need to account for clustering of observations and in clinics over time.
  3. The denominator (clinic size/eligible patients) is [unsuprisingly] influential with small clinics that have 1 EBP/2 pages, etc. significantly inflating EBP reach, but appropriately capture a particularly clinics reach in any given month.
  4. Any discrepancies between apples/apples (e.g., EBPharm "Issue Date"/"Issue Date") will really inflate the issues above.

I've asked @staceypark and @ritahitching to schedule us for 10AM on Tuesday instead of the co-I meeting to work on this.

Have a wonderful weekend, everyone!!! 🍂 🍂

lzim commented 4 years ago

Good Morning ☕️ @holbrooa and @saveth

  1. I believe we’re schedule at 10AM tomorrow instead of co-I meeting to pin down remaining details.
  2. Ping me when the details requested on OSF are provided in terms of the rationale/justification for having either 1) issue date/release date, or 2) release date/release date.
    • On face, it makes sense to me as a reviewer, to have apples/apples or oranges/oranges. So, I need your rationale in the documentation for why you did issue/release in order to understand EBPharm as it is now, and “option C.”
  3. For Option B in CY2018 the divisor of 12 fits the 12 monthly observations for a given clinic.
    • So, if summing across 12 months, then dividing, to ignore monthly variance removes observations > 100% what does that say about the window of observation for EBPharm for patients engaged in care month-by-month?

FYI: @ritahitching and @staceypark

holbrooa commented 4 years ago

I'm working on adding more verbiage to osf. I don't see an appropriate place to add this particular nugget there, so I'll just add it here. @lzim the idea behind your first observation in your summary is really key:

"The summed approach is consistently lower than than the averaged approach, inappropriately ignores any clinics who have 0 EBP in the numerator, and inappropriately ignores the monthly variation we need."

...but you're wrong about that generalization and about which method is ignoring what. The summed approach is only lower because you picked the examples that way, with the biggest denominators lining up with the lowest percentages. See, when you average the percentages you throw away the information about the quantity, so 0/1 = 0/100 and 1/2 = 50/100 etc. For a quick counter-example, look at what happens if a clinic does 8/9 one month, and 0/1 the following month. Then you have 80% (summed) vs 44% (avg the pcts) This will be true for any example with different denominators in the different months. Maybe it's easier to see it by looking at the inverse: if a given clinic had the same denominator in every month, then the two methods are mathematically equal. The more the monthly denominators vary, the more the two methods will yield different results, with the summed version responding to the denominator variance. Like, to add on to that previous example, say the clinic has 0/991 in the second month instead of 0/1. The averaging the percentages method still says 44%, but now the summed version is 0.8% I guess the summed version looked to you like it was ignoring variation in the numerator, but it isn't. It's just also taking into account variation in the denominator, and your example denominators vary enough that that really affects the result.

Tl;dr: we're talking about weighted vs non-weighted averages, basically.

holbrooa commented 4 years ago

Oh, and if it isn't obvious, that throwing away of the quantity is where the >100% comes from. So image 2/1 in the first month and 1/3 in the second month.

saveth commented 4 years ago

@lzim @holbrooa One of the things that confuses me is that we don't time bound the numerator or have that individual counted towards the month when the individual was counted towards the denominator. Like the 2/1 example @holbrooa indicated. If the individual is not counted towards the numerator and denominator in the same month, then how can we interpret: of all folks in a given month, X received treatment. How is it possible that more people receive treatment relative to the diagnostic cohort that month?

I think this discussion about average of monthly EBP Reach or average of yearly EBP Reach is us trying to untangle the epidemiological concepts of whether we're measuring prevalence or incidence. If we're measuring prevalence, which is proportion of cases in a given period of time, then the one-year period is a very good estimates of the diagnostic population who actually had initiated treatment. This requires us not to just sum across the numerator and denominator, but to ensure that they are unique individuals in the numerator and denominator. If they are counted towards the denominator in June and August, but only once for initiating treatment in June, then we are underestimating the prevalence of initiation in the patient cohort for the given year.

However, if we are trying to estimate the incidence, the rate of occurrence of an event or the risk of getting treated, the monthly proportion would be a good estimates. If we are trying to convey to our audience the chances of getting treatment in clinic A in a given month, it's the mean of our monthly rate for that clinic that would indicate how successful they are able to treat their pool of eligible patients in a given month.

FYI: @ritahitching @staceypark @saveth

holbrooa commented 4 years ago

Also, as for which method "ignores" things: 0 in the numerator isn't ignored by either option. But 0 in the denominator is significant.

staceypark commented 4 years ago

Decisions from: https://meet.lucidmeetings.com/lucid/meeting/228871

  1. We will take empirical approach and review the data to what the impact of using Option B and and Cleaning up Option A code to remove 0's is.
  2. Anytime there is a 0 in the denominator or when there is 0/0, that clinic cannot be part of the divisor.
  3. We will present the results of Option A and Option B to co-I's for direction.

EBPharm defending Issue/Issue 

  1. Questionable whether just getting refills without seeing a "doc" is really meeting the critera for "EBPharm" (i.e., "evidence-based")
  2. Primarily because it's crisp/clean definition for the purposes of trial design (e.g., comparing two clinics in each arm on the same month).

EBPharm Course/Dose

Next Steps

saveth commented 4 years ago

@lzim @holbrooa The r function is working as I originally thought it would, it truly did remove the NAs from the calculations and only compute based on existing values. For instance:

denominator numerator proportion
NA NA NA
1 2 2
1 NA NA
NA 1 NA
3 2 0.6666667
1 NA NA
NA NA NA
1 1 1
NA NA NA

mean(data$proportion, na.rm = TRUE) # answer is 1.22

Based on today's discussion, the change I would need to make to my code for our data is to replace all NA values in the numerator where there is a denominator, so that the monthly proportion for that month is a true 0. Such that the example would look like:

denominator numerator proportion
NA NA NA
1 2 2
1 0 0
NA 1 NA
3 2 0.6666667
1 0 0
NA NA NA
1 1 1
NA NA NA

mean(data$proportion, na.rm = TRUE) # answer is 0.73

which will reduce the average proportion by including the true 0s.

FYI: @ritahitching @staceypark @saveth

saveth commented 4 years ago

@lzim @staceypark @ritahitching

  1. We will take empirical approach and review the data to what the impact of using Option B and and Cleaning up Option A code to remove 0's is.
  2. Anytime there is a 0 in the denominator or when there is 0/0, that clinic cannot be part of the divisor.
  3. We will present the results of Option A and Option B to co-I's for direction.

Are we expecting a second set of analysis similar to the one generated under current conditions (heatmap, boxplot, tree analysis, etc.)?

FYI: @holbrooa

saveth commented 4 years ago

Hey @holbrooa, we were talking today during lunch, and I'm given the impression that our denominators are counted as follow:

Denominator Omnibus EBP: anyone with a visit or a prescription EBPsy: we're counting people who have a visit with one of the relevant primary diagnosis EBPharm: we're only counting people who have a prescription issued

If this is the case, then it would lead to the problem where EBPharm Reach becomes 1issue/1issue = 100% in any instance when prescription issued occurs.

However, based on the OSF (https://osf.io/z7y8m/wiki/Data%20Definition/) the EBPharm denominator includes two types of people: 1) people who had a visit with the diagnosis or 2) receive a prescription. @holbrooa, can you clarify if our denominator definition in OSF is correct or the one @lzim discussed during lunch is our updated definition based on the meeting this morning?

FYI: @staceypark @ritahitching @saveth

holbrooa commented 4 years ago

The one in OSF is correct. Prescriptions are used as a supplement to visits in the denominator. Every denominator includes visits with a relevant diagnosis. We're changing how we're doing that supplementation, is all. Visits have always been the main source of our denominators, and they will continue to be.

holbrooa commented 4 years ago

As you correctly point out, it'd be pretty silly to stop using visits for the ebpharm denominator, because then our definitions for the initiate numerator and denominator would be the same.

You could look at it like this: Patients who get a visit with the relevant diagnosis make up the denominator. But, since a new prescription will get you into the numerator, and it is sometimes possible to get a script without a visit, we should go ahead and count you in both the numerator and the denominator, even with no visit.

staceypark commented 4 years ago

@holbrooa want to clarify that I'm reading the OSF definition correctly because there's a potential question that @ritahitching was going to raise... the denom counts 1) patients with dx who get visits with script, 2) patients with dx who get visit without the script, 3) patient who gets script without visit, 4) and anyone who already has a script (and we know they have it because of the issue date)

holbrooa commented 4 years ago

@saveth There's another version of the data table to look at. Only the BaseAndRx tab is different, because I didn't rerun the ebpsy stuff. Oh, and I updated the definitions, but not in osf yet, so I'll do that next.

@staceypark I don't really understand what you're saying. I think your number 4 is what we just decided not to include anymore in that last meeting, but I haven't updated the osf defs yet. Anyway, what's @ritahitching 's question?

ritahitching commented 4 years ago

@holbrooa thanks for offering to clarify

I was trying to understand how a given patient with more than 1 medication script (e.g. antidepressant and anti-anxiety medication) is counted in the denominator over time, as some prescriptions typically have a pharmacist release/issue date of 3 months before refill prescription is needed (e.g. 90 pills of supply of bupropion) and others 1 month supply (e.g.alprazolam) - unrelated to dosage prescribed.

FYI @staceypark @saveth @lzim

saveth commented 4 years ago

@holbrooa Thanks! @saveth I'm changing my code to include what we discussed yesterday (cross-ref #476 ):

I will need @lzim input next week to discuss how we should incorporate the new analysis into the co-I slide deck once we have a sense of what data looks like.

🤞 That this round of data validation will go well and our definitions and analysis will be finalized.

holbrooa commented 4 years ago

@ritahitching

That's a good question. Inscrutable wall-o-text incoming...

Thankfully, now that we're counting original prescriptions everywhere, the situation is much simpler than before. When a patient gets one of the prescriptions we're tracking, we count them in the relevant denominator in the month that the prescription was issued. If they receive multiple tracked prescriptions, we count them in multiple denominators, but there are separate columns of combined denominators where they won't be double-counted. Refills aren't counted for our denominators anymore. They're only used in the numerator now, where we aggregate on each prescription to see how many days' worth of medication a patient received within a 115 day window to see if they have a steady supply. So in terms of counting in the denominator over time, if they have to come back for a new prescription, then that means that the clinic's doctors are getting involved again and we re-count them in the month that the new prescription is issued. I think you are right about some drugs requiring more frequent involvement from the prescriber, via limits on numbers of refills or days of supply or whatever, but from the perspective of what we're trying to count as I understand it, I believe we're doing the correct thing now: when the prescriber gets involved again, count that patient again in the denominator. I.e. a patient coming in to the pharmacy for a routine refill without the prescriber's involvement isn't an opportunity for the clinic to do evidence-based care. But the prescriber writing a new prescription (even if it's for the same drug again) for that patient is such an opportunity. Have caution taking that logic to the bank though; I made this case months ago to use issue date in the denominator and Lindsey heard me out but went with release instead, so I may very well be misunderstanding something here.

Looking at your example, bupropion is a medication we're tracking for depression, but alprazolam isn't in our list. Any medications that we're not tracking are invisible to us - the patients we're tracking might be getting any number of other scripts, but we're just not pulling that information. But if a patient was issued a prescription for methadone in addition to bupropion in the same clinic in the same month (I have no idea if that's medically appropriate, but just exempli gratia) then they'll get counted once in each of a bunch of different denominator columns:

...understanding all of that is probably like the text version of hitting a home run to left field here in Boston.

ritahitching commented 4 years ago

@holbrooa thanks for the clarification - I believe it's a home run on my end.

staceypark commented 4 years ago

@holbrooa will get @saveth data for #615 by next Tues

saveth commented 4 years ago

@holbrooa We were looking at the comparator data this morning, and I need more clarification on some of the variables you've pulled so that I can better explain to @lzim the calculations based on these variables. The questions are posted on OSF: https://osf.io/z7y8m/wiki/Data%20Definition/

FYI: @saveth @staceypark @ritahitching

saveth commented 4 years ago

@holbrooa Have you had a chance to look at the questions on OSF? We are trying to figure out which of the columns to use for our sample size calculations.

FYI: @lzim @staceypark @ritahitching @saveth

holbrooa commented 4 years ago

Yeah, I just posted some answers. In general, the information is all already in the definitions of each column, but I'll work on a narrative description of the queries as part of my documentation these next couple weeks.

saveth commented 4 years ago

@holbrooa For each of the EBPsy diagnostic cohort, we include people with only a visit PTSD and Depression. Yet, for EBPsy AUD, we include people with prescription, and we use this for the denominator for EBPsy and EBPharm. Should we have two AUD diagnostic cohort? One with just the diagnosis for the EBPsy and one with the diagnosis + prescription for EBPharm?

FYI: @ritahitching @staceypark @lzim @saveth

staceypark commented 4 years ago

@holbrooa Making sure that I get your answer, on bullet three when you say "Once each in every denominator that includes one of the above" do you mean:

a) denominator includes any of the criteria: counted 1x regardless of how many criteria it suffices b) same scenario: counted 1x per criteria i.e. if denominator includes all 4 criteria then it would be counted 4x since it has all of it

seems like the answer would be A because of our "no double counting" principle. but the word "each" is throwing me off a little.

Are clinicians able to record multiple treatments on a visit? Can a patient go in for an EBPharm prescription and an EBPsy PTSD? If so, how are they counted towards OmniPsyCount, OmniPharmCount, TotalCohort, EBP Templates and DepPharm84Denom or DepPharm84Num?

So if a patient has a visit where the provider marks a diagnosis of PTSD and also a diagnosis of depression, and the patient gets some depression meds and gets evidence-based psychotherapy for their PTSD, then they are counted:

Once in the denominator for depression Once in the denominator for PTSD Once each in every denominator that includes one of the above Once in the EBPsy template numerator for whichever template the provider used Once in the depression meds numerator

staceypark commented 4 years ago

@saveth (cross ref #476) In-preparation for the Co-I meeting on 10/22:

  1. By COB Tuesday 10/15 detail all the changes you intend to make to the slidedeck
  2. By COB Thursday 10/17 ping @ritahitching and @staceypark with updated slidedeck. We will help review for clarity & typos before sending it to @lzim
  3. By COB Friday 10/18 send slidedeck to @lzim for final review to discuss and present to co-I's at Tuesday 10/22 meeting

Note: @lzim will be out most of next week for AHSR.

staceypark commented 4 years ago

cross-ref to @saveth handoff #615 @saveth has database for course going back to 2014