Clinical Recommender Review/Process and Scoring (kind of duplicating other expert panel review issues)

[jonc101]

• [x] Second pass review by clinical experts to score orders/decisions not originally anticipated

Just means we need to be confident in the grading system when finalize results. Go back and review some cases manually to assess if weird unexpected behavior that needs to be manually corrected.

• [x] Collate results of process like mini-study / paper to confirm functional process

[JonathanChiang]

• [x] Generate Queries for Physician Behavior from DB

[JonathanChiang]

• [x] Scheduled Jason H. for Tuesday (had to reschedule)
• [x] Scheduled Andre for Wednesday

[JonathanChiang]

Meeting with Andre

1)

• [x] table orders 2)
• [x] consult with expert panel

Next Steps

• [x] modified spread sheet
• [x] scheduling conversations
• [x] blind physicians to each other's grading

meeting notes: 
- mix of 1 and 2? 
- 3 is the most time intensive 
- 4 classical test theory (multiple forms) (not recommended) 
- how complex? simple: (0, 1, -1) (cardio-version should be worth more than echo)
- partial grading model?  
- Angoff method versus Delphi 
- guidance? across panel (tier grading by treatment versus diagnostic?) 
- creating a scale 
- 0 to 10 scale and 0 to 10 confidence (with negative implications) 
- conditional grading (grouping) 
- add distractors to google sheets/ excel? may not matter with Delphi method conversations?
- per state grading 
- composite versus line by line (average) 
- inter-reliability score ? kappa, find consensus
- time flag (signals whether or not sim_state is important) 
- remove and screen for only physicians
- we have retroactive scores (but there might be prospective scores for new orders) 
- differences for type of doctor may change (our study focused on internal medicine initially) 
- instruction document for other physicians (Andre) Publish in supplementary
- evidence-based grading ?? structured + consistency 
- points for diagnosis (physician bias?) may not reflect guidelines
- funding for expert panel?? *steak??* fun? conference (research goals)
- tab (intermediary) and calibrate (iterative approach for inter-rater reliability) 
- zoom meeting or intermediate meeting (scheduling) email less valuable 

[JonathanChiang]

Delphi Method:

• blinding doctors to other grading:
• inter-reliability reading

What Is the Delphi Method? The Delphi method is a forecasting process framework based on the results of multiple rounds of questionnaires sent to a panel of experts. Several rounds of questionnaires are sent out to the group of experts, and the anonymous responses are aggregated and shared with the group after each round. The experts are allowed to adjust their answers in subsequent rounds, based on how they interpret the "group response" that has been provided to them. Since multiple rounds of questions are asked and the panel is told what the group thinks as a whole, the Delphi method seeks to reach the correct response through consensus.

Understanding the Delphi Method The Delphi method was originally conceived in the 1950s by Olaf Helmer and Norman Dalkey of the Rand Corporation. The name refers to the Oracle of Delphi, a priestess at a temple of Apollo in ancient Greece known for her prophecies. The Delphi method allows experts to work towards a mutual agreement by conducting a circulating series of questionnaires and releasing related feedback to further the discussion with each subsequent round. The experts' responses shift as rounds are completed based on the information brought forth by other experts participating in the analysis.

https://www.investopedia.com/terms/d/delphi-method.asp

[JonathanChiang]

• Jason/Andre/Lisa to convene first

• Atrial Fibrillation first go-through

• then reconvene

[JonathanChiang]

• [x] send HnP for each one

[JonathanChiang]

• [x] Updated Physician names to google responses

[JonathanChiang]

• [x] in house meeting: with all physicians:
• [ ] WIP physician review

[JonathanChiang]

Mini Methods:

We are consulting three expert physicians that have deep understanding of *hospital metrics and blah blah blah** . We are organizing several working meetings that incorporate a delphi method of consultation. A delphi method

The Delphi method is a forecasting process framework based on the results of multiple rounds of questionnaires sent to a panel of experts. Several rounds of questionnaires are sent out to the group of experts, and the anonymous responses are aggregated and shared with the group after each round. The experts are allowed to adjust their answers in subsequent rounds, based on how they interpret the "group response" that has been provided to them. Since multiple rounds of questions are asked and the panel is told what the group thinks as a whole, the Delphi method seeks to reach the correct response through consensus.

The clinical application of the delphi method is to create several rounds of grading of five different clinical cases designed for the "Clinical Recommender System" CRS. After each round of scoring the physicians will reconvene and discuss grading decisions.

Rating clinical decisions incorporates different models of clinical decision making. One must consider the clinical confidence of a treatment decision. For example given the initial diagnoses and HnP of a patient do the clinical orders match clinical guidelines.

[JonathanChiang]

Deescalating current priority, since development is contingent on physician feedback.