Calculation of baseline risk scores in the app

garethbowen commented 5 years ago

Currently, families in the pilot area have been manually assigned a baseline risk score which was calculated offline using R. For the pilot it has been assumed that this risk score will not change, though in reality it is based on several things which may change periodically (equity indicators, family size, etc.). Newly registered individuals are assigned an average risk score for any applicable risks and any household/community level factors

This approach poses 2 challenges:

Baseline risk scores are currently static but in reality the underlying factors can change over time, and baseline risk score should be recalculated. The frequency of this recalculation does not need to be high as it is assumed that these variables will not change frequently.
New individuals are assumed to have an average baseline risk, so important family context is lost for these families.

What we want:

Periodic calculation of baseline risk scores: The system should be able to calculate key risk factors for individuals on a periodic basis (monthly?) and update the baseline risk score stored in contact documents on the health workers’ phones.
Calculation of baseline risk scores for newly registered patients: The system should be able to calculate key risk factors for individuals after registration and update the baseline risk score stored in contact documents on the health workers’ phones.

Details of the calculation.

Original issue

MaxDiz commented 5 years ago

Hi @ecsalomon you completed a deep analysis that speaks to errors noted from initial trial. Does this modify needs and/or urgency related to this ticket for the next phase of the trial? Also related to the ability to record risk scores over time #5708

analysis: https://docs.google.com/document/d/1OOHeGdyLhFOrVT6er8JVFUdBwPbVKz5FJLBT90xW2VU/edit#heading=h.vw4sipkqxu4e

ecsalomon commented 5 years ago

I think I might like to reframe this issue a bit as there are some other considerations that have arisen from the pilot:

The major blocker to calculating baseline/full risk scores in app is the inability to access family- or CHV-level data from the patient profile; if we can do that, we do not need to store baseline scores at all as the entire risk score would be calculable.
@benkags and I have created a workflow for updating the baseline scores with pushes from the server and storing the history of baseline scores in a table. The process is still manual for now, but it's not labor-intensive, and the code is in a satisfactory state to continue this process for the remainder of the trial.
There are actually a few reasons why we may want the ability to store and update full risk scores for patients:
1. At the last design visit, CHVs expressed frustration that the introduction of PA had substantially slowed the app. Calculating the scores for all patients over short time spans is unnecessary, and we could speed up the app with some reasonable minimum refresh time for scores.
2. As we move towards evaluation of the models, one of the lingering questions is how much do patient risks change over time? For some risks, we may actually be able to keep risk scores for longer time frames (e.g., a month) without needing to recalculate, so the ability to control how frequently they are refreshed would be useful.
3. At the last design visit, CHVs also expressed frustration that, for example, an action they take could increase a patient's score during the task window, and a task would suddenly appear and already be overdue (because of fixed task windows). They didn't mention this happening, but it's also possible for risk scores to decrease during the task window and the task to disappear before the CHV has a chance to complete it. It's probably worth thinking about what custom task windows created from risk scores would look like from an implementation perspective, but one solution to this issue is to generate the risk score at the beginning of the task window and preserve it until that window ends.

MaxDiz commented 5 years ago

Thanks @ecsalomon this is helpful additional context.

@garethbowen it sounds like #5708 is not needed for trial purposes based on @ecsalomon @benkags work around. Do you agree?

...have created a workflow for updating the baseline scores with pushes from the server and storing the history of baseline scores in a table. The process is still manual for now, but it's not labor-intensive, and the code is in a satisfactory state to continue this process for the remainder of the trial.

MaxDiz commented 5 years ago

Also noting overlap with #5776

At the last design visit, CHVs also expressed frustration that, for example, an action they take could increase a patient's score during the task window, and a task would suddenly appear and already be overdue (because of fixed task windows). They didn't mention this happening, but it's also possible for risk scores to decrease during the task window and the task to disappear before the CHV has a chance to complete it. It's probably worth thinking about what custom task windows created from risk scores would look like from an implementation perspective, but one solution to this issue is to generate the risk score at the beginning of the task window and preserve it until that window ends.

garethbowen commented 5 years ago

@MaxDiz I think #5708 was designed as a nice to have in order to remove the need for the manual process and make PA ready for use outside of trials.

ecsalomon commented 5 years ago

ecsalomon commented 5 years ago

@MaxDiz Some additional context on the tasks appearing already overdue: In particular for this implementation, the CHVs receive incentives for tasks only if they are completed before the due date. A task appearing already overdue likely feels quite unfair (even if they can still complete it) since they never even had a chance to do it on time. (It's also probably undesirable from a UX perspective in any context, but the incentive structure makes it even more so.)

ecsalomon commented 5 years ago

From @mmureithi's design report:

CHVs are displeased by this occurrence, “Sometimes I take my phone to charge and when I pick it, I see a PA task indicating that its over due...Sometimes the tasks appear late in the evening and am left wondering when I will complete them and if I will qualify for incentives… Some tasks are generated immediately after an assessment is done” as expressed by the participants. It is evident that CHVs like to plan ahead and get infuriated by due or past due tasks appear from nowhere

https://docs.google.com/document/d/1HFBS7NzVCFhpmhz4PL5JGrNFSe2qnZIRNC1MbJ4Rnvc/edit?ts=5d70e3d7

ecsalomon commented 4 years ago

Revisiting this, I see two major issues that could be considered under this title:

The ability to access documents from multiple levels in the hierarchy to prevent the need to push "baseline" scores from the server
The ability to calculate, store, and occasionally update components of risk scores (e.g., relatively static and/or computationally intensive features)

yembrick commented 4 years ago

Lets re-scope this issue to be just #1 above from @ecsalomon comment

The ability to access documents from multiple levels in the hierarchy to prevent the need to push "baseline" scores from the server

And consider 2) The ability to calculate, store, and occasionally update components of risk scores (e.g., relatively static and/or computationally intensive features) as component of issue #4700

ecsalomon commented 4 years ago

Digging through our PA documentation, I found this document that summarizes many of the use cases and considerations relevant for this request.

ecsalomon commented 4 years ago

medic / cht-core

Calculation of baseline risk scores in the app #5706