Trigger an SMS to be sent when multiple reports that meet certain conditions are sent in from the same community

[sglangevin]

The Community Based Disease Surveillance (CBDS) use case includes a variety of SMS alerts that go out to specific people in the health system who are responsible for monitoring and responding to any potential outbreaks. Most of these alerts are triggered by one report of a certain set of symptoms. In a few cases, the SMS needs to be triggered when a certain threshold of reports with a certain set of symptoms is reported. For example, trigger an SMS to be sent when a CHW reports 5 cases of non-bloody diarrhea or if 5 cases are reported by multiple CHWs in the same community.

A few other examples in this doc: https://docs.google.com/spreadsheets/d/10pi7EkKB73zqnR4eBarI88VbcSGfc2aZ77piHcL1HbM/edit#gid=0

@abbyad and I discussed this briefly and thought that a custom Sentinel transition might do the trick, but I am interested to see what other ways we could implement this.

cc @samkanga

[sglangevin]

I gathered some more information about this feature request after talking with @samkanga and @katanu:

• Each of these SMS messages needs to go to a specified group of people
• This group might change over time, so it needs to be possible to change the list of phone numbers the messages go to (this could be done in config)
• In general the group to send to will be based on the Parish or Link Facility that the CHP belongs to (this info is currently stored in the CHP Area doc but could be moved)
• Some messages will need to go to others higher up in the health system

This feature is required for a deployment in July.

[estellecomment]

@katanu @sglangevin @samkanga I think this also needs to have a time window, no? E.g. 3 death reports in the same area in the last month. (without a time window, this will be triggered eventually for all areas, once enough people are dead)

[sglangevin]

@estellecomment the time window is 1 week for this project. So if you get a death report today, you have to get 2 more death reports (total of 3) within a week from today from the same community.

[estellecomment]

If the reports to count are all from the same form, then we can use the existing alerts feature, with a really complicated condition :

// Function returning true if you should count this report towards the threshold.
var isAProblem = function(report) {
  return (report.area === FORM_CODE(0).area && // same CHW area
    report.symptoms === REALLY_BAD_SYMPTOMS); // whatever checks you want to run
};

// Function returning true if the report is within TIME_WINDOW of the current report.
var isWithinTimeWindow = function(report) {
  return report.submission_date > FORM_CODE(0).submission_date + TIME_WINDOW;
};

var problemReportsCounter = 0;
var i = 0;
var report;
do {
  report = FORM_CODE(i);
  if (!isWithinTimeWindow(report)) {
    return false; // break the loop, we’ve gone further back than the time window
  }

  if (isAProblem(report)) {
    problemReportsCounter++;
  }
  i++;
} while (report);

return problemReportsCounter >= THRESHOLD;

but if this is a common use case then we want something simpler to configure, and if the reports come from several different forms then we'll have to write custom code anyway.

In that case, we can add an app_settings field called thresholdAlerts or whatever, something like :

{
  isReportCounted : function(report, currentForm) {
    return report.area === currentForm.area && // from same area
      (report.form === FORM_CODE_1 &&
          report.somefield === REALLY_BAD_VALUE)  ||
      (report.form === FORM_CODE_2 &&
         report.someOtherfield === OTHER_BAD_VALUE);
  },
  timeWindowInDays : 7,
  numReportsThreshold : 3
}

and add a sentinel which does something similar to the current conditional_alerts.js (or reuse the same app_setting and transition with extra bells and whistles, whichever)

[estellecomment]

Update : the reports can be from several forms.

So I'm going for adding a new transition whose config will look something like this :

{
  timeWindowInDays : 7, // look through all reports from the last 7 days
  isReportCounted : function(report, lastReport) { 
    // report : a report from the last 7 days
    // lastReport : the report that triggered the transition
    return report.contact.parish === lastReport.contact.parish && // from same area
          (report.form === FORM_CODE_1 &&
             report.somefield === REALLY_BAD_VALUE)  ||
          (report.form === FORM_CODE_2 &&
             report.someOtherfield === OTHER_BAD_VALUE);
  },
  numReportsThreshold : 3, // if we count 3 reports, then trigger the alert
  message: ‘3 patients with big problem in 7 days, reported at {{countedReports[0].contact.parent.parish}}. Report by {{countedReports[0].contact.name}} ({{countedReports[0].contact.phone}}) for patient {{countedReports[0].patient_id}}. Report by {{countedReports[1].contact.name}} <...>’,
  messageRecipients: [ 
    ‘+254777888999’, // super-supervisor
    ‘countedReports[0].contact.parent.parent.contact.phone’ // contact for the district that the CHW area is in
  ]
}

Specifying message recipients by parish is trickier, this is not part of our standard hierarchy. I also don't know who in the parish we're targeting, can we have examples @sglangevin? Is there a parish contact person (and if so where is it specified)? Are we sending to everyone in the parish?

Inside, we'll

• query the data_records medic-client/reports_by_date view, indexed by reported_date, to fetch within the timeWindowInDays.
• filter reports with isReportCounted func. Stop at numReportsThreshold reports counted.
• Rehydrate the selected reports.
• message the recipients in messageRecipients with message text from message
• store the sent message text and recipients on the report (or on all the counted reports) for records

We might have to rehydrate the reports for the isReportCounted func too, depending on the func. That could be a lof of queries though. If we don't, then we have to specify for the people configuring that the reports in isReportCounted are not hydrated, while the ones in countedReports are, which is pretty awful UX.

[sglangevin]

@estellecomment I think we were still waiting to confirm that the parish is the community in which we need to count reports. @samkanga @katanu can you please confirm this?

@samkanga and @katanu can also give more information on who receives the alert. I believe it will be a group of people which might differ by community. @samkanga @katanu can you please help provide more info so we can get this feature built?

[samkanga]

@estellecomment we have done some side enquiries as we wait for confirmation from LG on the level for reports count. The village is the lowest administrative unit followed by a parish. On average, a parish has between 100 to 150 families. In terms of who receives the alert, we presently have identified the surveillance person at the health facility (serves a parish), while at higher levels we have the district health office and the national level (EoC) who would also ideally need to receive the alert. The LG branch managers also receive the alerts.

[sglangevin]

@samkanga we don't currently track the village that each CHP covers. If we are going to count these forms by village, then we would need to add a field (village) for every CHP area. Otherwise we should start with counting by parish, since that information is already there. Does each CHP cover a particular village?

[estellecomment]

@samkanga @sglangevin The info I need to implement is : for each of these people that need to me messaged, where is the person stored in our docs? Given a report, how do you get to the message recipient? e.g. report.contact.parent.parishContact for instance.

If they can't be specified that way, then we might have a problem.

[samkanga]

@sglangevin we don't need to add the village field since CHP areas may span more than one village. It is in order to conduct the count by parish. @estellecomment am not very clear on the organization of our docs. However, a recipient need not exist as a user on the system to receive an alert. The plan is to have a capability to capture at least the cell numbers of the recipients so that a message can be sent to those contacts once generated.

[sglangevin]

@estellecomment I talked with @samkanga and here is what we agreed on:

There are about 10 people who need to receive SMS alerts for a particular branch. Each branch contains multiple parishes but only 1 link facility (first level of MOH hierarchy who receive alerts). In addition to the link facility people, there are district-level and national-level contacts who receive the alerts, as well as LG branch managers. The district-level and national-level contacts technically are above the branch level in the hierarchy, but we don't have levels in our system for those people.

So one option would be to create all 10 contacts at each branch and send to all of them. The downside of that is some of the 10 will be the same across multiple branches and we aren't allowed to have multiple contacts in the system with the same phone number. Another option might be to have a way of specifying a list of phone numbers for each branch without actually creating contacts. As long as we can find a way to specify the list of contacts for each branch and send to them, we're good, so maybe you have a better idea?

And to be clear, we are still going to count reports within a particular parish, so we should trigger these messages based on meeting the conditions within a specific parish.

[estellecomment]

we are still going to count reports within a particular parish

Yup, that's fine : report.contact.parent.parish is the parish name (right?), so you can compare the parish name for two reports.

a way of specifying a list of phone numbers for each branch without actually creating contacts

We don't actually need to have a person doc in the system for the message recipients. We can

• hard-code the number in the app_settings (we'll have to edit app_settings if they change numbers, hopefully that's ok). That's appropriate for numbers that should get the alert regardless of the parish it comes from.
• add a field in the branch doc, for instance alertRecipients : [ '+254777888999', '+254111222333'], and specify it in app_settings as report.contact.parent.parent.alertRecipients. That's appropriate for the numbers that should get the alert if it comes from a parish within the district. (and if we need different numbers for different types for alert, use alertRecipientsCholera, alertRecipientsAlienAttack, etc.) If these numbers change, we just need to edit the branch doc accordingly.

Does that cover all cases? Anything I'm missing?

[sglangevin]

Cool, yeah, that sounds like it will cover all bases. I think it would be fine to put all of the contacts (branch, district and national) in the branch doc alertRecipients field. Or if you want to make it possible to put national contacts directly in app_settings, that would also be fine, and we could put just the branch and district contacts in the branch doc alertRecipients field.

[garethbowen]

@estellecomment I haven't made any further progress on this, just cleaned up what you had and got the tests passing. Back to you.

[estellecomment]

Cleaned up and ready for review!

Code in sentinel : https://github.com/medic/medic-sentinel/pull/138 Documentation in webapp : https://github.com/medic/medic-webapp/pull/3609/ More documentation in medic-docs : https://github.com/medic/medic-docs/pull/26

(My script to assign random reviewers : return '@garethbowen')

[estellecomment]

Still todo :

• [x] Weird shard error happening when querying the lineage view : No DB shards could be opened. Reproduce.
• [x] Newlines in SMS messages are disappearing. Is that a feature or a bug?
• [x] Should the time window be from now or from date of latest report? (discussing offline) -> from latest report, nothing to change
• [x] getReportsWithinTimeWindow queries reports_by_date within time window. On big instances, and/or with long time window, is that too many reports and will it cause problems?

[estellecomment]

No DB shards could be opened error was because I was querying the view in parallel, not in series, and if there's more than 25 queries in parallel (from testing by hand, could be approx), it throws that error.

[estellecomment]

Newlines are a CSS issue, filing separately : https://github.com/medic/medic-webapp/issues/3612

[garethbowen]

Back to you @estellecomment

[ngaruko]

Any clues on how to AT this @sglangevin @estellecomment ?

[abbyad]

I have already been acceptance testing this as part of #3715. Will share a config to help you get started with this feature.

[abbyad]

I see this working with the following config:

  "multi_report_alerts": [
    {
      "name": "test",
      "num_reports_threshold": 2,
      "time_window_in_days": 7,
      "is_report_counted": "function(report, latestReport) { return latestReport.contact.parent.parent._id === report.contact.parent.parent._id; }",
      "message": "[nr.0._id                                 {{new_reports.0._id}}] [nr.0.c._id                         {{new_reports.0.contact._id}}][nr.0.c.name                        {{new_reports.0.contact.name}}][nr.0.c.parent._id                  {{new_reports.0.contact.parent._id}}][nr.0.c.parent.name                 {{new_reports.0.contact.parent.name}}][nr.0.c.parent.parent._id           {{new_reports.0.contact.parent.parent._id}}][nr.0.c.parent.parent.name          {{new_reports.0.contact.parent.parent.name}}][nr.0.c.parent.parent.parent._id    {{new_reports.0.contact.parent.parent.parent._id}}][nr.0.c.parent.parent.parent.name   {{new_reports.0.contact.parent.parent.parent.name}}][nr.0.c.parent.parent.parent.contact._id    {{new_reports.0.contact.parent.parent.parent.contact._id}}][nr.0.c.parent.parent.parent.contact.phone   {{new_reports.0.contact.parent.parent.parent.contact.phone}}] ----- [nr.1._id                                 {{new_reports.1._id}}] [nr.1.c._id                         {{new_reports.1.contact._id}}][nr.1.c.name                        {{new_reports.1.contact.name}}][nr.1.c.parent._id                  {{new_reports.1.contact.parent._id}}][nr.1.c.parent.name                 {{new_reports.1.contact.parent.name}}][nr.1.c.parent.parent._id           {{new_reports.1.contact.parent.parent._id}}][nr.1.c.parent.parent.name          {{new_reports.1.contact.parent.parent.name}}][nr.1.c.parent.parent.parent._id    {{new_reports.1.contact.parent.parent.parent._id}}][nr.1.c.parent.parent.parent.name   {{new_reports.1.contact.parent.parent.parent.name}}][nr.1.c.parent.parent.parent.contact._id    {{new_reports.1.contact.parent.parent.parent.contact._id}}][nr.1.c.parent.parent.parent.contact.phone   {{new_reports.1.contact.parent.parent.parent.contact.phone}}] Report by {{new_reports.0.contact.name}} ({{new_reports.0.contact.phone}}) for patient {{new_reports.0.fields.patient_id}}",
      "recipients": [
        "+123456789",
        "new_report.contact.phone",
        "new_report.contact.parent.parent.contact.phone"
      ],
      "forms": [
        "D",
        "delivery"
      ]
    },
    {
      "name": "flag",
      "num_reports_threshold": 3,
      "time_window_in_days": 7,
      "is_report_counted": "function(report, latestReport) { return latestReport.contact.parent.parent._id === report.contact.parent.parent._id; }",
      "message": "{{num_counted_reports}} patients with {{alert_name}} reports in {{new_reports.0.contact.parent.parent.name}} over the last {{time_window_in_days}} days. New reports from {{#new_reports}}{{contact.parent.name}}, {{/new_reports}}since the last alert.",
      "recipients": [
        "+14169876543",
        "new_report.contact.phone",
        "new_report.contact.parent.parent.contact.phone"
      ],
      "forms": [
        "F"
      ]
    }
  ]

[sglangevin]

Thanks @abbyad, this is great!

@estellecomment: For the real project that is using this feature, it's not just the form submission that matters. It's forms that meet certain conditions, so for example, an assessment form has a field blood_in_diarrhea = 'true'. Where/how are we able to specify such conditions? Can we do that in the forms array?

[abbyad]

That would be done in the is_report_counted function.

For example:

      "is_report_counted": "function(report, latestReport) { return latestReport.contact.parent.parent._id === report.contact.parent.parent._id && latestReport.fields.blood_in_diarrhea == true; }",

[sglangevin]

Ok cool. Thanks! I think we can move this to Ready, unless we wanted to create some config for the project that requested this feature? I'd also be happy to assign that to the tech lead who can go from the config posted here.

[abbyad]

Ya, from what I saw in the request this feature is ready enough to meet the needs of the project. We can open new issue if we learn of anything more needed while configuring the project.

[sglangevin]

Great! Moving to ready.

[estellecomment]

:)

this feature is ready enough to meet the needs of the project

Applying the 4.63 - Ready Enough tag

Thanks for the review and troubleshooting help!