MoH Nepal erred reports vs. messages

daniellearon commented 5 years ago

Context

Currently, SMSes sent by FCHVs are divided into two webapp tabs:

Reports
- Valid reports — accepted
- Invalid (erred) reports — a report-specific error message is sent with an example of the correct syntax for the type of report that the FCHV was attempting to send, as well as other information pertaining to the specific error that was made
Messages
- All messages — a generic reply message is sent, indicating that we have received their SMS, but if they were trying to send a report, they should check the format and resend

“Invalid Report” - further detail If an FCHV sends an SMS such as [form code_space_non-existent ID number], or [form code_space_omission or mistake in required syntax] the SMS is categorized as an “invalid report” in the reports tab. The system recognizes which report the FCHV was attempting to send by the [form code_space], and replies with a report-specific error message, including an example of the correct syntax for that type of report, and any other information to help correct the error that was made ("99999 ID has not been registered, please confirm the ID and resend the form")

“Message” - further detail In the case, however, that the FCHV sends an SMS starting with [form code_no space], the system will categorize the report as a “message” in the messages tab. After receiving any type of “message”, the system sends a generic reply, indicating that if the FCHV was attempting to send a report, it was not recognized, and to check the format and try again. This response does not outline correct syntax or point to other specific errors, as it does not recognize which form the FCHV was attempting to send.

Feature Requirement

As an FCHV, when I make a mistake in my report, I want to be able to know both that it was a mistake, and how I can correct it, so that I can accurately report on my patient as well as improve my own skills using the mHealth tool.

As a Medic teammate, I want to be able to distinguish form codes followed by no space (a report error) from a free-form message that an FCHV sends through the mHealth tool (text communication) so that I can...

have an accurate count of, and can therefore analyze, error rates
have an accurate count of, and can therefore analyze, other indicators pertaining to engagement, performance, and timeliness
easily sort through SMSes intended as messages to understand what FCHVs are communicating via text (rather than through reports)

Unmet need or opportunity

Unmet need: in the webapp and analytics, we currently only capture and categorize certain types of SMS errors, as errors. Therefore, we cannot understand the full extent to which FCHVs are making errors in reports versus sending text messages to communicate. Missed opportunity: to help FCHVs to correct and learn from their minor mistakes.

Impact of this “pain” for developers, patients, providers, managers, etc.? Analytics data is not entirely accurate (an issue for Medic, as well as for managers and health coordinators as they gain access to external dashboards); this could misguide greater programmatic decisions. FCHVs may feel confused or be mislead by receiving clear feedback on certain types of errors, but not others; this could lead to future inaccuracies, missed reporting, and/or a feeling of discouragement.

Proposed solution

Describe how you would like to solve the “pain” that you identified:

Categorize any SMS starting with [form code_no space_number], as an “invalid report” rather than a “message”, generating the correct report-specific error message, thereafter
Categorize any SMS starting with several characters linked together as a "message", except for the following characters: जघद, जघ, जस, जश, जष, जसद, जशद, जषद, as these are potential report attempts for delivery (if an FCHV forgot to put a space between the form code and the delivery type)

OR

Categorize any SMS starting with [form code_no space_a character or number], as an “invalid report” rather than a “message”, generating the correct report-specific error message thereafter. This potential solution may, however, capture many SMSes intended to be "messages", as "invalid reports"

Examples of the future fix: A 25 Sarita (अ 25 सरिता) = valid report A Sarita 25 (अ सरिता 25) = invalid report, triggers error message P25 Sarita (अ25 सरिता) = invalid report, triggers form-specific error message (proposed change) JSD 35105 3 (जघद 35105 3) = invalid report, triggers form-sepcific error message (proposed change) Baby 25 Sarita (बच्चा 25 Sarita) = message Sarita is having a baby! = message

Example use case for this feature

All use cases in Nepal MoH deployments

Provide an example of the solution value in terms of the “pain” it will alleviate using a real-world application:

FCHV Rama is trying to register Sarita’s pregnancy. She makes an error in her form by forgetting to put the space between the form code and Sarita’s age. She receives a report-specific error message in reply, telling her that there was an error, and that if she wants to register a woman’s pregnancy, to please try again using this format: [अ age name]. Rama tries again, successfully registers Sarita’s pregnancy, and learns from her mistake.

Municipality Health Coordinator Mina has an accurate view of the FCHV error and engagement rates of each HF in her municipality, and can therefore make an informed decision about where to prioritize refresher trainings.

Service Designer Danielle has access to accurate data about the most frequently erred reports, in order to inform the redesign of form structures and syntax.

Links to supporting information

Nepal MoH Basic Messaging Deep-Dive - a look into SMSes received to understand what FCHVs are communicating via message, and how many “messages” are actually erred reports

cc: @ranjuts

garethbowen commented 5 years ago

@daniellearon Thanks for the report! In your examples above you have P25 Sarita and JSD 35105 3 and their Nepali equivalents. In those cases what is the desired form code? "P" and "J" respectively?

daniellearon commented 5 years ago

@garethbowen Apologies for the delay - you were so speedy that I missed your reply!

Correct, "P" and "J" for pregnancy registration and delivery confirmation, respectively. In the correct syntax, both form codes would have a space after them.

garethbowen commented 5 years ago

Ok. The only problem is we can't tell for sure which form was intended, for example, you could have one form code "J" and another "JS". In practice though I think it's unusual to have two letter form codes at all, let alone two that are so similar.

What we could do is:

check if the first word matches a form code (as we do now)
otherwise, check if the first word starts with a form code - if we find one (and only one) match then we can send a useful error (eg: "were you trying to submit a J report?").
otherwise, send a generic error message (as we do now)

Would that suit you?

Another option is when we get to item 2 we could assume it's a "J" form and autocorrect it, but I think that's dangerous if someone typoed the code. What do you think?

daniellearon commented 5 years ago

I am pretty confident that we won't design any new form codes that start with the same character as an existing form code, so that shouldn't be an issue.

To clarify - by 'one and only one match', you mean one and only one matching form code? If so, that would work given my comment above + the fact that all of our form codes are currently only 1 (unique) Nepali character.
So, the generic error message would only be sent if the first character of the incoming message did not match any of our form codes. Is that correct?

I agree that autocorrecting is dangerous and feel we should focus more on helping FCHVs correct their mistakes, rather than trying to guess what they might have meant.

One additional idea worth mentioning is, if can we identify common form errors (for example, FCHVs frequently sending "I" instead of "J") could we also send the report-specific error message for what we think was the intended report (in this case, if they sent "I", the error message for "J" would be sent)?

If all is as I understand it, this should work!

garethbowen commented 5 years ago

the generic error message would only be sent if the first character of the incoming message did not match any of our form codes. Is that correct?

Correct!

One additional idea worth mentioning is, if can we identify common form errors (for example, FCHVs frequently sending "I" instead of "J") could we also send the report-specific error message for what we think was the intended report (in this case, if they sent "I", the error message for "J" would be sent)?

I think this would be possible by making a dummy "I" form and setting up validation that always fails and auto-responds with your chosen error message.

daniellearon commented 5 years ago

Got it - thanks! We can look into the data here & see if we find any significant patterns among errors before settling on that aspect of the feature.

Beyond that, is there anything else you need from me at the moment?

garethbowen commented 5 years ago

That's it for now, thanks!

1yuv commented 4 years ago

Here is the link to some analysis on whether FCHVs respond after getting error message from system or not and how do they perform if they are getting specific response from the system on their error.

medic / cht-core