Balance User Testing v1

[lippytak]

The plan: Text some folks and ask them nicely to give us a ring if they're in the mood to give ~5 mins of feedback. We'll update this issue with methods once we finalize.

Script: https://docs.google.com/a/codeforamerica.org/document/d/1YyQLlWMbuYEeF-Qem1VcXODjtUShF24NDkwWBw-aOig/edit?usp=sharing

Let's try using the comments below to summarize notes. We'll close this issue once we're done discussing/processing them into issues.

[lippytak]

v1 plan for the group to consider:

1. Filter top ~20 English speaking users from past week
2. Text them a solicitation from their current balance phone #: Hi! We are gathering feedback today to make this service better. If you have a few mins, please give us a quick call at (###) ###-####. Thanks and have a great day! We can include different phone numbers for different teammates who want to participate in testing.
3. When you get an incoming call, use this script: https://docs.google.com/a/codeforamerica.org/document/d/1YyQLlWMbuYEeF-Qem1VcXODjtUShF24NDkwWBw-aOig/edit
4. Jot down anonymous notes in a comment here.

We can start with a small number of outbound texts to see approximate call rates before we increase volume.

[daguar]

:+1: doooo ittttt

[fureigh]

Awesome. Added some suggestions in the Google Doc. Go go go! :)

Another approach to try: do it as a sequence of six texts, preceded by: "Hi! We are gathering feedback today to make this service better. Can we ask you six quick questions?"

Upside: removes the "oh no, I have to make a phone call and talk to people" hurdle, removes the need to transcribe replies, more accessible to users who might be using Balance for specific reasons of convenience (i.e., people with hearing impairments), might fit better into users' schedules. Downside: replies might taper off. Worth a shot, though!

[lippytak]

Awesome. I made a few edits given your feedback above. I just send this outbound msg to 5 users.

@daguar @fureigh @ajw @RebeccaCoelius anyone want me to send a few messages with their number today? Let me know. I'll report back either way and if we don't hear anything today we can try another approach tomorrow AM.

[lippytak]

Report back: 0 calls on this.

@daguar - maybe we could change the CA English balance responses today to solicit some feedback via SMS? Ask a few questions and then ask them if it’s okay to call? I could setup a brief survey this morning if you’re down.

[daguar]

@lippytak Yeah! That sounds great. It's a pretty small task on my end to add that to responses, so why don't you go ahead and set up that survey and:

1. Tell me what phone # to direct people to
2. Tell me where you'd like the prompt to be (eg, only after successful balance response? any other time in user flow?)
3. Provide the copy if you have an opinion (eg, "We would love your feedback on this service — please send a text to XYZ")

[alanjosephwilliams]

I think its time to ship this. I think we will learn quite a bit from getting it out there, and we can iterate on our process from here.

[lippytak]

@daguar good to go:

1. (415) 877-4154
2. I actually think the preferred point would be while they're waiting for their balance. They have a minute to kill anyway, right? And they haven't been completely biased by a success/error response by then. So this would involve editing the thanks_please_wait for English.
3. Here's my suggestion but feel free to edit:

Thanks! Please wait 1-2 minutes while we check your EBT balance. Give us some feedback while you wait! Please text START to (415) 877-4154 for a quick survey.

Ready to go live when you are. Thx! I'll review responses daily for a few days and we can decide if/when to turn it off.

[fureigh]

Awesome. If offering a gift card, let's try doing that up front (rather than as a surprise thanks at the end) so the incentive can encourage people to take the survey.

[lippytak]

COUNTDOWN TO LAUNCH…….3

[lippytak]

2

[lippytak]

1

[lippytak]

……...

[lippytak]

@daguar you ok??

[lippytak]

WE’RE GREEN. I REPEAT. WE. ARE. GREEN.

@daguar BLASTOFFFFFF!!!!!!?!?!?!?!?!?!?!?!?!?!?!?!?!?!?!!!!!!

Pushed to prod. Survey is live on every English balance response. No incentive for v1. I’ll DM y’all credentials to the textit.in survey.

[alanjosephwilliams]

v2 results as of Friday, let's report them here (feel free to edit through COB):

Total number of responses: 11 Total number of unique phone numbers shown feedback prompt: 120 Response rate: 9.2%

We're planning to ship v3, which features a $5 Amazon credit incentive, ASAP. We'll be delivering those semi-maually. We'll be tracking the purchase, delivery and redemption of those codes here: https://docs.google.com/a/codeforamerica.org/spreadsheets/d/1-5aPSlnLY4PkEotcQyxHkVuzlaNG1gXUa8vXToT1Op8/edit#gid=0

[daguar]

Updated with denominator to find (holy shit):

Total number of responses: 11 Total number of unique phone numbers shown feedback prompt: 120 Response rate: 9.2%

9 percent response rate...

[alanjosephwilliams]

New copy for initial solicit (160 char limit): Thanks! Please wait 1-2 min while we check your EBT. Want to help make this better & earn $5 in Amazon credit? Text (415) 877-4154 & answer 3 quick questions.

Old Copy for reference: "Thanks! Please wait 1-2 minutes while we check your EBT balance. Want to help us make this better? Send a text to (415) 877-4154 and answer 3 quick questions.

[daguar]

After wordsmithing on Slack for character limit:

"Thanks! Please wait 1-2 min while we check your balance. Want to help make this better & earn $5 in Amazon credit? Text 415-877-4154 & answer 3 quick questions."

[alanjosephwilliams]

@daguar let's turn off our solicitation as soon as possible. We are currently oversubscribed, so let's cool it while we regroup and consider what segment to survey next.

[daguar]

I've added #195 to do this. PR #196 implements — pending CI green.

[daguar]

The solicitation has been removed from production.

[alanjosephwilliams]

@daguar @lippytak @RebeccaCoelius it would be awesome for each of you to take on 45 minutes to an hour of user interviews this week.

Since the text message surveys were conducts a little while ago now, I'd like to send an text message thanking each person who said "yes" to additional interview questions again, and letting them know that we will be calling soon. To that end, let's pick a day (either tomorrow, wednesday or thursday) to begin the calls.

In the tracking document below, you can find who has yet to be called and who can be called again. In #health-testing you can find a pattern for documenting your interviews.

I also welcome feedback on the interview scripts, which I evolved a bit during the course of the interviews I conducted last week. Here is what I recommend as a starting point. If you have suggestions, let's document them here and get it consistent across all of our interviews. The script is useful mainly for SNAP recipients—we did interview one person who received only GA. I welcome thoughts on good questions for GA only folks.

Script

Hi this is [ Alan ] calling from the EBT balance check service—thanks again for responding to our survey last week. Do you have a few minutes for a few more questions?

• Which benefits do you receive? SNAP, General Assistance, or both?
• Do you use your benefits to support just yourself, or others as well?
• How do you typically travel to the store to use your benefits?
• How did you learn about this balance check service?
• What kind of phone or device do you use the service with?
• Thanks so much for your time. Do you have any other feedback or advice for improving the service?
• Have a great day!

Tracking document: https://docs.google.com/a/codeforamerica.org/spreadsheets/d/1-5aPSlnLY4PkEotcQyxHkVuzlaNG1gXUa8vXToT1Op8/edit#gid=0

19 responses. 11 agreed to follow up questions via phone. 6 have been called. 3 have been reached.

[lippytak]

Few ideas to consider:

• Say we work for a nonprofit or otherwise make clear we're not with HSA.
• When/why do you check your EBT balance? They already answered via the survey but I wonder what we would change in an open ended context.
• Maybe drop the second question per your finding that people were uncomfortable with it.
• Ask if they have a cash balance > if yes ask if they ever get charged to use ATMs.
• Ask if they've ever overspent and what happened/what it was like.
• Ask if they have questions for us.

[alanjosephwilliams]

Agreed on many of the points above. I do think there was a natural plateau in the number of appropriate questions for such an interruption—5 or so felt right. Here is an iteration of the script that culls from your suggestions.

• Slipped in a statement that we are a non-profit and that we won't share responses with the government, which might help put folks at ease. We may need to refine that language to be above board.
• Did not include an open ended question about context of use. I think we have a good understanding of that through data collected so far, and we have a need to keep the script concise.
• I've removed the family vs. individual question here—I thought the new language was sufficient to avoid discomfort, but ultimately it's not clear what value that segmentation would provide us right now.
• Cash balance +1
• The "Have you ever overspent" question is really compelling. It feels like an adjacent thrust of research, but let's include it in a few interviews and see how it goes.

I'd love for @cydharrell and @daguar in particular to chime in on this—hopefully its clear what we are trying to learn from each question, but a more critical perspective is welcome before we move forward.

Script

Hi this is [ Alan ] calling from the EBT balance check service—thanks again for responding to our survey last week. Do you have a few minutes to answer some additional questions?

• Great, just to let you know, this is a non-profit service. We're not affiliated with the government, and we keep your response anonymous. They will just be used to help us improve. First question: Which benefits do you receive? SNAP, General Assistance, or both?
• Only if they receive cash assistance: Do you ever pay fees when you withdraw your GA funds from an ATM?
• How do you typically travel to the store to use your benefits?
• How did you learn about this balance check service?
• What kind of phone or device do you use the service with?
• Thanks so much for your time. Do you have any questions for us, or other thoughts about how we can improve the service?
• Have a great day!

[lippytak]

Notes from my first phone call with a user: https://github.com/codeforamerica/balance/issues/206

[alanjosephwilliams]

This issue is being closed in favor of more granular issues, as our user testing efforts are growing beyond Balance. Moreover, this particular effort has largely drawn to a close.

Please see the Health meta-repo's issues with the 'user testing' label as well as #214 for outstanding matters related to user testing.