FU with ad test group

[lippytak]

• [x] Monitor/pause ads when we learn the learnings
• [x] Build textit.in FU
• [x] Build email/survey FU
• [x] Phone feedback survey script
• [x] QA both surveys and ZenDesk process
• [x] Do 10 day FU (who? how many?)
• [x] Qualitative phone interviews
• [ ] Do 30 day FU
• [ ] Writeup

BG: https://github.com/codeforamerica/clean/issues/479

Final stats will live here: https://docs.google.com/a/codeforamerica.org/spreadsheets/d/1vfjxik5oYBRnm4ssXVoR5H7aYwuNWLbUr2yqIRp8fZU/edit#gid=0

[alanjosephwilliams]

Some QA of Textitin showed positive results. At mid-day we had:

• 12 confirmed screening surveys initiated.
• 8 completed and approved.
• 0 errors.

There is some gray area about the agnostic way we handle income and household size responses, but one clear way we could improve the experience is by offering a response to out-of-syntax answers (i.e "I start in the fall" rather than "No" to the question, "Are you a full-time college student?") that acknowledges that a reply was received, rather than simply repeated the question.

[lippytak]

Could do FU with Zendesk + Surveymonkey integration: http://help.surveymonkey.com/articles/en_US/kb/How-do-I-collect-customer-satisfaction-responses-from-SurveyMonkey-with-Zendesk

[lippytak]

Hit $100 budget. Paused Bing. @alanjosephwilliams how are we looking for Facebook and Twitter?

[lippytak]

Twitter got marked as spam so we're calling Twitter :+1: for now.

[alanjosephwilliams]

As of AM on Thursday 4/16:

• 73 clicks on 3,592 which is ~= 2.03% CTR. Not too shabby. We've spenet ~$35 of 100.
• FB accounts for 4 of our 49 completed typeforms.

On Wed, Apr 15, 2015 at 10:47 PM, Jake Solomon notifications@github.com wrote:

Twitter got marked as spam so we're calling Twitter [image: :+1:] for now.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/clean/issues/480#issuecomment-93639620 .

t: @alanjosephwilli p: 817 713 6264

[lippytak]

Eeep, check if the main URL is loading on mobile.

[lippytak]

10day FU with 10 people to start

• Intro context 'making it better' / feedback
• Did you apply?
• Did you interview?
• (typeform only) Overall, how is the application process going? (easy > hard)
• (typeform only) What has been the most challenging part so far?
• $20 in Amazon credits for a 20 minute phone interview? We'll get in touch to schedule if we select you.

30day FU:

• Did you apply?
• Did you interview?
• Did you verify?
• Did you get approved?
• Did you get/use your EBT card yet?
• (typeform only) Overall, how is the application process going? (easy > hard)
• (typeform only) What has been the most challenging part so far?
• $20 in Amazon credits for a 20 minute phone interview? We'll get in touch to schedule if we select you.

[lippytak]

Textit.in draft done, would like to review/QA with someone else soon

[lippytak]

Waiting for DNS to propogate for our survey to work at feedback.getcalfresh.com

[lippytak]

Sent out:

• 5 email surveys to typeform respondents
• 5 SMS surveys to textit.in respondents

[lippytak]

Email template:

Subject: GetCalFresh feedback

Hello! This is Jacob from GetCalFresh.com. You started to apply for CalFresh (Food Stamps) last week and I just wanted to check in to see how things were going and to get your feedback. If you have a minute please take our short feedback survey to help us improve: www.getcalfresh.com/feedback Feel free to reply to this email with any questions. Thanks! Jacob

[lippytak]

Phone interview guide below. Goal is to get qualitative understanding of bottlenecks/pain points.

• Intro
• Purpose: Work at nonprofit that helps people get CalFresh. Talk to people to understand how to make the process better.
• How did you hear about CalFresh?
• Why did you decide to apply now?
• App: How did you apply? How did it go?
• Interview: Did you do the phone interview? How did it go? Probe for length, confusion,
• Verify:After interview, did you have to submit more documents? How did that go?
• Approved: Did your application get approved? How did you know?
• EBT Card: How did you get your EBT card? How did that part go? When/where did you use your card for the first time?
• If you could change one thing about the process what would it be?
• Anything else that's important for me to know?
• You have any questions for us?
• How can I send you $20 Amazon codes? Confirm email or SMS

[lippytak]

After ~20 hours:

• 3/5 SMS surveys responded, 2 applied, 1 did their interview, and 1 was open to a phone call
• 0/5 email surveys responded

Going to send another wave of ~20 this time.

[lippytak]

Sent batch 2 (10 emails, 2 SMS surveys)

[lippytak]

One user interview: https://github.com/codeforamerica/health/wiki/User-research#20150225-user-from-getcalfreshcom-ad-test

[lippytak]

Sent batch 3 to the remaining 25 textit.in respondents who opted in.

[lippytak]

Send batch 4 to the remaining 21 typeform respondents who opted into SMS FU.

[lippytak]

We have ~10 people who opted into FU phone calls. Let's see if we can reach some of them Monday to get quick follow up interviews.

[lippytak]

Some quick but interesting stats:

• We solicited 61 FUs via SMS
• Got 29 responses (48%)
• 18 (62% of respondents) had NOT finished the application (self report)
• 11 (38% of respondents) HAD finished the application

In the future we might want to run another test that aims to estimate the % of eligibles who successfully coplete the online application by themselves. We could also get some insight on this via usertesting.com or other sources.

[alanjosephwilliams]

Super interesting

On Mon, Apr 27, 2015 at 5:37 PM, Jake Solomon notifications@github.com wrote:

Some quick but interesting stats:

• We solicited 61 FUs via SMS
• Got 29 responses (48%)
• 18 (62% of respondents) had NOT finished the application (self report)
• 11 (38% of respondents) HAD finished the application

In the future we might want to run another test that aims to estimate the % of eligibles who successfully coplete the online application by themselves. We could also get some insight on this via usertesting.com or other sources.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/clean/issues/480#issuecomment-96858738 .

t: @alanjosephwilli p: 817 713 6264

[lippytak]

Just sent 10 remaining email FUs so hopefully we can get another couple phone interviews.

[lippytak]

Done until 30 day mark

[alanjosephwilliams]

great job leading and conducting the bulk of this effort.

Let's sync up about our other test FU this afternoon.

On Thu, Apr 30, 2015 at 10:03 AM, Jake Solomon notifications@github.com wrote:

Done until 30 day mark

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/clean/issues/480#issuecomment-97882379 .

t: @alanjosephwilli p: 817 713 6264

[lippytak]

This is now officially on pause until ~May 1 (30 days from app date).