GCF vs. MBCW vs. C4 Evaluation

alanjosephwilliams commented 8 years ago

Recap of thinking about this project to date, piped over from private repos:

Originally spec'd out here

https://docs.google.com/document/d/1EQuk9JVVFQ1df6Ui_GybsnV-E44BK7Y7K22k8QHCGOs/edit#heading=h.wqr0jsxjwkip

Quicker turnaround scope originally intended to be delivered before Reid pitch

By 22nd we can might be able to get a good baseline & comparison data for:

Conversion rate
Customer satisfaction of the online application
If they know the next steps
+ Side by side videos of people using (a la My Benefits CalWIN in 90 seconds) if helpful

Roughly here's how we would do it:

Adwords in SD > recruiting form > BCW or GCF > 4 hour email FU survey
I think we can set this up with google forms + zapier + getdrip

Open Qs

What incentive, how much, how to send? $25 visa for 4 hour FU and another for 30 day?
Do we want to include anymore questions in the 4 hour FU? Trust in gov? Expectations of gov service? Why they're applying?
Can we get enough people now? Right now we're only getting ~2 SD apps per day, so optimistically we'd get 1 participant per day which isn't enough for the 22nd.

Flow gcf-eval

High Level Summary of Activities & Costs

Summary

Approach = run adwords > opt into study > randomly assign to BCW or GCF > email survey (likely with google forms, zapier, getdrip)
Via an email survey we could collect comparison on:
- Application conversion rate
- Customer sat
- If they know the next steps
- Anything else? Approval rate at 30 days (if so we can only do SD)?
Approx $2500 in adwords and $1500 in incentives
BIG open question is timing...if we crank up adwords in CalWIN counties how fast can we recruit ~100 participants? Guessing we can get ~10-20 apps/day > ~5-10 study participants/day (same assumptions as above) > 10-20 days total

Cost

$1.50 adwords cost per click
$5 per calfresh application
$20 per study participant assuming 25% choose to participate (big open Q)
$40 per survey response, assuming 50% response rate (big open Q)
So $40 * 60 = $2400 in adwords
Maybe...$25 incentive per survey response? That would be $1500 for incentives. Amazon codes would be easiest, cheapest, but may bias sample to more tech savvy? Alt is visa gift cards in the mail, which we can send out via online vendor.

alanjosephwilliams commented 8 years ago

Relevant data from past studies and current analytics

I'm going to recap what we know already, from our own data on GCF and Caroline's research on MBCW, as we prep to share with others in the org and scope the next tranche of work.

First up, GCF data: (All of Jan/Feb 2016, excluding Feb 29)

Step	Users	Data Source
visitors to `/apply` from `/county`	1.3k (66%)	GA
visitors that drop off at `/county`	677 (34%)	GA
typeform sessions initiated (saws-driver IDs created)	1018	saws-driver
typeform apps submitted (dupes removed)	449	GCF-cases
typeform conversion rate	44.1%	saws-driver/GCF-cases

MBCW data, from Caroline's research: https://docs.google.com/presentation/d/1A5l1x0QiahDJlov3qHyEb_uP2NszdGnRTwud_ueOlrA/edit (Summer 2015)

Population is defined as those that responded to a survey request and follow up interview.

Step	Users
Shown screener	2,222
Screener responses	111
Phone Interview (1 hr to 2 days	30
30-Day Follow Ups	15

40% conversion rate among the interviewed respondents. (conversion rate)
20% approval rate among interviewed respondents. (approval rate)
1 of 3 approvals did not require an office visit at any time. (office caseload)
42% still had not been approved or denied 30 days later (processing time)
25 minute average time to apply among interviewed respondents. (time to apply)