alsomanya
commented
7 years ago Stepping back a bit, the overall theme with this email (and other features right now) is "Awareness" — we believe that victims aren't aware of the information they need to effectively get through the process. To that end, we're building features that increase access to information: a digital receipt of their rights, a court date notification tool, connections to resources that can help (per yesterday's meeting), etc.
To measure success for this email, I'd look at signals that indicate that a victim got more information (or the right information) than they had previously. I like your first sentence, though I might modify it to only include the specific thing we're testing — "We believe that... a digital receipt of victim rights makes people feel more competent about their role in the criminal justice system." And then for testing, I'd think about the following:
- Open rates and CTR are good indicators for an email's value, but I hesitate that this email will perform better than industry marketing emails by orders of magnitude. I'm also unclear which industry standard we would compare it to: government emails (is there a standard)? Digital receipts from retail stores? Let's look into whether there's a baseline that feels appropriate as a comparison; if not, I'd target the average-iest numbers. Something like a 10-20% open rate and a 5-10% CTR feels normal here. I might even go lower, because it sounds like we don't know what % of the email list is "legitimate", i.e. victims gave the DA email addresses they check frequently. If typos happened between the transfer of the paper form to the DA's CMS, or victims just don't have regular access to check their email, those could significantly impact your metrics (sidenote: as a secondary metric, I'd be very curious about deliverability %).
- For qual indicators, I like the path you're on with getting feedback from users directly. Is there a way to do this in the email itself? For example, Square receipts ask you "how was your experience?" and users can click 😀 or 😕. Could you do something similar to measure awareness? "How helpful was this email?" with good/neutral/negative buttons? Only a self-selecting group will complete the survey so I wouldn't use this metric as the only indicator, but it will at least give you a baseline for future iterations.
^example of a receipt from Square
One final thought: based on the screenshot we looked at in yesterday's meeting, I'd be interested in thinking deeper about ways this email can deliver more "awareness". We all agreed that the language of the rights themselves is complicated and filled with legalese — could we potentially "translate" the rights in this email to plain human speak? i.e. "Here are the rights you selected: [x,y,z]. Here's what that means in English: [a, b, c]" Could we refer a victim to free legal services (other than the DA)? Basically, I think we all believe that the victim rights process is broken for victims. This email goes a long way in addressing that by meeting victims where they are (aka on the internet), but are there other ways to make progress? Could we go beyond digitizing a user experience that needs improvement, and think about a new and better user experience from the beginning? This is getting into pie-in-the-sky territory and may be a strong swing away from what your stakeholders expect, so feel free to ignore my ramblings for the context of this experiment :) But it's something I've been mulling over since our brainstorm last week — happy to discuss!
tdooner
pamdinevaCfA
We believe that... a digital receipt of victim rights is useful (makes people feel more competent about their role in the criminal justice system and therefore more empowered)
We will make... an email in Mailchimp that we work with Libby and Co. to send out an one-off basis to a couple of victims and track opens
We will know the hypothesis is valid by... Thursday
Quantitative measure... Something like a 10-20% open rate and a 5-10% CTR feels normal here
Qualitative measure... people seem thankful (maybe we ask for feedback or replies to email) for letting them know their rights have been recorded
Qualitative measure #2...ask Libby and Co. if it spurred any conversation, did they understand their rights better?