Review results and draw observations from suggested donations test

[equivalentideas]

Extracted from #224 read more about the actual test there.

Our aim with this donations push is to maintain the number of contributions and revenue from our end-of-financial-year drive in 2015, but scale it up with the increase in users.

We send our first email to PlanningAlerts users on June 11th to 25063 people. There were 10254 opens (40% open-rate) and 403 clicks (4% click through). We received $4069 before we sent our second email from 102 contributions (25% conversion rate on the page, .4% from total sent).

In this tests we're sending to 1900 people (2000 minus people who've opted-out) for each version of the suggested amounts.

If we scale down the results from the 2015 email, our comparison baseline is $320 from 8 contributions (.4% from total) from each group of 1900.

That's not very many people so these tests are going to many much statistical accuracy. But we might see if something is a disaster. We've also pushing for regular donors and have made considerable changes to the form.

I'm glad we're sending something to a smaller group before we send to everyone, but I'm not yet sure if we'll see much of an indication of which suggested amount set works better.

[equivalentideas]

4 days into test

From last years email, if you scale down the numbers, by 4 days in we would have had:

• 88 donations (0.35% conversion)
• $3139 total annual revenue ($0.12 pp emailed)

from our two emails we're had:

Higher suggestions

• 4 donations (0.21%)
• $355 total annual revenue ($0.18 pp emailed)
• 25% recurring (1/4)

Lower suggestions

• 4 donations (0.21%)
• $710 total annual revenue ($0.37 pp emailed)
• 25% recurring (1/4)

Observations

The time periods we're comparing don't exactly match and there are other variables at play. With such low number a single donation makes such a big difference, so the amount comparison is pretty useless too.

But... I think we can see that promoting the regular donations more is making a big difference in terms of total annual return. We can see from these few people we've sent to a much higher annual return than when we weren't emphasising and defaulting to recurring. I think that's pretty promising.

We also had two people email after that they meant to make a one-off but made recurring accidentally. That's a very high error rate. I can see how people might visually jump past the frequency selection to the amounts. In Google Analytics we're tracking interaction with this selection, and we can see quite a lot of use—so the button’s aren't invisible to people.

I think one solution to try and make the state clearer is to include it in the 'next step button' as 'Next step with monthly contribution' or 'Next step with one-off contribution'. We could also add 'Confirm monthly contribution' or 'Confirm one-off contribution' to the final button.

Gonna come back to this is a few hours to think about next steps.

[equivalentideas]

I think one solution to try and make the state clearer is to include it in the 'next step button' as 'Next step with monthly contribution' or 'Next step with one-off contribution'. We could also add 'Confirm monthly contribution' or 'Confirm one-off contribution' to the final button.

I've implemented this, I think it makes it much clearer, and it also reinforces that this process is about making a contribution as you step through. I'm pleased with that for now.

[equivalentideas]

I'm feeling good about the form changes we've made to emphasis the regular donations.

I think we should go with the smaller suggested donations set. This is because people still used them to give us more money than the higher variation, and I think they're a bit more inclusive to people on lower-incomes with a '$5' month and '$15' amounts—I think this matches OAF’s character better. I think we should also set the default regular amount to $15 as I think this is something more people will consider and it's much better for us it they come on for $15 a month than donate much more once off.

I'm gonna set this up to run across the different pages, and close this.