Alaveteli Experiments

A collection of experiments and research to learn about what makes an Alaveteli grow.

An "experiment" is an iteration of the "Build → Measure → Learn" cycle from The Lean Startup.

Build, Measure, Learn Cycle diagram

Experiments should ideally return a yes or no answer, or return a single number in answer to a question, e.g. 89%.

Experiments are here to change our behaviour, and consequently, what we build.

Experiments should generally be focused around three areas:

Acquisition: Get new users
Engagement: Keep users using
Retention: Stop users leaving

Results

A summary of results from what we've tried so far. These results show what we found given the experiment conditions. They are by no means definitive. We encourage you to try and tweak these experiments for your Alaveteli, and let us know what the results were.

Acquisition

Question	Summary
Do Twitter bots posting the built-in RSS feed increase traffic?	No
Will using social media increase the growth of sites?	Maybe. The usage picture is unclear, but if there is an answer it's probably Facebook.
Which were the top referrers when WDTK first launched?	Unknown. We don't hold data going back that far.
Which countries' FOI law supports commercial use?	All that we've asked so far.
What proportion of users enter on the homepage?	15-20%
Will promoting the classification game help drive traffic?	Maybe. The classification game does not get many visitors but some sites have such a high classification rate that there would be very little that a passing classifier could do.
Which pages are being linked to on social media? And does that explain the high conversion rate?	A handful of request pages. We have some concerns about the accuracy of the apparent conversion rates though.
Can we drive traffic to AsktheEU from our newsletter/blog post?	Yes. But only a very, very small number.
Are the widgets effective in growing the site?	No. Not enough traffic to make an impact.
Can we drive traffic from WDTK to AskTheEU?	Yes. Placing boxes advertising AsktheEU on WDTK has increased traffic and transactions on AsktheEU. 7.67% of all traffic came from WDTK.
Where is our existing traffic being referred from? Where might there be potential users that we're not currently reaching?	Predominantly social media. Sites with a strong journalistic focus get a greater proportion of traffic from news and campaign sites.
Does using Facebook advertising on Right To Know day drive usage of Alaveteli sites?	Yes, Facebook ads helped to bring new users to sites, but in the majority of cases it was only a small number. Overall, 0.5% of people who saw the ads then visited an Alaveteli website. Only one person went on to make a request after seeing a Facebook advert. However, there are indications that using Facebook ads in Uganda has led to sustained higher usage levels of the site afterwards.
Does commenting on news articles encourage traffic?	Not significantly, but it can bring lower numbers of very engaged users.
Can we use the `PublicBody#notes` field to improve Adwords traffic?	Yes, but this got complicated.
Can increase traffic through a series of blog posts?	No. Only a handful of visitors came through the links.
How much traffic comes to RightToKnow.org.au through Adwords?	12.79% in June 2016.
How much traffic comes to KiMiTud through Adwords?	25.54% between 20 March 2016 and 27 September 2016.
Adwords A/B: “FOI made easy” vs “Freedom of Information Made Easy”	“Freedom of Information Made Easy” has around double the CTR and slightly higher average position when tested on RightToKnow.

Engagement

Question	Summary
Is an increase in FOI charges discouraging people from using KiMitTud?	Maybe. The perception of users might have changed, but the behaviour of authorities is unchanged.
Does the response from the authority influence whether a requester makes another request?	Yes. There is a higher success rate (and a marginally lower "unresolved" rate) for the first requests of people who go on to make further requests than for those who stop after the first one.
Has upgrading AskTheEU.org to the responsive theme had an impact on usage numbers?	Yes. Bounce rate has decreased and site is now "mobile-friendly".
Does the requirement to log in before starting the request process decrease conversions?	Yes
Will promoting the classification game help drive traffic?	Maybe. The classification game does not get many visitors but some sites have such a high classification rate that there would be very little that a passing classifier could do.
Are some sources of traffic better in terms of people making requests?	Social Media. A small but valuable resource?
Is anybody clicking the social media links on the site?	Yes. But it's only a small percentage of visitors. On WDTK - we don't have data for any other sites yet.
Which pages are being linked to on social media? And does that explain the high conversion rate?	A handful of request pages. We have some concerns about the accuracy of the apparent conversion rates though.
What are popular topics on Alaveteli sites?	This is hard to answer currently. But contracts and costs are popular across sites.
What are the different user personas that use an Alaveteli?	30%–50% of users do nothing. 20%–25% make a single request. 15%-20% are requesters, followed by followers (5%–13%) and commenters (0.5%–2%).
Is the "similar requests" sidebar a valuable source of traffic?	Yes. 10% of traffic to request pages comes through the similar requests. Engagement is also increased.
Does showing recent successful requests on the homepage increase conversion rate?	No.
Does a new Alaveteli with ~100 pre-seeded requests grow better?	It won't grow without them. Initial requests are not a direct growth mechanism – you need marketing too – but no Alaveteli has lots of traffic but a low number of requests.
Do prominent donation banners increase donations?	Yes. We saw a 23.5% increase in visits to our donation page from WhatDoTheyKnow over the 2016 Christmas donation drive. We don't know how many extra conversions that resulted in, but it shows people are interested.
How long did it take for WDTK to hit target level?	6-9 months.

Retention

Question	Summary
Has upgrading AskTheEU.org to the responsive theme had an impact on usage numbers?	Yes. Bounce rate has decreased and site is now "mobile-friendly".
Does the requirement to log in before starting the request process decrease conversions?	Yes
What are the different user personas that use an Alaveteli?	30%–50% of users do nothing. 20%–25% make a single request. 15%-20% are requesters, followed by followers (5%–13%) and commenters (0.5%–2%).
Does a new Alaveteli with ~100 pre-seeded requests grow better?	It won't grow without them. Initial requests are not a direct growth mechanism – you need marketing too – but no Alaveteli has lots of traffic but a low number of requests.

Other

Question	Summary
Would admins prefer mySociety host their Alaveteli?	Yes. Most were warm to the idea.
What factors caused WhatDoTheyKnow to grow?	Lots of things, but mainly Google Adwords (and SEO optimisation) and lots of press coverage.