Does an automatically generated list of recent successful requests result in more people clicking on the start of the 'make a request' funnel than a curated set of 'top requests'?

[garethrees]

Q: Does an automatically generated list of recent successful requests result in more people clicking on the start of the 'make a request' funnel than a curated set of 'top requests'?

[Explain how you will measure this]

Measure change in conversion funnel

[What do we need to do to be able to take the measurements?]

Make version of front page that uses list of successful requests - A/B test in google. Make sure funnel is defined in google analytics

[garethrees]

Related https://github.com/mysociety/alaveteli-experiments/issues/23

[garethrees]

Pull? https://github.com/mysociety/asktheeu-theme/pull/26

[garethrees]

Blocked by https://github.com/mysociety/asktheeu-theme/issues/35

[lizconlan]

Experiment "Successful requests on front page 2" now running (barring any further problem reports), should be able to see the results in 2 weeks' time

[lizconlan]

interesting - I set the experiment to run for too long by mistake so Google is not yet ready to "determine a winner" but this is currently showing a higher conversion rate - 2.67% vs 2.38% - for our variant page which is being hailed as having a 67% probability of outperforming the original.

[lizconlan]

I set the experiment to run for too long by mistake

ah, that's not entirely true - it's set to run for a 2 week minimum (which has passed) and until it hits a confidence threshold of 95%; hopefully running it for longer will swamp our (my?) configuration error of allowing the experiment code be served to the dev environment.

[garethrees]

Current progress:

How long are we going to leave this run for?

[lizconlan]

Nick says that there's no harm in leaving it running until Analytics declares a winner, but...

...as we've run it for 60 days+ with no clear winner we could equally say that it's not making a significant difference and shut it down ourselves.

We could even consider re-running it once we've got #3360 sorted out

[garethrees]

Personally I'd say the answer to this question is pretty much "it makes no noticeable difference".

This doesn't really surprise me, given that we know the homepage isn't viewed by a large proportion of visitors.

[lizconlan]

Experiment stopped, moving it to needs-writeup

[lizconlan]

(we could take out the experiment-specific code on the page at this point if we wanted)

[garethrees]

• [x] Stop experiment
• [x] Write up (should be a short one)
• [x] Remove experiment code (being done as part of the 0.25 upgrade)