Does varying the text and number of buttons on the donation call-to-action change click-through?

[ajparsons]

Q: Does varying the text and number of buttons on the donation call-to-action change click-through?

We will measure this with via an A/B test in google analytics. Campaigns will be marked up to track conversion value.

Goal is to see if naming exact amounts 'donate £5 now!' is more effective than 'donate' - and is so use this to fine-tune our default options for donations.

[ajparsons]

I'd leave it till tomorrow in case, but I'm not sure it's tracking conversions properly.

There have been two clickthroughs to mysociety.org from that campaign (one yesterday, one today) - but none recorded by the experiment. Both of these were 'experiment_no_0' (default) - so could be people not seeing it, but also then odd if only people not seeing it are clicking through at all.

https://analytics.google.com/analytics/web/?authuser=1#/report/trafficsources-campaigns/a660910w61534424p63017268/_u.date00=20190813&_u.date01=20190814&explorer-table.plotKeys=%5B%5D&explorer-table.secSegmentId=analytics.adContent&_r.drilldown=analytics.campaign:foi_request_page

[ajparsons]

Okay, clicked the button and it didn't recorded a conversion - looking at how this is working the problem seems to be we're not recording any of the 'Request page donate button' events.

https://analytics.google.com/analytics/web/?authuser=1#/report/content-event-overview/a24614779w48006686p48388638/overview-dimensionSummary.selectedGroup=visitors&overview-dimensionSummary.selectedDimension=analytics.eventCategory

[garethrees]

We have UTM params – is that not enough? Do we need to also add an explicit "event" to the link like we do on the sidebar exits?

[ajparsons]

Yes, that's what's being tracked but it's already in the the code (https://github.com/mysociety/whatdotheyknow-theme/pull/591/commits/636b881b6e66a4d0177a48f4a819f4ac8275bd5d - line 25) - but doesn't seem to be being fired.

[garethrees]

@zarino anything obvious that you can see that may cause this?

[zarino]

Sorry this whole ticket has been such a pain for us all @garethrees @ajparsons!!

I think the event isn’t being recorded, because the donate button click handler was generating an exception, so the browser just followed the link as normal, rather than firing off the analytics event. The exception was being caused by e.preventDefault() – in our callback, e was undefined because, at some point, it got missed out of the function () { definition. Grr!

Here’s a PR which fixes that: https://github.com/mysociety/whatdotheyknow-theme/pull/604

I’ve checked it locally and can confirm, with that fix in place, ga(…) gets called correctly on click, and the page doesn’t go anywhere until the hitCallback is called.

I can‘t see anything else, code-wise, that could be preventing the event from being tracked.

[ajparsons]

The session to conversion rate is off because of the initial recording problem, but in general this isn't returning conclusive results.

It might eventually show that £10 is slightly better or £5 is slightly worse - but the different is going to be so small the actual impactful difference is small regardless.

So my verdict is there's no reason to think this makes a difference.

[garethrees]

Any better data at this point? What's the next step on this?

[ajparsons]

Data is the same - can just hit end on the experiment in optimise