A/B testing, Split tests

[anonymous-matomo-user]

Which of our landing pages are the most successful and convert our visitors better?

How often did we wonder if we have the right words and images on our landing page (the page where visitors arrive from your advertising)?[What if changing just one image on the page could increase the visitor conversion rate several times?[BR] We will never know this without testing.

The good news is that there is a technology that can help us quickly determine the best version of our landing page. It is called Split-Testing and it should be implemented in Piwik.

Piwik should randomly show different versions of the same landing page to our visitors. It should make sure that each version of the page is displayed the same number of times overall and it should even display the same version of the page to the same visitor (to make their experience with our site consistent).

With split-tests we should be able to quickly see which version of the page has a good conversion rate and which one is just a waste of the advertising money. Get rid of the unsuccessful version, keep the good one and our overall conversion will increase.

[robocoder]

Other services:

• !http://visualwebsiteoptimizer.com/
• !http://www.optimizely.com/

[robocoder]

See https://github.com/mysociety/abtests

[robocoder]

WebSite Optimizer is being folded into GA.

[mattab]

I think it was reasonnable for us to recommend using Website Optimizer, but now that they are forcing users to use GA for split testing, we should consider implementing similar functionality in Piwik :)

[mattab]

It would be fantastic to have A/B testing feature built in Piwik, as a plugin. If you are interested and could help us make it happen, please email at hello@piwik.org!

[nkuehn]

my two cents in case you consider implementing A/B testing support (which would be awesome):

An extremely powerful way to realize the A/B testing capabilities but also much more would be to add a generic ability to compare segments. The new segment editor is awesome and could be leveraged even more.

A/B variant tags would then just be custom variables (typically visit-scope) like any other.

A feature like Google Analytics' capability to actively control which users get which variant from inside the analytics system is just for convenience, but not a core requirement IMHO. One can do the actual traffic split on the website server (or loadbalancer) or in the webisteJavascript as well.

[mattab]

Facebook has released their A/B testing tool: http://facebook.github.io/planout/ https://github.com/facebook/planout - it's written in python. Maybe we could get some inspiration from them.

[pattonwebz]

I'm in agreement with nkuehn about it being unnecessary to have the ability to control variants from inside the analytics system.

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

I've been tracking split-test campaigns in GA for a while now using Events. Earlier today Piwik also introduced Event tracking (http://piwik.org/docs/event-tracking/) which I was then able to integrate with my exiting split testing framework (based on this javascript library on github: https://github.com/jamesyu/cohorts/).

Using almost exactly the same code as I use for GA tracking I am now able to track split-test campaigns with Piwik through with the Javascript API and Events. I'm using the following code which reports the same data to both GA and Piwik. First line is passing data to GA, second to Piwik.

_gaq.push(['_trackEvent', category, action, opt_label, opt_value, int_hit]);
_paq.push(['trackEvent', category , action, opt_label, opt_value]);

That code is called onInitialization of the test (consistency is handled by cookies before test initialization) to track how many times each test runs and then it's allso called a second time onClick passing a different label so that you can segment views from clicks.

[mattab]

Kissmetrics A/B testing product is interesting also: http://blog.kissmetrics.com/kissmetrics-ab-test-report/

[mattab]

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

It is not the "best place" if you are already using a A/B testing tool, but many people are only using Piwik on their website as "third party javascript" and would enjoy basic A/B tests feature in Piwik.

Maybe we could make it possible to use Piwik to measure A/B testing results from another tool? Users could use their favorite A/B testing tool to define and manage tests and variants, and send the data directly to Piwik (or Piwik fetch it from their API).

Our vision is to empower users with the toolset to measure and make changes on their website. Running A/B tests (create variants, serve them, measure performance, report results) is a key part of this vision :+1:

[pattonwebz]

Hey Matt,

At least we agree that the 'best place' isn't directly within the Piwik platform, but it obviously wouldn't be a breaking change for me if it did include some functions for implementing tests.

I have seen the roadmap showing that A/B testing is to be implemented into the core - which is how I found this ticket - and I don't necessarily think it's a bad idea.

My primary reason for stating objections is not to prevent it being added (because it would be a greatly welcomed featured for a lot of users) but rather to note that some users would rather make use of a platform independent testing model that does not tie you directly to any one anayltics platform or the other. As such it would mean that the testing would need to be turned on/off at the provider end to prevent bloat in the tracking code for users who don't want it.

I get that having to use an external provider is not ideal but self-hosted libraries are also an option to negate the requirement of an external provider.

I guess what I'm really saying is that platform independence is important FOR ME but I get that it's not as important for others.

As I stated I already run tests and track them with Piwik using Events and I'd actually love to be able to help shape the integration of A/B testing to the core - which is why I commented on this ticket in the hope that someone from the core team would see it and I'd be able to make suggestions and push for my choice of perfect match features in they system ;)

[mattab]

Good we're having this discussion, it's useful! Here are some thoughts on how it a new A/B Plugin could be broken down:

1. let user define A/B tests and their variants via new UI and API. Variations include:
   • URL variations of same page new (mvp?),
   • let user create tests variants by inputting a HTML text (mvp?),
   • advanced UI to create test variants directly within the website like commercial A/B testing tools do very nicely (not in mvp)
2. Javascript tracker (client side)
   • the Javascript tracker fetches the tests variants for this the web page or app,
     • (we need to let user in JS know about which A/B tests are running on the site. let's think of how we can avoid leaking all details of running A/B tests in the process?),
   • decides how to assign a visitor to a given test and given variant based on the variant probability,
     • maybe we start with a simple algorithm eg. 0.5 probability of A and B variants
     • (we may need to process the probabilities server side in case some of the variants will need to be given more or less traffic)
   • selected test variant info is persisted in a new first party cookie,
     • a user will be always shown the same variant (using this cookie we know which variant was served before),
   • Maybe we could reuse some code or ideas from this project for JS API
3. Tracking API (server side)
   • Store the tests and variants viewed by a visitor, for each visit and goal conversion
     • do we also need to store variants for each action / page/ event?
4. Reporting:
   • process the KPIs for each A/B test such as: % change in Actions per visit, Time on site...
   • process Conversion rate of each A/B test variant for each Goal known in Piwik (eg. Goal "Download brochure" conversion rate increased 50% in test variant A")
   • create an overview report listing most important metrics and in information, and also let user view all % changes in KPIs for each running A/B tests,
   • Let user know when test is finished, eg. this website calculator http://www.experimentcalculator.com/#lift=20&conversion=5&visits=1000

Other notes & ideas about A/B plugin:

• when disableCookies is used, maybe do not disable the a/b testing cookie
• let user know when a A/B test has enough data to be statistically significant.
• often (for low-med traffic websites) it may take several weeks to get statistically significant results, maybe we could notify users that a A/B test(s) is finished?
• maybe we could automatically pick the winning variant when a test is marked as "statistically significant better" (useful if user cannot directly implement the changes in site or app) )
• A/B tests could be available as Segment so users can see all Piwik reports segmented by experiments
  • this also would let users open in one click, a segment of visitors that were running in variant [abTestVariantName] in [abTestName]
• maybe it could work for tests created with other A/B tools that offer Javascript APIs to get the tests and variants (such as optimizely).
• maybe in Visitor log we could show which tests & tests variants were seen by a given user and/or given action.
• maybe users already use other A/B tests tools where they create variant and just want to "import them in Piwik" to measure them... supporting this may be complicated)
• random testing idea: we should run A/A experiments and check that the A/B testing engine does not declare one experiment as being "statistically significant" better!
• (not mvp) a A/B test could be customised to be served only to a particular "high value" segment (based on a Custom Segment in Piwik) (optimizely does something like this)

Keep feedback coming!

(edited)

[mattab]

Interesting also: http://bentilly.blogspot.co.nz/2012/10/ab-testing-scale-cheat-sheet.html

Most Winning A/B Test Results are Illusory On the importance of running the power calculation: What is sample size needed? Indicate how long will the test likely take before reaching sample size.

[nkuehn]

Hi erveryone, great to see that there's a lot of interest. What do you think about splitting the ticket into two:

1. one about the proposed feature that allows controlling the test run from inside piwik (I agree with Matt that there's relevant market pressure to have some kind of feature like this, no matter how big or small it is - I personally don't need it, but that's not relevant)
2. one about the necessary analysis features (especially visualizing differences between segments / test groups instead of having to switch back and forth manually), which are required anyways, no matter which way you run the tests (this new piwik feature, homebrew, optimizely, ... you name it).

The "only" thing that both definitely need is a conceptual decision on how the A/B comparison is realized in the piwik core (e.g. visual segment comparison vs. a specialized feature with own database structures vs. ... ). Probably the hard part...

@williampatton: I personally think that custom session-scope variables are more suitable to tag the visitors into A/A2/B/etc than events (conversion is the typical success aspect and conversion is usually session-scope, which events aren't). Unfortunately their number is limited as far as I understand the Piwik API.

[mattab]

Interesting article: What you should know about the Optimizely vulnerability which discusses the privacy implications of A/B testing implementations, and how, we should aim to avoid 'leaking' A/B tests details in the javascript of A/B testing users.

[tassoman]

I agree that having a full AB Testing feature would be having too much inside Piwik.

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

[mattab]

A/B testing tools article

[mattab]

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

current experiment picked by A/B testing logic will be stored in a cookie and in a custom variable in piwik visit and page rows, and we may also use it to segment the report data for each experiment.

edit: interesting template: http://www.bloggerchica.com/test-design-the-official-doc/

[Tobias-Conrad]

Hi what is the status? How could A/B Testing be used?

[nkuehn]

FYI: I have a new Job and therefore can' participate with ideas and testing based on real world data any more. The comparison dashboard looks very nice as far as the screeshot can tell. Hopefully you get the funding campaign completed!

[marclaporte]

20 lines of code that will beat A/B testing every time http://stevehanov.ca/blog/index.php?id=132

What’s Happening In The A/B Testing Market? https://blog.builtwith.com/2013/07/19/whats-happening-in-the-ab-testing-market/

Seeking Edge, Websites Turn to Experiments ; Optimization technology is reshaping publishers’ decision-making process—and the Web itself. http://www.technologyreview.com/news/523671/seeking-edge-websites-turn-to-experiments/

"On 1 June 2012, Google announced that GWO as a separate product would be retired as of 1 August, and some of its functionality would be integrated into Google Analytics as Google Analytics Content Experiments. Google Website Optimizer allowed webmasters to test an alternative version of an entire page, known as A/B testing — or test multiple combinations of page elements such as headings, images, or body copy; known as Multivariate testing. This tool was part of Google Analytics, though accessed through a different user interface. As it was part of Google Analytics, it used the Google analytics scripts." http://en.wikipedia.org/wiki/Google_Website_Optimizer

[Tobias-Conrad]

ticket opened 2008, added to major 2014, may it is time to give a easy solution made by piwik. like https://clicky.com/help/faq/features/split-tests/how

[Tobias-Conrad]

@mattab just do it. why not finished?

[abishekrsrikaanth]

This is a PHP based implementation of Facebook PlanOut. https://github.com/vimeo/ABLincoln

[mattab]

the scientist project looks also possibly interesting https://github.com/daylerees/scientist

[mayannath]

since 2012.. no further insights into this thread? This is an interesting topic. Dev's please provide further details on Split-Testing mechanism through Piwik.

[mattab]

Today Google re-launched their A/B testing tool as Google Optimize. Here are a couple screenshots.

Google Optimize Editor

Example of Google Optimize Report

Source: https://www.e-nor.com/blog/google-analytics/google-analytics-360-suite-from-measurement-to-optimization

[tassoman]

It's quite big amount of informations but in my opinion is missing of page loading time. More, I would draw the "to be Best" graph in green and the other in red.

[JLuc]

Is there a simple way we can testA/B with piwik now ?

[tsteur]

We have released a plugin that works on top of Piwik to run A/B Tests, Split Tests, Experiments see http://www.ab-tests.net/ . A full list of features is mentioned on the Marketplace at https://plugins.piwik.org/AbTesting . It lets you easily run experiments on websites, servers, apps and in campaigns and has lots of features to customize your experiment and to define your success and your expectations for an experiment to increases your sales, revenue, conversions, pageviews, and more.

More docs are available at: User Guides, FAQ, Developer docs

[williamtheaker]

That plugin is proprietary, which is quite disappointing given Piwik's emphasis on the importance of software freedom.

Can you reopen this issue until there's a libre/open source A/B testing plugin for Piwik? It's still on the Piwik.org roadmap.

[pattonwebz]

I 2nd the notion that this issue should remain open until there is an open source alternative available.

[mattab]

We hope that although not open source, many will enjoy the new A/B testing product.

Researching, building A/B testing, documenting, testing, took months of work. By purchasing a license, you get a fully working product, with free updates for duration of license, and you stay in full control of your data.

And on top of this, you help the Piwik core platform engineers to grow and keep innovate. That's because a % of all non-open plugin license sales earnings directly go to funding Piwik engineering. In particular what plugins sales fund are:

• core Piwik platform innovation: where everything is open and will remain so,
• our free and open source plugins we ourselves create and maintain. Already 12 of them and we will create more in the future.

Next week we'll publish a blog post on the Piwik blog, with a bit more information.

ok to reopen the issue as suggested.

[pattonwebz]

Glad this is reopened and hopeful to see some kind of a core implementation for split testing some time in the future.

Also +1 for the licencencing fees contributing to core development. I've suggested to a client that this plugin is worth considering for them and am excited to get the chance to trial it in production.

[mattab]

Thanks @pattonwebz for advocating the plugin to clients, we do appreciate your support a lot :+1:

http://www.ab-tests.net/

[mayannath]

I think the plugin idea is great! Hope to see development on this front soon. Thanks for re-opening.

On Thu, Oct 27, 2016 at 4:50 PM, Matthieu Aubry notifications@github.com wrote:

Thanks @pattonwebz https://github.com/pattonwebz for advocating the plugin to clients, we do appreciate your support a lot 👍

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/piwik/piwik/issues/459#issuecomment-256801544, or mute the thread https://github.com/notifications/unsubscribe-auth/AOMRo6eFn-UDYP7gZ4OcfeNCYpEzis_xks5q4TivgaJpZM4CVR55 .

Mayan N.

[mattab]

Available at A/B Testing (Paid feature at this time).

[pattonwebz]

Wow, a decade since I found this issue almost and 15 years since the issue was opened. This might be the longest issue thread I've been on that ended with a resolution.

I've gone through many A/B testing tools in that time, I guess it is time for me to try one more haha