Document A/B testing

[mquandalle]

It's not easy to do A/B testing with a static site generator like Gatsby but it still a feature that developers would like to implement in some case. It would be great to have some documentation about how to achieve it.

I don't know if someone has already implemented something like that, but I assume it would require a gatsby plugin to create two different versions of the same page (or more) and then having a plugin for serve to decide which version to serve (randomly or based on some metric).

[brotzky]

Gatsby integrates well with Netlify which makes it very simple to A/B test with branches. All A/B testing implementations I've tried work at a routing level (Netlify, Google) that split incoming traffic instead of built into the framework.

An interesting idea to play with serve in Gatsby. Curious to hear other people's thoughts.

[lightstrike]

@brotzky You've been able to get Netlify split testing working? I've hit some issues, were there any modifications to your config you made or did it work out of the box?

[brotzky]

@lightstrike I've never used it with Gatsby, but other static website I've built. I didn't know there were issues with Gatsby and Netlify split testing, my bad! In theory, if it worked integrated properly, it would be very simple to implement 😂.

[KyleAMathews]

Is this a Gatsby issue? I'm not sure Gatsby can do a/b testing by itself without help from a service so any docs would necessarily be part of the service e.g. Netlify. Happy to add core support where needed. Something I've thought about is the idea of adding support for "Page Variants" which could then be used by a service.

[lightstrike]

I don't believe this is a Gatsby issue today -- once Netlify support is figured out it would be nice to add some built-in support within the netlify plugin. The approach I take is to set a cookie value based on variant, then use Google Tag Manager to set a custom dimension value in Google Analytics.

[ghost]

Hi Guys,

One humble suggestion don't offshore A/B testing or other personalization features to hosting vendors(like netlify) or other tricks. Because people in open source simply assume that Netlify kind of solutions are universally accepted which is not the case.

In principle I look forward to Kyle & other key contributor to suggest best way Gatsby can support this. It may be my naive thinking, but such capabilities can be achieved if there is source of data to be consumed by front-end React components through defined behavior or configured tests. These source of data can be plain CMSs like Contentful or Middleware which allow websites authors to configure these behavior or tests. For example., Algolia, Coveo etc., or develop such custom middleware which can take content in from any data source we want, defined tests, behavior when to show what content and frontend components renders such content using API from client side. Now obviously, this could be out of scope of Gatsby Core framework but Gatsby as company should have way outlined that provide very open way to adopt personalizations.

Other simple way to implement personlization is to simply use monetate etc., But without personalization getting supported, big brand's adoption is far dream.

Regards Vishal

[lightstrike]

Potentially all is well with v2, I just threw together a quick demo site and the splitting seems fine (control is dull red and test is hot pink).

I'll circle back to my problem project when I get the chance to upgrade it to v2 and will see if that fixes things.

[KyleAMathews]

@vishalniit personalization & a/b testing is definitely something we will support! Definitely agree any web framework needs 1st class support for it.

[KyleAMathews]

Due to the high volume of issues, we're closing out older ones without recent activity. Please open a new issue if you need help!

[AmberWilliams]

@vishalniit personalization & a/b testing is definitely something we will support! Definitely agree any web framework needs 1st class support for it.

Will support when?