Investigate Website A/B Testing

[joepavitt]

Story

No response

Description

A/B Testing in PostHog is known as "Feature Flags" - https://posthog.com/manual/feature-flags

Given we generate static content through Eleventy, we need to investigate what we can get away with and our our site would render the Feature Flags accordingly.

This task will be scoped to exploring the options, and testing out results in PostHog, to understand what the workflow looks like.

[joepavitt]

Currently knee deep in learning about Netlify's Edge Functions as Eleventy Edge (https://www.11ty.dev/docs/plugins/edge/) seems an option, although still trying to understand constraints.

Our biggest challenge is that we're generating static content via Eleventy, but need a dynamic variable for each user depending on the response from the PostHog Feature Flag API call https://posthog.com/manual/feature-flags

[joepavitt]

turns out we want "Experiments" in PostHog (https://posthog.com/manual/experimentation) not Feature Flags

[joepavitt]

Having played with this more, these are my findings:

PostHog Experiments

• Experiments in PostHog are very easy to setup, and configure.
• You can define a single primary target metric (e.g. CTA to Join FF Cloud) and it will auto-optimise for that.
• You can also define secondary metrics, which will dashboard alongside your primary to see additional impact of any changes/variants.

Eleventy Limitations

• Our biggest challenge continues to be our use of Eleventy.
• Eleventy is a static site generator, we need dynamic content to render different on a person-by-person basis.

Eleventy Edge Functions (docs)

• Explicitly advertises that it can be used for AB Testing, although with no examples
• Requires Netlify server, we would no longer be able to host in GitHub pages
• No external functionality/injections, everything must be done locally to the edge function, meaning all of the "magic" of Experiments are lost, we'd just be doing a random selection each time.
• Can set cookies (e.g. AB Testing on Netlify Edge) which means we could in theory package over the PostHog experiments stuff, but it's a lot of manual cranking
• Not so easy to format into AB Testing on a line-by-line basis, this generally works better for complete page redirects, the code to include in the templates is {% edge %} content {% endedge %} where you can then access cookies & query params directly inside the template.

Eleventy Shortcodes:

• I thought I was on to something with Async Paired Shortcodes, it's really easy to build and formats nicely, managed to get a working demo with PostHog experiments that looks like this in a template:

{% abtesting "flag", "control"%}
    <h1 class="text-gray-50 max-w-lg m-auto">
        DevOps for Node-RED
    </h1>
{% endabtesting %}
{% abtesting "flag", "test"%}
    <h1 class="text-gray-50 max-w-lg m-auto">
        Run Node-RED in Production
    </h1>
{% endabtesting %}

and then uses the PostHog Experiments call in the shortcode handler:

eleventyConfig.addPairedShortcode("abtesting", async function (content, flag, value) {
        const fFlag = await posthog.getFeatureFlag('test-flag', 'joepavitt@flowforge.com')
        console.log(fFlag)
        if (fFlag === value) {
            return `${content}`
        } else {
            return ''
        }
})

• The limitation comes in two forms:
  • We need to pass the distinctid/userid into the call each time for PostHog, which is a limitation of their library, but allows us to track exactly which users saw which content, and allows for PH to auto-optimise content.
  • This is all still rendered upon server-start, and is not ran on the edge, therefore the same content still gets rendered to everyone (I'd misinterpreted the docs and thought it still ran at edge, only when building it, did I realise that wasn't the case)

Our Options

From my understanding, we are now left with 3 options:

1. No A/B Testing, stay with Eleventy
2. Have "edge" A/B Testing to randomly show different content to different users, then workout posthog's cookie structure and set the relevant cookie client-side, and call to PH to let it know what we have chosen, so that we can store the relevant data. With this option, we can no longer host on GitHub pages, and would need to switch to Netlify.
3. Switch to something like Astro which is also a static site generator, but have JS built in natively, so can run (where we want to) server-side rendered code. It also comes with much better Headless CMS Integration which would mean we could move the editing/maintenance of the Handbook, Docs & Blog away from the website repo. However, this would be at least a few days of work in order to make that transition as it requires a re-write of the website.

[joepavitt]

Just discovered that, whilst the Eleventy Edge Function documentation is terrible, the Netlify Edge Function documentation is much better and details all of the available Web APIs - including fetch, so we could call an API directly, additionally, it details here about how we can actually use remotely served js libraries too

new investigation beginning.

[joepavitt]

Eleventy is so fiddly and tricky to work with. Following examples and best-guessing this to the best of my ability, but I just don't think this is possible for what we want to do.

[joepavitt]

Edge functions are very broken in Eleventy unfortunately. Work okay first time round, as long as you're just rendering raw content, then on refresh, they break. Really am swaying more and more towards option 3 here.

[joepavitt]

Seems as though I'm not the first to be having these issues with Eleventy: https://github.com/11ty/eleventy/issues/2585

[joepavitt]

okay, breakthrough... I have a working solution that just uses cookies and hardcoded option, but it toggles between content using an Edge Function to parse cookies.

To avoid the bug above, all of our Edge Functions must be written in liquid instead of njk.

I also have to list all of the cookies we plan on using with the Eleventy config, so we can't just add a new cookie for each feature, as such, I nest the feature flags inside a JSON object, and then encodeURIComponent() and JSON.stringify() that in order to store it. We reverse this process in the edge function with a toObj filter.

End Result in the template .njk files is something like this:

{% edge "liquid" %}
    {% assign feats = eleventy.edge.cookies.feats | toObj %}
    {% if feats.flagA == 'optionA' %}
        Hello Option A
    {% endif %}
    {% if feats.flagA == 'optionB' %}
        Hello Option B
    {% endif %}
{% endedge %}

[joepavitt]

Now added

eleventyConfig.addGlobalData("feats", () => {
        const encoded = context.cookies.get('feats')
        const decoded = decodeURIComponent(encoded)
        const feats = JSON.parse(decoded)
        return feats
      });

into our Eleventy Edge function which makes feat a global property (sensible given we' can use it store all feature flags), and now reads as this in templates:

{% edge "liquid" %}
    {% if feats.flagA == 'optionA' %}
        Hello Option A
    {% endif %}
    {% if feats.flagA == 'optionB' %}
        Hello Option B
    {% endif %}
{% endedge %}

Our main constraint is the cookie size limit of 4kB. A very rough, back of paper calculation, I've estimated we can handle about 150-200 feature flags, which is never going to be an issue.

[joepavitt]

Thought I would try reverting back to using Eleventy Shortcodes within Edge Functions now that I wasn't using njk edge functions, and it works:

{% edge "liquid" %}
    {% abtesting "flagA", "optionA" %}
        Hello Option A
    {% endabtesting %}
    {% abtesting "flagA", "optionB" %}
        Hello Option B
    {% endabtesting %}
{% endedge %}

I prefer this formatting as it makes it explicitly clear that it's A/B Testing taking place, and not easily confused with other if conditionals we have within the website.

[joepavitt]

Latest updates - fully working end-to-end PoC is in place. Still to-do:

• [x] Change the PH API key to be a an environment variable, its now used in multiple places across the repository
• [x] Look at "backup" content, with the change to shortcodes, it means we need to see something in a flag, going to improve that so that the control option shows if no flag or PostHog is found.
• [x] Need to investigate in more detail how we're going to host this.

[joepavitt]

https://answers.netlify.com/t/edge-functions-negate-endpoints-in-netlify-toml/88455

above link for me investigating the challenges around getting edge function to just run on html endpoints, and not for all images, css files, etc.