Running A/B/n tests

[awoehrl]

Hi,

is there maybe an option to run tests with more than one variant? I tried to setup an experiment with two variants, but I think this doesn't work?

Cheers Andreas

[kingo55]

Hey @awoehrl,

Sorry it's not clear - Mojito sure does support A/B/n tests. You can create them like so:

state: staging
sampleRate: 0
id: w137
name: W137 Example test
recipes:
  '0':
    name: Control
    js: 0.js
  '1':
    name: Treatment 1
    js: 1.js
    css: 1.css
  '2':
    name: Treatment 2
    js: 2.js
    css: 2.css
trigger: trigger.js

Just add extra recipe keys and Mojito will split the traffic evenly amongst the variants.

If you need a specific traffic allocation, the recipe keys accept a sampleRate parameter, but you must add this key to each variant and they must add up to 1, like so:

state: staging
sampleRate: 0
id: w137
name: W137 Example test
recipes:
  '0':
    name: Control
    js: 0.js
    sampleRate: 0.5
  '1':
    name: Treatment 1
    js: 1.js
    css: 1.css
    sampleRate: 0.25
  '2':
    name: Treatment 2
    js: 2.js
    css: 2.css
    sampleRate: 0.25
trigger: trigger.js

A handy tip is to validate your test YAML against the JSON schema in a project. E.g. If you're using VS Code, you could modify your User settings to automatically validate your tests:

    "yaml.schemas": {
        ...
        "./wave.schema.json": "lib/waves/*/*.yml"
    },

Is this what you were looking for?

[awoehrl]

Thank you so much @kingo55! It works now! The YAML validation trick is also really helpful. I realized that the wave id's are not meant to have underscores in the names. :-)

[kingo55]

Glad to hear it! YAML schema validation is amazing once you start using it... makes configuring tests less error prone - Just as you might use ES Lint to validate your JS.