magneticio / vamp

Vamp - canary releasing and autoscaling for microservice systems
http://vamp.io
Apache License 2.0
625 stars 55 forks source link

First filtering, second weight in router #395

Closed dragoslav closed 8 years ago

dragoslav commented 8 years ago

From @olafmol

Thoughts

Filtering is segmentation/personalization, i.e. you create a specific visitor segment that is routed to a specific endpoint/frontend. This is like “personalization”. Inside such a segment it should be possible to run A/B experiments just the same. Outside (i.e. above/upstream of) the segment you can run different experiments but for a valid A/B experiment I would suspect that you always run the A/B test inside a specific segment. Because it’s about variations testing, and the segmentation should not influence the results of the test.

Example

I have 2 segments: desktop users and mobile users. I have 2 different containers for them. For mobile users segment I want to A/B test between a mobile oriented website, and a responsive HTML5 website: https://vwo.com/blog/split-testing-mobile-website-desktop-website-tablet-users-increased-revenue/ So first I set up filtering on mobile devices, and in this segment I do a 50/50 split test on mobile site and HTML5 site. More advanced: I could segment for 5% of my mobile visitors because I don’t want to expose to all my visitors as the new html5 site is not performance-optimized yet, if this gives me enough volume to do a significant test. Other alternative: I want to do an A/B test of 2 new features, but only want to build and test them for Firefox browser first due to time-constraints in my team. So first segmentation on Firefox visitors, and then create an 50/50 A/B test for the 2 versions I want to test.

More good examples of combinations of targeting and percentage-rollouts:

http://docs.launchdarkly.com/docs/targeting-users http://docs.launchdarkly.com/docs/controlled-rollouts

dragoslav commented 8 years ago

Firefox client will access 1.0.0 (50%) or 1.1.0 (50%), any other browser only 1.0.0.

name: sava:1.0

gateways:
  9050: sava/port

clusters:

  sava:
    routing:
      sticky: instance
      routes:
        sava:1.0.0:
          weight: 100%
        sava:1.1.0:
          weight: 50%
          filters:
          - condition: user-agent = Firefox

    services:
    -
      breed:
        name: sava:1.0.0
        deployable: magneticio/sava:1.0.0
        ports:
          port: 8080/http
        environment_variables:
          debug[SAVA_DEBUG]: true

      scale:
          cpu: 0.1
          memory: 256MB
          instances: 1
    -
      breed:
        name: sava:1.1.0
        deployable: magneticio/sava:1.1.0
        ports:
          port: 8080/http
        environment_variables:
          debug[SAVA_DEBUG]: true

      scale:
          cpu: 0.1
          memory: 256MB
          instances: 1