search

facebookincubator / GeoLift

GeoLift is an end-to-end geo-experimental methodology based on Synthetic Control Methods used to measure the true incremental effect (Lift) of ad campaign.
https://facebookincubator.github.io/GeoLift/
MIT License
182 stars 55 forks source link

Feature Request: Handling of multiple test cells #35

Closed michaelwexler closed 2 years ago

michaelwexler commented 2 years ago

A simple test is a basic A/B, a single or few test geos vs. a single or few control geos. However, we often want to test variations, such as 50% more spend and 100% more spend, each vs. control, or Radio vs. TV. vs. both vs. control.

In these cases, to help with planning, the tool could either ask for # of test conditions needed and provide either a) test and control sets, deduping test geos (this means multiple control measurements, as they may all be different for each test set), or b) test sets that are equivalent enough to be compared, and a single (synthetic) control which can be used as a baseline for all conditions.

I'm assuming this is a natural extension of the current tooling, but I could be (very) wrong.

sohammarin commented 2 years ago

I was able to set up a multiple cell test by making use of the exclude_markets and include_markets parameter.

Example setup:

  1. Start with list of N markets and run GeoLiftMarketSelection to return list of A markets
  2. Remove markets A from original dataset and run GeoLiftMarketSelection on (N-A) to return list of B markets
  3. Remove markets B from original dataset and run GeoLiftMarketSelection on (N-B) markets while setting parameter include_markets = A to force similar results as step 1 with different control group
  4. Cell 1 Test A, Cell 2 Test B, Cell 3 Control (N-A-B)

Not sure if this helps your use case but was effective in setting up 3 cell test for me.

marguegit commented 2 years ago

Hi @sohammarin ,

thank you for sharing your approach this is really smart and useful. Small remark / question on step 3. In my opinion, it's also necessary to exclude the markets which are part of the weights for the control in step 2 for B markets. If not, there is a risk that you would expose A markets (from the adjusted set up of step 3) to A treatment, which would not be valid as a true control anymore for B.

What do you think?

ArturoEsquerra commented 2 years ago

Hi @marguegit, @sohammarin, and @michaelwexler. We're excited to announce that today we rolled-out a new version of the code (v2.5) which includes Multi-Cell capabilities!