search

facebookincubator / GeoLift

GeoLift is an end-to-end geo-experimental methodology based on Synthetic Control Methods used to measure the true incremental effect (Lift) of ad campaign.
https://facebookincubator.github.io/GeoLift/
MIT License
181 stars 55 forks source link

Understanding control and test split #166

Closed ar-asur closed 3 months ago

ar-asur commented 1 year ago

Hi team,

I am trying to use GeoLift package for a particular test design and would like to understand if it is possible and any suggestions. I am planning to run few incrementality tests on marketing channels. I would like to split the US states into test and control (possibly 3 groups). I understand it is possible with the package to get specific number of control groups with matching test groups. My question is How do I make sure that both groups mimic the whole US? I want to extend the results/insights to national level. Thanks!

kesemadi commented 3 months ago

Hi, To design a GeoLift experiment that ensures your test and control groups accurately represent the whole US, you need to carefully select and balance your groups. Here are the steps and considerations to help you achieve this using the GeoLift package:

  1. Clearly define the KPIs you want to measure (e.g., sales, conversion rates). Determine the desired effect size and statistical power for your test.

  2. Collect historical data for all US states. This data should include the KPIs and any other relevant covariates. Ensure your data is clean and standardized.

  3. Use the GeoLiftMarketSelection() function to select the most suitable test and control markets. Set parameters such as the number of treatment periods, number of locations in each treatment group, effect size, and holdout proportion. Ensure to include the correlation analysis to check how well the test and control regions mimic the national level.

  4. Ensure that the selected test and control groups have similar historical KPI values and trends. This helps in mimicking the national behavior. Use the correlations parameter in the GeoLiftMarketSelection() function to ensure high similarity between the groups.