cetagostini
commented
5 months ago Hey team, this topic is super interesting 🔥
First I have a personal opinion regarding the Meta approach, because I feel can be improved. Many times knowing where you can execute an intervention is useless. Users don't want to experiment for the sake of experimenting, they want to intervene in regions/geos where they can derive information.
Example: Let's imagine I'm running a mobility company that operates globally. I decide to use GeoLift package's MultiCellMarketSelection feature to identify the best location for a marketing intervention. The tool suggests the Shetland Islands as the ideal place to run a test. After conducting the test, I find that our marketing campaigns perform exceptionally well there. As a result, I replicate the same strategies in other regions in the UK. Surprisingly, the outcome is not as successful despite investing more resources. Could it be that the initial test results were misleading?
The test never lied, but we analyzed a geography with totally different patterns from the other geographies that really interested us. If the geography where we can intervene is not of our interest or is it too different from the place where we really dedicate 90% of our budget, even if it is the best place to experiment, why should we do it there?
We can extrapolate this example to more than one region. What if the pull of intervention regions are not relevant?
Based on my experience, all marketers have a clear idea of where they want to test new strategies. We aim to gather insights from London or Manchester, where the majority of our resources are concentrated, to better understand local consumer behavior, as this is where 80% of our impact lies. The challenge lies in identifying the most suitable geographic locations to accurately represent my target area and minimize the MDE in my experimental methodology.
What if we turned the question around and asked, "If I want to conduct an experiment in X Geo (London), what would be the optimal control geo (City) to select?". Where our objective function is increase the sensitivity of my experiment (reduce MDE).
We can certainly train several models estimating the power or minimum detectable effect for each combination of predictors and target variables. This process will provide us with two potential courses of action:
- We train different models with different cities as controls and select the one where our posterior is narrower. This should mean lower MDE.
- We train different models with different cities as controls, adding a simulated effect, and we see those models where said effect is outside our "ROPE" threshold. Picking the one which capture the smaller effects.
Based on my experience, a low MDE usually corresponds to a low MAPE or a high R2. With this in mind, optimizing these metrics typically results in a good MDE. When it comes to optimizing R2 or MAPE in our regression, we have multiple options to consider.
Lasso for Control Selection
I have been using Lasso technique to select the best regressors from a pull. Because Lasso regressions tend to shrink the coefficients of less important variables to zero during model fitting, we can use it to measure feature relevance. Additionally, Lasso is easy to implement within the Bayesian Framework.
Here is an example of how we could apply it, using our synthetic control dataset as a demo!
As I've been explaining: let's suppose we aim to conduct our experiment in the 'actuals' geography. By using Lasso approach, we can observe alternative locations that may account for a greater variance in our target, which ultimately contributes to a better counterfactual generation.
import pymc as pm
from sklearn.preprocessing import StandardScaler
df = cp.load_data("sc")
actual_treatment_time = 70
fake_treatment_time = 60
power_df = df[:actual_treatment_time].copy()
X = power_df.drop(columns=['actual', 'counterfactual', 'causal effect']).values
y = power_df['actual'].values
# Standardize the predictors
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Number of predictors
n_predictors = X_scaled.shape[1]
# Hyperparameters
lambda_param = 1 # Regularization parameter, can be optimized or tuned
with pm.Model() as model:
# Define priors for the intercept and coefficients
intercept = pm.Normal('intercept', mu=0, sigma=10)
betas = pm.Laplace('betas', mu=0, b=lambda_param, shape=n_predictors)
# Linear model
mu = intercept + pm.math.dot(X_scaled, betas)
# Likelihood
sigma = pm.HalfNormal('sigma', sigma=1)
likelihood = pm.Normal('y', mu=mu, sigma=sigma, observed=y)
# Posterior sampling
idata = pm.sample(2000, tune=1000, random_seed=42)
# Extracting the beta coefficients
betas_trace = idata.posterior['betas']
# Mean of the betas across samples
betas_mean = betas_trace.mean(dim=['chain', 'draw']).values
# Determine which coefficients are significantly non-zero
credible_intervals = az.hdi(idata, hdi_prob=0.95)
# Extract the 'betas' variable from the credible intervals dataset
betas_credible_intervals = credible_intervals['betas']
# Loop through each beta coefficient's indices to check their credible intervals
selected_columns = []
columns = df.drop(columns=['actual', 'counterfactual', 'causal effect']).columns
for idx, column in enumerate(columns):
lower, upper = betas_credible_intervals.sel(betas_dim_0=idx).values
if not (lower <= 0 <= upper):
selected_columns.append(column)
print(f"Selected Columns: {selected_columns}")With this application, we can determine the best geos to estimate our target (actuals). Those are a, c, and f. This means that these geographies would be the best to leave as a control, if we need to intervene in 'actuals'
Does this really work?
- If we compare the fit of the model, before the intervention, using all the regressors or using only those three, we can see that the estimate is almost the same. Consequently the power should be almost equal.
Using a, c, and f only:
Using all:
- Additionally, you can see that the amount of effect detected with only these three regressors only is almost the same as that detected when using all the regressors.
Conclusion
We can create a method where the user defines only the geos of the dataset where they are interested to intervine. intervention_geos=[ a,b,c, . . .]. Then we execute the lasso selection on each one, excluding the geos of interest from the pull as well. As a result, the user will know which regions to leave in test, if he wants to do a test in regions A, B, C.
By doing this we could:
- Simplify the process, we do not need to define a scoring function, we can use more traditional and sufficiently effective methods.
- We make code execution more efficient, because we would need fewer loops.
- We add clarity to the problem statement around the experiments.
This is just my opinion (Happy to be challenge), and a little of context on how I have approached the problem, and the issues I've experienced when using the models from Meta. But I would love to hear them and know if I'm crazy, or maybe there is some sense! 🔥
drbenvincent
In the context of geo testing... Let's say we have historical data of some KPI (such as sales, or customer sign ups) across multiple geographies. And we are considering running some geo testing on a single (or multiple) regions. That is, we are planning on an intervention in one (or multiple) geographical regions. The question at the top of our minds is "what region (or regions) should we select to receive the intervention?"
This question has been tackled in this GeoLift docs page, and uses power analysis as the basic approach.
We can formalise this a bit as follows: How to best select
ntest regions out of the total pool of all regions?The general approach we can use here is to iterate through a list of all possible valid test regions. For each case, we can use some kind of scoring function. We can then evaluate the scoring function for each possible valid set of test region(s), then pick the best. So something along the lines of:
That will give us a list which looks like this:
Let's continue with our approach:
So the question is, what happens in
scoring_function? Well that depends on what we want to do, but it will be along the lines of:Note: Running synthetic control with multiple treatment regions would benefit from having addressed https://github.com/pymc-labs/CausalPy/issues/320 first.
All of this should be wrapped into a function (or class) called something like
FindBestTestRegionswith a signature of something like:Input:
data: the raw panel dataregion_list: list of all regions/geo'sn_regions: list of number of regions we want to consider testing in, e.g. [1, 2, 3]scoring_function: either an actual function, or a string to dispatch to a specific function that determines the scoring method we are using.modelandexperiment: although these might be pre-defined and not necessary as inputs.Output:
results: a dataframe where each row represents a different combination of test regions. There will be a score column which is the result of the scoring function. This will be ordered from best to worst in terms of the scoring function.