Ability to exclude locations from control

[adrialuzllompart]

Is your feature request related to a problem? Please describe.

We're trying to use experimental data to validate GeoLift.

We've run a geo-test in which we've assigned half of a country's locations to control and the rest to treatment. Although we have already analysed and arrived at a conclusion using conventional methods, we're now trying to validate GeoLift with this data i.e. the objective is to see the causal effect that GeoLift estimates and how that differs from what we've found using other methods.

While we've been able to tell GeoLift which locations NOT to consider as potential treatments (via the exclude_markets parameter), the issue we're having is that the locations that (1) were actually treated and (2) are not selected as treatments might end up as part of the synthetic control. This is problematic - these locations were treated in reality so if they're not part of the treatment subset suggested by GeoLift, they shouldn't be part of the synthetic control either.

There doesn't seem to be a way to exclude certain locations from the control group. The include_markets and exclude_markets both seem to target the treatment group only.

Describe the solution you'd like

We've tried using the include_markets parameter in combination with the N parameter. Our hope was that we would be able to tell GeoLift to consider (as potential treatments) all locations that were actually treated in reality, but force it to provide a subset of the actual treatment group via the N parameter. We were hoping that this, in turn, would make GeoLift not consider the other treatment locations when building the synthetic control.

However, when trying this, GeoLift raises the following error Error: More forced markets than total test ones. Consider increasing the values of N.

Describe alternatives you've considered

We can't think of any other way of achieving this.

Additional context

I think it should be clear but I'll be happy to provide any further details if you need them.

[NicolasMatrices-v2]

Hi @adrialuzllompart, thanks for your detailed feature request!

I think I understand the problem statement: you would like your treatment group to be a subset of a set of locations that were treated and exclude the remaining locations that were treated from the control donor pool when they are not a part of the treatment group. However, I am missing what would be the use case when designing an experiment, given that at the design stage, you have not applied a treatment to any location.

Both the GeoLiftMarketSelection and GeoLiftPower look to help you design and find the best treatment group for your experiment. If you have already designed your experiment and selected your treatment group, you can:

1. Check how the power curve looks for that treatment group, using the dataset prior to applying the treatment.
2. Run the GeoLift function and check what the estimation of impact looks like.

Let me know if this makes sense to you. Best, Nicolas

[adrialuzllompart]

Hi @NicolasMatrices-v2 - thanks for your reply!

Your understanding of the problem statement is spot on:

I think I understand the problem statement: you would like your treatment group to be a subset of a set of locations that were treated and exclude the remaining locations that were treated from the control donor pool when they are not a part of the treatment group. However, I am missing what would be the use case when designing an experiment, given that at the design stage, you have not applied a treatment to any location.

The issue is with the bit I've highlighted in bold. In this instance we're not trying to design an experiment. The experiment was designed and launched in the past, and we're now trying to use GeoLift to backtest the tool - essentially we're trying to answer: "what would we have observed if we had used GeoLift to design this test?". There are two ways in which we could answer this question:

1. We tell GeoLift which locations were treated, let it build the best synthetic control for those locations, and then see which estimate it comes back with. We have already done this, and the estimated effect is similar to what we have obtained via other methods.
2. We tell GeoLift to select the best subset of M locations for treatment, out of the complete pool of N locations that were actually treated. Then we need GeoLift to find the best synthetic control for this subset (but we need to avoid it using the N-M locations that were actually treated but which weren't selected in the treatment subset). This is what we're trying to do now.

As an example:

• Let's say we ran an experiment in a country that had 100 cities. When we designed the experiment (not using GeoLift), we decided to randomly treat N=50 cities, and use the other 50 cities as the control group.
• The experiment ended and we estimated the causal effect of our intervention via conventional methods (e.g. a t-test).
• Now, since we're planning to use GeoLift for future experiments, we'd like to backtest it with this past experiment to check that it produces an answer that is similar, or at least compatible, with the more conventional methods the business is used to.
• In order to do this, we want to tell GeoLift to select the best treatment group of M≤50 cities out of the real treatment group of N=50 cities. Ultimately, our hope is that GeoLift allows us to get a similar answer while treating fewer cities - this would be very appealing from a business perspective since it would reduce the cost of running an experiment.
• Then, we want GeoLift to find the best synthetic control for the M-city treatment group. The constraint here is that GeoLift doesn't use any of the cities in N that are not in M. This is because these cities were actually treated so it wouldn't make sense to have them as part of the synthetic control.
• Finally, we want GeoLift to estimate the causal effect of our intervention by comparing the counterfactual (as estimated by the synthetic control) with the M-city treatment group.

At the moment, we're having issues with the second-to-last point as we're unable to apply that constraint.

Please let me know if you have any further questions.

Thanks!

[SG-GM]

Hi @adrialuzllompart were you able to solve this issue. I am thinking of running a similar test and was wondering if you were able to solve this.

[SG-GM]

Thanks @adrialuzllompart how did you set up this part, I mean how did you specify treatment group based on all the treatment locations that the test was run in : We tell GeoLift which locations were treated, let it build the best synthetic control for those locations, and then see which estimate it comes back with. We have already done this, and the estimated effect is similar to what we have obtained via other methods.

[Th3Bust3r]

@adrialuzllompart looks like this has been pretty quiet for several months, but I have done exactly this exercise, so hopefully I can contribute something here that will help you and others.

To outline my scenario: I ran a test using a different geo-based methodology. I ran ads in a random 25% of the US, while using the unexposed 75% as the control. Several months later, as I began to explore GeoLift, I reevaluated the old test by simply passing the data and actual test/control assignments into the functions. GeoLift's estimate was very similar to what the previous method had produced, but we wanted to know if we might have selected fewer markets, and thus spent less money on the test if we had used GeoLift to design it.

The way to accomplish this boils down to passing all of the control geos into a list in the exclude_markets parameter. Do not put anything in the include_markets parameter, since you want the function to select the markets it wants from your test market pool. From there I just set holdout = c(0.75, 1), although that may not be needed since it's already forced to be true based on the markets it has left to choose from. Similar with N - you could give it a number or a range, as long as its less than the number of test markets available to choose from. Setting values may make the function run faster, but you should be able to just leave them out and let the function decide.

Once you have a sudo market selection, you would then need to create another copy of your dataset with the rest of the test markets removed in order to evaluate the observed lift in your test period for only the subset of test markets you just chose.

I should note, this currently would only work for a single cell design, since there is no exclusion parameter yet for multicell. In any case, hopefully this makes sense and is helpful.

[raphaeltamaki]

@adrialuzllompart Hi, I'm coming quite late to the party, but let me try to give some guidance. As per you example:

• Let's say we ran an experiment in a country that had 100 cities. When we designed the experiment (not using GeoLift), we decided to randomly treat N=50 cities, and use the other 50 cities as the control group.
• The experiment ended and we estimated the causal effect of our intervention via conventional methods (e.g. a t-test).
• Now, since we're planning to use GeoLift for future experiments, we'd like to backtest it with this past experiment to check that it produces an answer that is similar, or at least compatible, with the more conventional methods the business is used to.
• In order to do this, we want to tell GeoLift to select the best treatment group of M≤50 cities out of the real treatment group of N=50 cities. Ultimately, our hope is that GeoLift allows us to get a similar answer while treating fewer cities - this would be very appealing from a business perspective since it would reduce the cost of running an experiment.
• Then, we want GeoLift to find the best synthetic control for the M-city treatment group. The constraint here is that GeoLift doesn't use any of the cities in N that are not in M. This is because these cities were actually treated so it wouldn't make sense to have them as part of the synthetic control.
• Finally, we want GeoLift to estimate the causal effect of our intervention by comparing the counterfactual (as estimated by the synthetic control) with the M-city treatment group.

What you can do is: 1) using the Pre-Experiment Data you have we have two group of locations $Locations{Treated}$, where the treatment was applied, and $Locations{Control}$, where no treatment was applied 2) you run the GeoLiftMarketSelection and GeoLiftPower while using in the input exclude_markets the countries in $Locations{Treated}$ to obtain the list of optimal locations to conduct the experiment, as well as their statistical power 3) From the list of optimal locations, you select one group of location $Locations{recommended}$ 4) Now you load the Post-Experiment Data, which contains locations from $Locations{Treated}$ and $Locations{Control}$ 5) You want to filter this dataset to remove any location in $Locations{Treated}$ that are not in $Locations{recommended}$. To do this, you can apply the following filter: PostExperimentData = PostExperimentData %>% filter((location %in% locations_recommened || location %in% locations_control) ) (requires that you have dplyr, which you should as it is a required package) 6) Then you can GeoLift with locations as locations_recommened (i.e. $Locations_{recommended}$)

You can repeat steps 4~6 for any group of locations within $Locations_{Treated}$ and it will provide you with the estimated result. One dimensions you should be careful is that the money invested for the sub-group of locations may not match the recommended investment from GeoLift, thus you would still have some uncertainty regarding the hypothetical scenario of "What would have happened if we used GeoLift recommendation to design the experiment", as the group of locations is just one of the dimensions on the design of an experiment

[JussanN]

Hi everyone, I'm closing this task as it seems to be solved. Please reopen if the issue persists. Thank you.