Feature Request: Provide limit for number of regions in control markets

[michaelwexler]

Is your feature request related to a problem? Please describe.

The GeoLift tool considers all regions to be available as a control outside of the forced test regions and any exclusions we set. However, there can be a cost to tracking all US DMAs, for example, or there can be a need to run multiple tests at one time, so we want to limit the regions that need to be left alone. There is no easy way to do this in the current implementation.

Describe the solution you'd like

A solve would be to provide a parameter giving an upper bound on the number of regions to include in the control.

Describe alternatives you've considered

An alternative is to provide helper functions that, for the best solution, provide truncated output that orders by weight and keeps only the regions within the upper bound count. That function would then rerun, say, Market Selection to provide the new fit with the reduced set.

Additional context

The current approach assumes no cost to collecting control data, and also assumes that all regions in the set are eligible to be used as controls (except for the filtering parameters mentioned previously. While we do want all regions considered, we may have a constraint such that we really want to know out of all the regions, what's the best, say, 10 region, best 9 region, on down solution that also meets the needs for power, etc. Running any test that meets our power requirements but has fewer cities involved wins on parsimony, though a small number of control regions may either be difficult to match or have higher probability that a rare event in one region has higher impact in your analysis. Still, having that upper bound as a parameter allows more data folks to take advantage of the tool to solve resource-constrained tests: I want to consider every DMA in the US to find the best combo for my test... but I can only keep 10 as controls.

[raphaeltamaki]

Hi @michaelwexler thank you for your time in writing this and your engagement

I'm not sure I understood your request perfectly. My understanding is that you need to find the best combination of up to X DMAs (with X being an arbitrary number) that gives you the highest expected power?

If so, with the functionGeoLiftMarketSelection you are able to define with parameter N the different number of locations to consider for selecting the DMAs that will receive the treatment. By providing a list of number from 1 to X, you will get all the possible combinations of 1, 2, 3,..., X locations and the estimated power, thus satisfying your need (if this is indeed your request)

I will close this thread for now, but if my answer doesn't solve your problem or I missunderstood the problem entirely, please, do reopen this issue.

[michaelwexler]

Thank you, but this isn't quite it. You are talking about how many regions to include in the treatment group. I am asking for the same capability to limit the control group, aka the "holdout" group. That is, it's difficult to answer the simple question of "Out of all US DMAs, give me some options of a single treatment region vs. controls of no more than 3 regions". Instead, GeoLiftMarketSelection() always assumes that, other than named regions to exclude, all submitted DMAs should be added to the control calculation if it adds power.

As I stated, in some cases, there is a cost to measuring a control region. Therefore, rather than looking for the unfettered best match among all submitted DMAs, I want to see the best options of a constrained set. I do want all regions available, so I'd keep all DMAs in the submitted data; the suggestions of "just submit fewer DMAs" assumes I know which DMAs are the best to use as tests and controls.

The holdout parameter is a good start, but it refers to the %age of conversions, not actual count of regions. If I had to guess, I suspect the authors wanted to make an allegory to the RCT concept of 2 condition split percentage, but that's only one aspect of how folks think about wide-region holdouts. Since each region we hold out can be a large opportunity cost (a region is a big unit!), having the ability to simply cap this is helpful.

With this additional context, is there a way to workaround this through clever calls to GeoLiftMarketSelection()?

Due to time constraints, we did a brute force cycling of excluding all but 4 regions, getting best option, and then looping again, eventually permuting through all DMAs. I suspect there is a better approach.

[raphaeltamaki]

Ah, now I understand, @michaelwexler , thank you for the clarification. You are right, there is currently no currently capability to apply that same limitation to the control group. The closest thing to what you described would be a hack using the holdout parameter as you mentioned, but normalizing the number of conversions so that each location/DMA appears to have the same number of conversions (but keeping the daily variance). With this, you can use the holdout to specify the share of regions to use, and if you want a absolute number, then you need to find the share where holdout = limit_number_region / distinct_regions

Still, this would mean that you will not have the benefit of having an estimate based on the number of conversions as a consequence. We will add it to the roadmap of features to add, but it will take some time until this get deployed, as we have restricted resources allocated to GeoLift for now.