search

facebookincubator / GeoLift

GeoLift is an end-to-end geo-experimental methodology based on Synthetic Control Methods used to measure the true incremental effect (Lift) of ad campaign.
https://facebookincubator.github.io/GeoLift/
MIT License
175 stars 54 forks source link

negative average_MDE for positive EffectSize in GeoLiftMarketSelection() #176

Closed vpatcera closed 4 months ago

vpatcera commented 8 months ago

Hi I want to understand - is it OK to get negative average_MDE values from GeoLiftMarketSelection() for positive EffectSize? If yes - what does it actually mean?

This is what I got for my sample data:

ID Location Duration EffectSize Power AvgScaledL2Imbalance Investment AvgATT Average_MDE
1 market1 15 0.25 1 0.9307789 4614.375 21.429641 0.26419807
2 market2 15 0.10 1 0.6999374 299.250 -2.473729 -0.07795269
3 market3 15 0.25 1 0.9676958 5476.875 20.864286 0.34615385
4 market4 10 0.40 1 0.7422721 984.000 -2.621508 -0.05400550
JussanN commented 4 months ago

Hi @vpatcera I believe this is related to the high imbalance/poor fit between test and synthetic control. You can see that your AvgScaledImbalance is very high (closer to 0, better). I also recommend you to review al the details of the https://facebookincubator.github.io/GeoLift/docs/GettingStarted/Walkthrough/ to help you to design a good GeoLift experiment. I'm closing this issue for now. Thank you.