google / matched_markets

Matched Markets is a Python library for design and analysis of Geo experiments using Matched Markets and Time Based Regression.
Apache License 2.0
71 stars 20 forks source link

TypeError in step 3 #3

Open tomasvdb opened 1 year ago

tomasvdb commented 1 year ago

in step 3: Summary of the possible designs, if/when there are no suitable designs (e.g. if budget too low, time period too long, etc.. ) the code returns a type error. Perhaps a more descriptive error message would be helpful.

TypeError Traceback (most recent call last) in 6 matched_designs = MMclass.exhaustive_search() 7 else: ----> 8 matched_designs = MMclass.greedy_search() 9 10 if len(matched_designs) == 0:

1 frames /usr/local/lib/python3.8/dist-packages/matched_markets/methodology/tbrmmscore.py in score(self) 87 if self._score is None: 88 self._score = Scoring( ---> 89 int(self.diag.corr_test), int(self.diag.aatest.test_ok), 90 int(self.diag.bbtest.test_ok), int(self.diag.dwtest.test_ok), 91 round(self.diag.corr, 2), 1 / self.diag.required_impact)

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

titubs commented 7 months ago

@tomasvdb I got the SAME error and I have been pulling out my hair....can someone please help!

tomasvdb commented 7 months ago

In my case, what caused the issue is that I didn’t have enough data for the algorithm to come up with a a split .Try to increase the number of you geos or collect more data On 8 Mar 2024, at 06:40, tipatubi @.***> wrote: @tomasvdb I got the SAME error and I have been pulling out my hair....can someone please help!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

titubs commented 7 months ago

@tomasvdb Hey Tomas! This is an amazing break through. I wish they would have designed these error messages more descriptive as you said. Thank you so much. I will try that tomorrow!

titubs commented 7 months ago

@tomasvdb I have been trying with this shape and still get same error:

What is the size shape that worked for you? Thanks!

tomasvdb commented 7 months ago

That’s still not enough data , especially if you also set the required sensitivity of the experiment to be high or if the variance in the data is high.The underlying algorithm is trying to find groups of geos that behave in similar ways. So in theory if your data is highly correlated you get away with using less data, but typically, I use 3 months of daily data for each geo If in doubt about the cause of the error, you can use some fake sample data and test with that On 8 Mar 2024, at 20:59, tipatubi @.***> wrote: @tomasvdb I have been trying with this shape and still get same error:

14 geos 21 date units per geo total 294 total rows

What is the size shape that worked for you? Thanks!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

titubs commented 7 months ago

@tomasvdb Thank you for the tip and yes, I just tried feeding it random fake data and it worked actually!

The only thing that is very unintuitive to me and wanted to see what you thought is that the code to get matched markets applies DURING the planning of the experiment, so I dont know why they require the feature for historical "cost". In my case, I am trying to find similar markets based on the response (i.e. visits to the app). How do you interpret the use of the "cost" in the input to derive matched pairs?

Lastly, in which cell of the NB do they print out the actual pairs?

tomasvdb commented 7 months ago

It’s a required field but for historical data you can just make it zero On 9 Mar 2024, at 07:31, tipatubi @.***> wrote: @tomasvdb Thank you for the tip and yes, I just tried feeding it random fake data and it worked actually! The only thing that is very unintuitive to me and wanted to see what you thought is that the code to get matched markets applies DURING the planning of the experiment, so I dont know why they require the feature for historical "cost". In my case, I am trying to find similar markets based on the response (i.e. visits to the app). How do you interpret the use of the "cost" in the input to derive matched pairs?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

titubs commented 7 months ago

@tomasvdb Thank you so much.

As far the output goes, does Control in the output mean a geo that is NOT exposed (i.e. HOLDOUT) vs Treatment is a geo that is Exposed to the campaign, right?

tomasvdb commented 7 months ago

It gets exported to a Google sheet , just look in your recent Google sheets On 9 Mar 2024, at 22:12, tipatubi @.***> wrote: @tomasvdb Thank you so much. Do you know in which part of the notebook they print out the actual geo matches (pairs)? i do not see it on my end... :(

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>