search

pymc-labs / pymc-marketing

Bayesian marketing toolbox in PyMC. Media Mix (MMM), customer lifetime value (CLV), buy-till-you-die (BTYD) models and more.
https://www.pymc-marketing.io/
Apache License 2.0
648 stars 175 forks source link

Flaky fixture causing *occasional* test failure #639

Open ulfaslak opened 4 months ago

ulfaslak commented 4 months ago

So I noticed this while working on #628.

I was getting a failing test here. An assertion in test_new_data_predict_method raised an exception. It compares the mean of a fixture toy_y with the 2.5% quantile of the posterior predictive mean, and it failed. I then went to check the fixture toy_y and saw that it is a random integer sequence. Since it is random (with no seed) I just reran the test action and it passed.

That's probably not the behavior we want.

My suggestion is we just seed the RNG so it behaves deterministically, but just want @wd60622 s opinion on this before I make a PR for the change.

wd60622 commented 4 months ago

Definitely think that it could be improved! If you have a suggestion, I'm all ears. The quantile check is arbitrary so feel free to change! I think something else could be much better!

ulfaslak commented 4 months ago

What is the intuition for this check? https://github.com/pymc-labs/pymc-marketing/actions/runs/8734692111/job/23965893602#step:5:736 Maybe we can just change that.

wd60622 commented 4 months ago

Was trying to get at that the predictions (quantiles) are capturing the trend of data (toy_y). Maybe it wouldn't make sense to use man of toy_y. I believe it is fine to be reworked!

wd60622 commented 3 months ago

Same as #516