search

facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.13k stars 336 forks source link

Model Refresh #812

Open esanchezgarcia opened 1 year ago

esanchezgarcia commented 1 year ago

Hello to all,

I am working with Model Refresh and I have some doubts. My base model contains 203 weeks and I have run the Model Refresh including 15 new weeks.

What surprised me is that the output of the Model Refresh that reflects the 218 total weeks does not keep the coefficients of the variables of my initial model. In addition to observing a very relevant change in terms of ROIs, although this may be motivated by the level of investment, I do not understand how 15 weeks can weigh so much with respect to 203 weeks.

Therefore, why are there such differences between the initial model and a model refresh with 7% more observations than the initial one?

Many thanks in advance !!

gufengzhou commented 1 year ago

Hi, in general the refresh function is still not very well studied yet and we're actually seeing surge of usage just very recently. In this sense, sorry for the unsatisfying result. The refresh is actually remodelling using a proportionally narrower range of the initial hyperparameters. It's designed to maintain more stability for baseline variables while reflecting the changes of investment in the additional weeks. So if certain channels have seen strong spending differences in the 15 weeks, it's expected to have bit more change. I can't make much judgement without looking into your data. You can try increasing the refresh iterations and trials to achieve better convergence. Let me know if it helps.

I'm also considering ways to improve the stability of refresh, for example by putting more weight on the decomp convergence. However it's very difficult for us to do it "blindly" because we don't have real dataset. So this might take a bit.

esanchezgarcia commented 1 year ago

First of all, thank you very much for your reply.

There are no major changes in investments in these new weeks. But anyway, what should we expect when there is a dramatic change in investments? Both when there is an increase and when there is a decrease. Should we expect an overestimation, in which case?

Many thanks in advance !!