search

facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.08k stars 322 forks source link

Budget Allocator Question #809

Open bart-vanvlerken opened 10 months ago

bart-vanvlerken commented 10 months ago

Hi everyone, my model one-pager displays the following values:

image

However, when I run the budget allocator on the same date range, it gives the following values:

image

As you can see, the effect share differs between the two. Does anyone know what can be the cause of this?

gufengzhou commented 9 months ago

what is your date range in the allocation? the first plot/ model onepager is default to the total modelling window.

bart-vanvlerken commented 9 months ago

@gufengzhou I set date_range = "all", so I would expect the effect shares to be the same.

gufengzhou commented 9 months ago

hm that's weird. just to be sure, you're comparing the same model ID right? if yes, I'd need your data (unfortunately) to debug.

bart-vanvlerken commented 9 months ago

@gufengzhou Yes, I am using the same model ID. Is there a way to share it with you privately?

gufengzhou commented 9 months ago

gufeng@meta.com. I'd need the dataset and your model specific json file.

bart-vanvlerken commented 9 months ago

@gufengzhou I sent it over, let me know what you think.

gufengzhou commented 9 months ago

Hey, thank you very much for raising the issue. There's indeed a bug when having date_range = "all" and I just fixed it. However, there's an inherent difference between the allocator and initial modelling which is explained in detail in this issue when reproducible validation script. I hope the answer makes sense to you.

bart-vanvlerken commented 9 months ago

@gufengzhou Thanks for the update! I understand now how the allocator operates in allocating effect share, but I must say it doesn't make a lot of sense and it is difficult for me to communicate this to my clients. How can the simulated effect share be so different from the historical response, wouldn't it make more sense when these values are the same when using the same date range?

Of course I can tweak the date range to produce 'desirable' results, but this doesn't feel right. In addition, most clients would like to include all marketing channels in the allocator which often forces me to use a large date range since some channels only ran campaigns during a certain period. Here is the model one-pager and allocator one-pager of another client, and I'm running into the same issue. I really appreciate all the work you're putting into this package, and I'd love to hear your opinion on this. 2_2082_3 2_2082_3_reallocated_best_cpa

gufengzhou commented 9 months ago

ok this definitely doesn't look right. I assume the high carryover from weibull is the reason. Would you mind sharing the dataset as well as this specific model json file with me? I can try to debug. gufeng@meta.com

iamlostcoast commented 7 months ago

I'm pretty sure I am experiencing this exact issue. Was there ever a resolution to this?

I'm not sure if there is a related underlying issue, but I was also finding that robyn_response() does not give results that match the model one-pager when using a weibull PDF. It does seem like the response curves that function produces could explain the unreasonable budget allocator results.