search

facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.16k stars 346 forks source link

Areas for Improvement When Using Robyn #1098

Open ghltk opened 4 weeks ago

ghltk commented 4 weeks ago

Hello :) I have been using Robyn quite effectively, but I have noticed a few areas for improvement that I would like to share.

  1. Issue with Budget Allocator Onepager Results:

    • The current values focus solely on paid media, leading to a significant gap with actual sales (Nonpaid + Paid). When collaborating with marketing practitioners, I can only explain that "The total response value is not the expected actual sales, as the impact of Non_paid_media is not considered, so the real value may be higher or lower." This explanation is not intuitive and is difficult to justify.

  2. Baseline Interpretation Issue

    • When defining Sales = Paid Media + Non Paid Media(=baseline), it becomes challenging to interpret if the baseline is negative. For example, when Sales is 1,000,000, Robyn may express this as 2,500,000 (Paid Media) - 1,500,000 (Non Paid Media). It would be beneficial to ensure that Non Paid Media does not become negative.

  3. Lack of Forecasting Features

    • There is a need for intuitive functionality, such as estimating "What will the expected sales be if the budget allocation strategy is used for 3 months?" I understand that Recast's MMM already includes this feature.

  4. Lack of Validation Functionality

    • To continually trust the model, tracking the error rate over time for previously used models is essential. A method to visually compare the total response value from the Budget allocator with the actual values after a simulation period using a time-series graph would be helpful. Additionally, separating validation for NonPaid Media and Paid Media could be beneficial. Although I understand that predicting NonPaid Media is challenging, having a rough estimate would enhance intuitive expression.

  5. Urgent Need for Refresh Performance Improvement

    • I wish to refresh the model weekly to observe changes in channel contributions, but the refresh process is unstable. For instance, if the Initial Model shows a 50% contribution from Campaign A, the Refreshed Model may drastically decrease to 20% one week later. The inconsistency and instability make it challenging to use the model continuously. Simply selecting the model with the lowest decomp value as the final model does not resolve the issue.

I would like to know if any of the issues mentioned above are currently being developed. I am also curious about the scheduled updates. Please let me know if any part is unclear. Thank you :)