search

facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.13k stars 334 forks source link

URGENT - robyn_recreate() with JSON produces different model #862

Open bart-vanvlerken opened 10 months ago

bart-vanvlerken commented 10 months ago

Hi everyone, I have quite an urgent issue I would like your help with. I exported a JSON of a model I selected with a client. The JSON itself is fine, as you can see below (I included the first 5 variables for illustration).

image

When running robyn_recreate(), I produce a completely different model (see below). I would really appreciate your help on this!

10_270_5

bart-vanvlerken commented 10 months ago

I tested robyn_recreate() for another client. Results are again different, but this time not by much. Here is the original model's ROI:

image

Here is the recreated model's ROI:

image

gufengzhou commented 10 months ago

sorry for the late reply. There's another thread/issue looking into this too, where we've fixed couple of bugs to close this gap. I just tested it and it looks good. Can you pls update to the latest github version and rebuild/retry?

bart-vanvlerken commented 10 months ago

I'm using the latest version and the issue still persists unfortunately, shall I send you the JSON + data for debugging?

gufengzhou commented 10 months ago

Yes please, gufeng@meta.com

bart-vanvlerken commented 10 months ago

UPDATE: I found the cause of the biggest discrepancy (I forgot that I hard-coded the covid variable within InputCollect$dt_mod before running the model). The results are looking better now, but there is still a discrepancy (the output of robyn_recreate still differs from the original). Here are the ROI figures side by side:

ORIGINAL image

RECREATED image

bart-vanvlerken commented 9 months ago

Dear @gufengzhou, have you managed to find a cause of the issue? I appreciate the effort!

gufengzhou commented 9 months ago

Hey sorry it's been quite busy. I found out that the recreated model has hyperparameters rounded to 4 digits, while the initial one doesn't. But still need to figure out where the rounding is from. will update here once we made progress. @laresbernardo FYI