search

facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.08k stars 322 forks source link

robyn_write() model performance metrics show NRMSE for the validation set not the test set. Is it normal? #823

Closed SeanRichterWalsh closed 9 months ago

SeanRichterWalsh commented 9 months ago

Project Robyn

Describe issue

In the model exported when using robyn_write(), the performance metric for NRMSE is on the validation set not the test set. But adjusted R2 is reported for the test set. Why is the NRMSE of the test set not reported?

Here are the modelling metrics from an example one-pager:

Screenshot 2023-09-22 at 09 48 04

And here is the same model when brought into R via robyn_write():

Screenshot 2023-09-22 at 09 48 28

Thank you.

gufengzhou commented 9 months ago

Yes, because when ts_validation = TRUE, the objective function to be optimised is nrmse_val. When whents_validation = FALSE, it's nrmse_test nrmse_train

SeanRichterWalsh commented 9 months ago

@gufengzhou thanks for replying. I am still a bit confused though because when ts_validation = FALSEthere is no test set, right? Is it that NRMSE on the full (i.e. training) data is optimised for? Thanks.

gufengzhou commented 9 months ago

Right sorry that was a typo. Just corrected it. It's train

SeanRichterWalsh commented 9 months ago

Clear. Thanks a lot.