robyn_refresh produces only models different from the original

angelinaparkina commented 6 months ago

Project Robyn

Describe issue

Hi team Robyn, I have a working model, where some channels were set to 0 by the model. Other models from the same run did set 2 or 1 more channels to 0, which did not seem realistic. So I chose the one that had the minimum amount of channels being set to 0, and most realistic ROIs.

After choosing the best model, I reproduced it multiple times to run the budget allocator. The ROIs from the reproduce function were the same as in the original model.

Now, I would like to refresh the model with 30 new weeks of data (original model has 173 weeks). When I do that with the refresh function, two channels are set to 0, which had an effect in the original model. For a test, I tried to refresh it with only two weeks of data, which produces the same effect - two channels are set to 0 in all the model variants. Since that does not seem realistic from the channels perspective, I would want to get more or less similar results to the original model, at least in the sense that the channels that had an effect before still do, even if we update the model with 2 weeks of data.

A couple of things that could be relevant here:

All the hyperparameters seem more or less the same as in the original model, except for penalty, which cannot be specified (it always runs with the range 0-1). It seems like the optimization process is going in the direction of the neighbour models of the original run (with those channels being 0), and cannot settle on anything similar to the original model. Is there any way to also specify the penalty? Maybe that would help nudge the process in the direction of the chosen model.
Several channels being set to 0 can be explained by multicolinearity between channels, however it is lower than 0.5, so i would not expect for it to have such a big effect.

What other reasons can there be for such a result?

Provide reproducible example

I'm attaching the results of the original model (upper), and the example of a refreshed model (lower) (with 2 weeks).

The code I am using for refresh:

Robyn <- robyn_refresh( json_file = json_file, dt_input = data, dt_holidays = prophet, refresh_steps = 2, refresh_mode = "manual", refresh_iters = 2000, refresh_trials = 3, version_prompt=FALSE, plot_folder = '' ) I'm also including hyperparameters from the refresh + hyperparameters for the original model. hyperparameters_refresh_issue.xlsx

Let me know if anything else is needed to debug.