Evaluation on test set

[milosz-twg]

Hello Robyn team!

Thank you very much for all your efforts. We have been working to implement Robyn-based solutions for a few of our agency clients, and we are a little stuck on the evaluation. Having looked through issues #65, #148 and #158 we've gathered that there is currently no direct way of using an existing Robyn model to make a prediction on an unseen dataset. In #148 we see the mention of constructing custom predictions from the model coefficients. We tried to go this route but weren't able to reproduce the components yet and were hoping that you might give us a hint. Specifically, this is what we are trying:

• Using the input data (the same dataset that comes in as _'dtinput') along with prophet decomposition of the target variable, parameters found in _paretohyperparameters.csv and coefficients found in _paretoaggregated.csv, obtain the values found in _pareto_alldecompmatrix.csv
• The reasoning is that the values in _pareto_alldecompmatrix.csv add up to the prediction (depVarHat), so if we can find the procedure of turning inputs into these values, we can essentially do the 'predict' operation on any dataset. Presumably we need something along the lines of the full ridge regression function. Are we on the right track here?
• Unfortunately so far we weren't able to put the whole thing together in the right way
• We're on Robyn 3.0

Do you think you're able to give us some direction/course-correct us on this?

Thanks, Twigeo DS team

[gufengzhou]

Hi, thanks for bringing this up. You're definitely on the right track.

• First, regarding reproducing depVarHat, the difficult part is that you need to apply adstock and saturation transformation to input data first, then you can multiply the transformed matrix with each the coefficients to obtain depVarHat.
• Second, when thinking about forecasting next week with a multivariate regression, we need all the next week's values of all predictors (media, prophet_vars etc) to predict next week's revenue. For prophet_vars, you need to conduct the prophet decomposition with all periods again and forecast trend/season etc for next week. For paid and organic media, you need to set couple of scenarios: 1 might be spending the same amount as last week or weeks, 2. might be manually insert media plan for paid and "plan" the organics, 3. might be forecast all of them (using prophet for example) individually and use them to forecast sales, 4. might be using different optimised media mix from allocator.
• Third, uncertainty is not easy to obtain in Robyn because we're neither determining the "significance" of each beta coefficient due to ridge nor doing any Bayesian simulation. Instead, we have to be creative. One idea could be using the coefficient estimates from the latter part of the hyperparameter iterations (e.g. the final 10%, because they're most optimised) and produce the lower and upper bounds based on those. This can be observed in the newly added roas convergence plot.

As you can see, we do consider building the forecast function too. Due to complexity and resource constraints, we can't promise any release dates at the moment though.

[gufengzhou]

feel free to reopen if necessary