Deterministic evaluation of probabilistic models (e.g. nbeats)

[mrtn37]

When predicting a time series, one can turn some models probabilistic by defining a likelihood parameter. E.g.: NBEATSModel( ... likelihood=GaussianLikelihood(), random_state=42 )

If done this way, the prediction is not deterministic any more. Two subsequent calls to predict(series=.., n=1, num_samples=1) yield different results.

For developing purposes, it would be nice to be able to fix the random state of the likelihood as well so that, independent of the num_samples defined, the result of the prediction always stays the same.

[dennisbader]

Hi @mrtn37 and thanks for writing. The model predictions are reproducible depending on the order of calls.

• If you create two models with identical parameters (and same random state) and train & predict, you will get the same results (assuming that both models called predict() the same number of times)
• If you save a model and load it, it will give identical predictions as the original model (assuming that both objects called predict() the same number of times).

The model's internal random state changes based on the number of times you called methods with the @random_method decorator. It is intended that two consecutive predict calls give different results when probabilistic.

What you suggest is adding an optional parameter to predict() to enforce that the random_state remains unchanged and therefore gives reproducible results upon successive calls?

[mrtn37]

Thank you for the explanation. That makes sense :) Even though: keeping in mind how often predict was called, resp. saving copies of the model might be a bit cumbersome during development.

What you suggest is adding an optional parameter to predict() to enforce that the random_state remains unchanged and therefore gives reproducible results upon successive calls?

Yes, that sounds like a good proposal. Just to clarify: the parameter would (re-)set the random_state to a certain value, such that two consequtive calls: predict(series=.., n=, num_samples=, random_state=) would produce the same two nummerical arrays.