Using MSE over Log Loss Probability to Produce Learning Curves for DeepAR Model

[jfrank94]

I'm curious to know why by default the DeepAR model uses log loss (w/ activation regularization) according to these two links: here & here. From my current understanding of time series forecasting, MSE seems to be the loss function of choice when producing learning curves for the training and validation set and RMSE used for prediction/evaluation. Is there any way to use MSE during training with the DeepAR model?

[StatMixedML]

As the name

GluonTS - Probabilistic Time Series Modeling in Python

suggests, gluonts aims at producing probabilistic forecasts, rather than point forecasts. Hence, the default in DeepAR is a distribution argument that allows you to choose from which forecasted distribution the samples are drawn, from which you can calculate quantiles, mean etc. Hence, I don't believe it is desirable to reduce the functionality of the model to produce point forecasts, since this is implicit in probabilistic forecasts. Say you specify a GaussianOutput, what the model is forecasting is not a mean or median, but it is the distributional parameters directly (which are mean and variance for GaussianOutput). This allows you to construct intervals via quantiles and also means.

If, however, you want to train using MSE, you can set the variance parameter to a constant (say 1) and then only train the mean.

[jfrank94]

@StatMixedML Thanks for that explanation. I can basically choose a distribution that produces the mean via the distr_output parameter, but I don't see anything that pertains to the variance in the estimator.

[StatMixedML]

@jfrank94 There is no explicit argument in the estimator that allows you to do that. Depending on which distr_output you choose, e.g., GaussianOutput(), you would need to create your own, e.g., GaussianOutputMean() by modifying the respective arguments here https://github.com/awslabs/gluon-ts/blob/master/src/gluonts/mx/distribution/gaussian.py.

However, an easier way would be to use GaussianOutput() and then use make_evaluation_predictions since this allows you to evaluate the forecasts with many measures, among with you also find the mse https://github.com/awslabs/gluon-ts/blob/04c838de6d273a0b69c699ebb8f7499cf30e107c/src/gluonts/evaluation/metrics.py#L52

[bisht-vivek]

If, however, you want to train using MSE, you can set the variance parameter to a constant (say 1) and then only train the mean.

How to set the variance parameter while chosing the distribution, can you give one example of that