awslabs / gluonts

Probabilistic time series modeling in Python
https://ts.gluon.ai
Apache License 2.0
4.44k stars 742 forks source link

Fix N-BEATS number of prediction samples to 1 #1594

Open borchero opened 3 years ago

borchero commented 3 years ago

Description

When using gluonts.evaluation.backtest.make_evaluation_predictions with a predictor obtained from NBEATSEstimator, forecasts of type SampleForecast are returned (with as many samples as specified via num_samples, which defaults to 100).

For N-BEATS, this results in num_samples samples from the network. Since N-BEATS inference is non-probabilistic, however, this only duplicates work and makes predictions num_samples times slower than they should be.

To mitigate this issue, the predictor obtained from NBEATSEstimator should fix the number of samples to 1.

lostella commented 3 years ago

One idea to address this: make num_samples default to None in make_evaluation_predictions, which results in num_samples=None being passed to Predictor.predict, in which case the model produces its default number of samples

borchero commented 3 years ago

Since multiple samples are essentially useless, I would rather adapt the strategy of the MQCNNEstimator, for example. Since it directly outputs quantiles, samples are not required and warning is logged that num_samples is unused. We have a very similar situation for N-BEATS -- only that we get point forecasts.

lostella commented 3 years ago

@borchero MQCNN predictors do that because they rely on the QuantileForecastGenerator. Depending on what loss N-BEATS optimizes, outputting a QuantileForecast object may or may not be a good idea: if MAPE is being optimized, then outputting the P50 prediction makes sense. If other losses are used (MASE and sMAPE are allowed if I recall correctly) then the prediction does not represent the median or any other quantile, strictly speaking, so using QuantileForecast may be misleading.

borchero commented 3 years ago

No, of course. It might return a PointForecast though (which might be a SampleForecast with a single sample).