What is the proper format for generating a cold start forecast?

[SkanderHn]

I'm trying to test cold start forecasting using the DeepAREstimator. My test setup is the following:

• Generated 4 categories of seasonal weekly time series using sine waves with added gaussian noise: Short winter times series (that last for only one winter / 13 weeks), short summer time series (that last for only one summer / 13 weeks), and long winter seasonal time series (that span 2 or more years) and long summer time series (that also span 2 or more years). Add a simple list of categorical features [0,1,2,3] to designate each type of series.
• Splint into 3 years train data and 2 years test data.
• Train a DeepAR model using the following call: estimator = DeepAREstimator(freq="W", prediction_length=13, context_length = 52, trainer=Trainer(epochs=50), use_feat_static_cat=True, cardinality=[4])

Try to generate a forecast by feeding the predictor data sets that have categorical features and start dates in the future (relative to the training set) but empty target arrays, something along this pattern:

{'target': array([], dtype=float32), 'feat_static_cat': array([3], dtype=int32), 'start': Timestamp('2017-12-31 00:00:00', freq='W-SUN'), 'source': SourceContext(source='list_data', row=1)},

The results are always zero or near zero. Using the estimator's default settings, the median forecast is close to zero, but the quantiles fluctuate a little.

The forecast look flat when compared to the actuals, so I assumed a bug in my forecast plotting code:

But if we zoom in, we notice very small prediction intervals (meaning that there isn't some bug in my plotting code, just a very flat forecast):

The forecast intervals get smaller and smaller if I increase some of the Hyperparameters (number of epochs, number of neurons, etc...).

I don't know if I am doing something wrong when trying to generate a cold start forecast, or if it is just a bad model / bad forecast outcome?

I know cold starts are going to be noise, but in this case (sine waves + slight gaussian noise) it seems like DeepAR should be able to figure it out easily, with minimal tweaking. Am I missing something?

I also get similar results using TransformerEstimator.

[vafl]

Can you attach an example notebook for this? Thanks

[SkanderHn]

@vafl Here is an example notebook with csv's of the synthetic data I used. DeepARExample.zip

[vafl]

The below settings seem to work.

The important changes are scaling=False and context_length=1. Using negative binomial is not necessary, but perhaps a better choice if you are dealing with integer values.

from gluonts.distribution import NegativeBinomialOutput

estimator = DeepAREstimator(
    freq="W",
    prediction_length=13,
    context_length=1,
    num_layers=2,
    num_cells=40,
    distr_output=NegativeBinomialOutput(),
    scaling=False,
    trainer=Trainer(epochs=20, learning_rate=0.01),
    use_feat_static_cat=True,
    cardinality=[4]
)
predictor = estimator.train(training_data=train_data)

[SkanderHn]

@vafl Thanks - this does work a lot better, and solves my immediate problem.

However, it seems I am misunderstanding how the context_length works then - I assumed the larger the better (but at the risk of overfitting),but you're saying reduce it to 1 to get better results.

Should it be treated as a lead time or forecast horizon of sorts?

[vafl]

Maybe larger context_length also work, I didn't try that.

[jaschau]

@SkanderHn I had a similar issue described in #684. There I describe a way how to get proper results, but I think it's only in workaround and there's in fact a bug.