unable to reproduce the resultsof TimeGrad on some datasets

[yantijin]

I'm very glad to read the paper "Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting"， it's a very intesting work and TimeGrad achieves state of the art results on multivariate time series forecasting tasks. However, I can not reproduce the results on some datasets with the model implemented in pytorch-ts. Could you release the hyperparameter settings of these datasets in the paper? Thx a lot!

[kashif]

thanks @yantijin I will put my experiment notebooks on github ASAP. Is there any dataset in particular you would like me help you with initially?

Best wishes, Kashif

[yantijin]

Thanks for your quickly reply！I have some problems when testing the performance on taxi and traffic. I run the model on a single gpu with 24Gb of memory and the best CRPS_{sum} values I got are approximately 0.059 and 0.140 on traffic and taxi. I guess the reason is that the hyperparameter settings on these datasets are different with those on electricity. Could you help me with that @kashif

[kashif]

So for traffic If I remember you need to set the scaling=False:

estimator = TimeGradEstimator(
    target_dim=int(dataset.metadata.feat_static_cat[0].cardinality),
    prediction_length=dataset.metadata.prediction_length,
    context_length=dataset.metadata.prediction_length,
    cell_type='GRU',
    num_cells=128,
    input_size=3856,
    loss_type='l2',
    freq=dataset.metadata.freq,
    scaling=False,
    trainer=Trainer(device=device,
                    epochs=30,
                    learning_rate=1e-3,
                    num_batches_per_epoch=100,
                    batch_size=32)
)

let me check my notes about taxi

[yantijin]

I have evaluated the model on traffic with the settings mentioned above and get similar results to the paper, thx very much! @kashif

[kashif]

@yantijin awesome so next is taxi... I'm going over the notebook again, note in the meantime, a number of things have happened in the underlying gluonts library: they have refactored how the metrics are calculated and I have fixed the random seed in multi-worker training so I will re-run the notebooks and report back the hyper-params for taxi shortly

[kashif]

So for taxi you should try something like:

estimator = TimeGradEstimator(
    target_dim=min(2000,int(dataset.metadata.feat_static_cat[0].cardinality)),
    prediction_length=dataset.metadata.prediction_length,
    context_length=dataset.metadata.prediction_length,
    cell_type='GRU',
    num_cells=128,
    input_size=2434,
    loss_type='l2',
    diff_steps=200,
    beta_end=0.15,
    freq=dataset.metadata.freq,
    scaling=True,
    trainer=Trainer(device=device,
                    epochs=50,
                    learning_rate=1e-3,
                    num_batches_per_epoch=100,
                    batch_size=32)
)

and perhaps even train it a bit longer...