Predicting on multiple timeseries, only returns one of the timeseries

[mehrdadfazli]

• PyTorch-Forecasting version: 0.9.2
• PyTorch version: 1.10.2
• Python version: 3.8
• Operating System: Ubuntu

Hi there!

Thanks for this great package. I have a problem with multiple timeseries prediction. I have 11 timeseries in my dataset. I followed the tutorial and built the dataset and model and trained it. However, when I wanted to make predictions, it only predicts one of the timeseries. After some investigation, I found out that there are 11 groups in my training but only 1 group in my validation. I am having a hard time figuring out the root of the issue. I appreciate any help. Below is some of my code:

Training and validation

training = TimeSeriesDataSet(
    data_agg[lambda x: x.time_idx <= training_cutoff],
    time_idx="time_idx",
    target="cases",
    group_ids=["name"],
    min_encoder_length=max_encoder_length // 2,  # keep encoder length long (as it is in the validation set)
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=["name", "state"],
    time_varying_known_categoricals=["month", "DayOfWeek", "WeekOfYear"],
    time_varying_known_reals=["time_idx", "Index1", "Index2", "Index3", "Index4"],
    time_varying_unknown_categoricals=[],
    time_varying_unknown_reals=["var_idx"],
    target_normalizer=GroupNormalizer(
        groups=["name"], transformation="softplus"
    ),  # use softplus and normalize by group
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_length=True,
)

validation = TimeSeriesDataSet.from_dataset(training, data, predict=True, stop_randomization=True)

# create dataloaders for model
batch_size = 64  # set this between 32 to 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=0)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size * 10, num_workers=0)

and my model:

loss_logger_tft = LossLogger()
early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4, patience=10, verbose=False, mode="min",
                                   check_on_train_epoch_end=True)

pl.seed_everything(42)
trainer = pl.Trainer(
    max_epochs = 150,
    auto_lr_find=True,
#     auto_scale_batch_size=True,
    accelerator="gpu",
    gpus=1,
    # clipping gradients is a hyperparameter and important to prevent divergance
    # of the gradient for recurrent neural networks
    gradient_clip_val=None,
    val_check_interval=1,
    check_val_every_n_epoch=1,
#     logger=True,
    callbacks=[loss_logger_tft, early_stop_callback],
)

tft = TemporalFusionTransformer.from_dataset(
    training,
    lstm_layers=2,            # Number of LSTM layers: 1 and 2 are good
    learning_rate=5e-5,       # not meaningful for finding the learning rate but otherwise very important
    hidden_size=16,           # most important hyperparameter apart from learning rate
    attention_head_size=4,    # number of attention heads. Set to up to 4 for large datasets
    dropout=0.2,              # between 0.1 and 0.3 are good values
    hidden_continuous_size=8, # set to <= hidden_size
    output_size=7,            # 7 quantiles by default
    loss=QuantileLoss(),
    # reduce learning rate if no improvement in validation loss after x epochs
    reduce_on_plateau_patience=4,
)
print(f"Number of parameters in network: {tft.size()/1e3:.1f}k")

# fit network
trainer.fit(
    tft,
    train_dataloader=train_dataloader,
    val_dataloaders=val_dataloader,
)

After the model was trained, I tried to plot the predictions for the counties:

for idx in range(10):  # plot 10 examples
    best_tft.plot_prediction(x, raw_predictions, idx=idx, add_loss_to_title=True);

and I got:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_2583/2995530739.py in <module>
      1 for idx in range(10):  # plot 10 examples
----> 2     best_tft.plot_prediction(x, raw_predictions, idx=idx, add_loss_to_title=True);

~/.conda/envs/darts_env/lib/python3.7/site-packages/pytorch_forecasting/models/temporal_fusion_transformer/__init__.py in plot_prediction(self, x, out, idx, plot_attention, add_loss_to_title, show_future_observed, ax, **kwargs)
    671             show_future_observed=show_future_observed,
    672             ax=ax,
--> 673             **kwargs,
    674         )
    675 

~/.conda/envs/darts_env/lib/python3.7/site-packages/pytorch_forecasting/models/base_model.py in plot_prediction(self, x, out, idx, add_loss_to_title, show_future_observed, ax, quantiles_kwargs, prediction_kwargs)
    759         ):
    760 
--> 761             y_all = torch.cat([encoder_target[idx], decoder_target[idx]])
    762             max_encoder_length = x["encoder_lengths"].max()
    763             y = torch.cat(

IndexError: index 1 is out of bounds for dimension 0 with size 1

It only plotted one of the timeseries. So, I investigated a bit and figured that in validation set there is only one group available:

In[]: training.data['groups']
Out[]: tensor([[ 0],
        [ 0],
        [ 0],
        ...,
        [10],
        [10],
        [10]])

and

In[]:validation.data['groups']
Out[]:tensor([[4],
        [4],
        [4],
        ...
        [4],
        [4],
        [4]])

I exactly followed the tutorial. The only difference was that in my data groups are identified by column name.

Moreover, I am struggling to find out the shape and structure of the raw predictions. Any help in that regard is also appreciated.

Thanks,

[hschmiedt]

Hi, try use predict=False in the validation TimeSeriesDataSet (in the function .from_dataset). The predict=True only uses the last timeseries in the validation data.

This worked for me.

Best

[mehrdadfazli]

@hschmiedt Thanks for your suggestion. I tried that (setting predict=False in for validation) but still I have one of my many timeseries in the validation dataset. So the problem persists in my case.

Best

[SC4RECOIN]

@hschmiedt's suggestion worked for me

cheers