TimeSeriesDataSet with multiple targets

miheHMR commented 2 years ago

PyTorch-Forecasting version: 0.9.2
PyTorch version:
PyTorch-Lightning: 1.5.7
Python version: 3.8.9
Operating System: Win10

I'm trying to create a dataset with multiple targets. In my case it looks like this:

training = TimeSeriesDataSet(
    data[lambda x: x.time_idx <= training_cutoff],
    time_idx="time_idx", 
    target=["target_cat1", "target_var1", "target_var2", "target_var7"],
    group_ids=["group"], # initialized column with 0
    min_encoder_length=max_encoder_length // 2,  # keep encoder length long (as it is in the validation set)
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    variable_groups={},  # group of categorical variables can be treated as one variable
    static_categoricals=[],
    static_reals=[],
    time_varying_known_categoricals=["cat2", "cat4", "cat3"],
    time_varying_known_reals=["time_idx","var3", "var5","var6","var8","var9","var10"],
    time_varying_unknown_categoricals=["target_cat1"],
    time_varying_unknown_reals=["target_var1", "target_var2", "target_var7"],
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_length=True,
)

I get errors in several places.

1. During the initialization of the TimeSeriesDataSet

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

My dataTypes:

timestamp      datetime64[ns]
target_var1           float64
target_var2           float64
var3                  float64
target_cat1            object
var4                  float64
var5                  float64
var6                  float64
target_var7           float64
var8                  float64
var9                  float64
cat2                   object
cat3                   object
cat4                   object
var10                 float64
time_idx                int64
group                   int64
dtype: object

If i remove the categorical target it works. Is it possible to mix categorical and continous target values?

2. During Init TemporalFusionTransformer

tft = TemporalFusionTransformer.from_dataset(
    training,
    # not meaningful for finding the learning rate but otherwise very important
    learning_rate=0.03,
    hidden_size=16,  # most important hyperparameter apart from learning rate
    # number of attention heads. Set to up to 4 for large datasets
    attention_head_size=1,
    dropout=0.1,  # between 0.1 and 0.3 are good values
    hidden_continuous_size=8,  # set to <= hidden_size
    output_size=7,  # 7 quantiles by default
    loss=QuantileLoss(),
    # reduce learning rate if no improvement in validation loss after x epochs
    reduce_on_plateau_patience=4,
)

Error:

TypeError: 'int' object is not iterable

Message looks clear:)

At this palce in class TemporalFusionTransformer

if self.n_targets > 1:  # if to run with multiple targets
    self.output_layer = nn.ModuleList(
        [nn.Linear(self.hparams.hidden_size, output_size) for output_size in self.hparams.output_size]
    )

I'm thankful for any help Maybe there exist any example?

jhbale11 commented 2 years ago

I have a similar issue, I'm trying to train multiple target model but do not know How to set target normalizer and group_ids Thx

miheHMR commented 2 years ago

@jhbale11 I made a new column as group_id and initialized it with 0. Because i've only one group. Normalizer i have not configured yet.

I made other changes and they seem to be working In the TemporalFusionTransformer.from_dataset part it seems that theres a list needed as output_size.

output_size=[7, 7, 7, 7], # 4 target variables

and for loss calculations i put in a MultiLoss

loss=MultiLoss([QuantileLoss(), QuantileLoss(), QuantileLoss(), QuantileLoss()]),

jhbale11 commented 2 years ago

thanks a lot!!!!

I have a further question about multi target variables.

I'm trying to get output for multiple targets with Softmax.

Which means sum of the target's outputs should be 1.

What should i do then?

Thanks again for your kindness :)

theoctopusride commented 2 years ago

thanks a lot!!!!

I have a further question about multi target variables.

I'm trying to get output for multiple targets with Softmax.

Which means sum of the target's outputs should be 1.

What should i do then?

Thanks again for your kindness :)

Were you able to figure this out? I am running into issues with multiple targets, and now I'm stuck at the trainer.fit step. I am working on a multi-class classification problem. I have 7 different target variables and would like the sum of their probabilities -- coupled with ( 1 - (the probabilities of all 7 added together)) -- to add to 1. I followed the suggestions in this thread (MultiLoss and output size as a list). The output is below:

`LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

| Name | Type | Params

0 | loss | MultiLoss | 0
1 | logging_metrics | ModuleList | 0
2 | input_embeddings | MultiEmbedding | 1
3 | prescalers | ModuleDict | 336
4 | static_variable_selection | VariableSelectionNetwork | 10.8 K 5 | encoder_variable_selection | VariableSelectionNetwork | 3.7 K 6 | decoder_variable_selection | VariableSelectionNetwork | 528
7 | static_context_variable_selection | GatedResidualNetwork | 1.1 K 8 | static_context_initial_hidden_lstm | GatedResidualNetwork | 1.1 K 9 | static_context_initial_cell_lstm | GatedResidualNetwork | 1.1 K 10 | static_context_enrichment | GatedResidualNetwork | 1.1 K 11 | lstm_encoder | LSTM | 2.2 K 12 | lstm_decoder | LSTM | 2.2 K 13 | post_lstm_gate_encoder | GatedLinearUnit | 544
14 | post_lstm_add_norm_encoder | AddNorm | 32
15 | static_enrichment | GatedResidualNetwork | 1.4 K 16 | multihead_attn | InterpretableMultiHeadAttention | 1.1 K 17 | post_attn_gate_norm | GateAddNorm | 576
18 | pos_wise_ff | GatedResidualNetwork | 1.1 K 19 | pre_output_gate_norm | GateAddNorm | 576
20 | output_layer | ModuleList | 952

30.1 K Trainable params 0 Non-trainable params 30.1 K Total params 0.120 Total estimated model params size (MB) Sanity Checking DataLoader 0: 0% 0/1 [00:01<?, ?it/s]

TypeError Traceback (most recent call last) Input In [45], in <cell line: 2>() 1 # fit network ----> 2 trainer.fit( 3 tft, 4 train_dataloaders=train_dataloader, 5 val_dataloaders=val_dataloader, 6 )

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:768, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path) 749 r""" 750 Runs the full optimization routine. 751 (...) 765 datamodule: An instance of :class:~pytorch_lightning.core.datamodule.LightningDataModule. 766 """ 767 self.strategy.model = model --> 768 self._call_and_handle_interrupt( 769 self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path 770 )

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:721, in Trainer._call_and_handle_interrupt(self, trainer_fn, *args, kwargs) 719 return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, *kwargs) 720 else: --> 721 return trainer_fn(args, kwargs) 722 # TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7 723 except KeyboardInterrupt as exception:

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:809, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path) 805 ckpt_path = ckpt_path or self.resume_from_checkpoint 806 self._ckpt_path = self.__set_ckpt_path( 807 ckpt_path, model_provided=True, model_connected=self.lightning_module is not None 808 ) --> 809 results = self._run(model, ckpt_path=self.ckpt_path) 811 assert self.state.stopped 812 self.training = False

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1234, in Trainer._run(self, model, ckpt_path) 1230 self._checkpoint_connector.restore_training_state() 1232 self._checkpoint_connector.resume_end() -> 1234 results = self._run_stage() 1236 log.detail(f"{self.class.name}: trainer tearing down") 1237 self._teardown()

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1321, in Trainer._run_stage(self) 1319 if self.predicting: 1320 return self._run_predict() -> 1321 return self._run_train()

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1343, in Trainer._run_train(self) 1340 self._pre_training_routine() 1342 with isolate_rng(): -> 1343 self._run_sanity_check() 1345 # enable train mode 1346 self.model.train()

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1411, in Trainer._run_sanity_check(self) 1409 # run eval step 1410 with torch.no_grad(): -> 1411 val_loop.run() 1413 self._call_callback_hooks("on_sanity_check_end") 1415 # reset logger connector

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/loops/base.py:204, in Loop.run(self, *args, kwargs) 202 try: 203 self.on_advance_start(*args, *kwargs) --> 204 self.advance(args, kwargs) 205 self.on_advance_end() 206 self._restarting = False

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py:154, in EvaluationLoop.advance(self, *args, **kwargs) 152 if self.num_dataloaders > 1: 153 kwargs["dataloader_idx"] = dataloader_idx --> 154 dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs) 156 # store batch level output per dataloader 157 self._outputs.append(dl_outputs)

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/loops/base.py:204, in Loop.run(self, *args, kwargs) 202 try: 203 self.on_advance_start(*args, *kwargs) --> 204 self.advance(args, kwargs) 205 self.on_advance_end() 206 self._restarting = False

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py:127, in EvaluationEpochLoop.advance(self, data_fetcher, dl_max_batches, kwargs) 124 self.batch_progress.increment_started() 126 # lightning module methods --> 127 output = self._evaluation_step(**kwargs) 128 output = self._evaluation_step_end(output) 130 self.batch_progress.increment_processed()

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py:222, in EvaluationEpochLoop._evaluation_step(self, *kwargs) 220 output = self.trainer._call_strategy_hook("test_step", kwargs.values()) 221 else: --> 222 output = self.trainer._call_strategy_hook("validation_step", *kwargs.values()) 224 return output

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1763, in Trainer._call_strategy_hook(self, hook_name, *args, *kwargs) 1760 return 1762 with self.profiler.profile(f"[Strategy]{self.strategy.class.name}.{hook_name}"): -> 1763 output = fn(args, **kwargs) 1765 # restore current_fx when nested context 1766 pl_module._current_fx_name = prev_fx_name

File ~/.local/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py:344, in Strategy.validation_step(self, *args, *kwargs) 339 """The actual validation step. 340 341 See :meth:~pytorch_lightning.core.lightning.LightningModule.validation_step for more details 342 """ 343 with self.precision_plugin.val_step_context(): --> 344 return self.model.validation_step(args, **kwargs)

File ~/.local/lib/python3.8/site-packages/pytorch_forecasting/models/base_model.py:413, in BaseModel.validation_step(self, batch, batch_idx) 411 def validation_step(self, batch, batch_idx): 412 x, y = batch --> 413 log, out = self.step(x, y, batch_idx) 414 log.update(self.create_log(x, y, out, batch_idx)) 415 return log

File ~/.local/lib/python3.8/site-packages/pytorch_forecasting/models/base_model.py:547, in BaseModel.step(self, x, y, batch_idx, *kwargs) 545 loss = loss (1 + monotinicity_loss) 546 else: --> 547 out = self(x, **kwargs) 549 # calculate loss 550 prediction = out["prediction"]

File ~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, *kwargs) 1106 # If we don't have any hooks, we want to skip the rest of the logic in 1107 # this function, and just call forward. 1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1109 or _global_forward_hooks or _global_forward_pre_hooks): -> 1110 return forward_call(input, **kwargs) 1111 # Do not call functions when jit is used 1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.local/lib/python3.8/site-packages/pytorch_forecasting/models/temporal_fusion_transformer/init.py:503, in TemporalFusionTransformer.forward(self, x) 499 else: 500 output = self.output_layer(output) 502 return self.to_network_output( --> 503 prediction=self.transform_output(output, target_scale=x["target_scale"]), 504 encoder_attention=attn_output_weights[..., :max_encoder_length], 505 decoder_attention=attn_output_weights[..., max_encoder_length:], 506 static_variables=static_variable_selection, 507 encoder_variables=encoder_sparse_weights, 508 decoder_variables=decoder_sparse_weights, 509 decoder_lengths=decoder_lengths, 510 encoder_lengths=encoder_lengths, 511 )

File ~/.local/lib/python3.8/site-packages/pytorch_forecasting/models/base_model.py:336, in BaseModel.transform_output(self, prediction, target_scale) 325 """ 326 Extract prediction from network output and rescale it to real space / de-normalize it. 327 (...) 333 torch.Tensor: rescaled prediction 334 """ 335 if isinstance(self.loss, MultiLoss): --> 336 out = self.loss.rescale_parameters( 337 prediction, 338 target_scale=target_scale, 339 encoder=self.output_transformer.normalizers, # need to use normalizer per encoder 340 ) 341 else: 342 out = self.loss.rescale_parameters(prediction, target_scale=target_scale, encoder=self.output_transformer)

File ~/.local/lib/python3.8/site-packages/pytorch_forecasting/metrics.py:452, in MultiLoss.getattr..func(*args, *kwargs) 438 new_args = [ 439 arg[idx] 440 if isinstance(arg, (list, tuple)) (...) 444 for arg in args 445 ] 446 new_kwargs = { 447 key: val[idx] 448 if isinstance(val, list) and not isinstance(val, rnn.PackedSequence) and len(val) == n 449 else val 450 for key, val in kwargs.items() 451 } --> 452 results.append(getattr(m, name)(new_args, **new_kwargs)) 453 return results

File ~/.local/lib/python3.8/site-packages/pytorch_forecasting/metrics.py:75, in Metric.rescale_parameters(self, parameters, target_scale, encoder) 61 def rescale_parameters( 62 self, parameters: torch.Tensor, target_scale: torch.Tensor, encoder: BaseEstimator 63 ) -> torch.Tensor: 64 """ 65 Rescale normalized parameters into the scale required for the output. 66 (...) 73 torch.Tensor: parameters in real/not normalized space 74 """ ---> 75 return encoder(dict(prediction=parameters, target_scale=target_scale))

TypeError: 'list' object is not callable`

jdb78 / pytorch-forecasting

TimeSeriesDataSet with multiple targets #820

| Name | Type | Params

30.1 K Trainable params 0 Non-trainable params 30.1 K Total params 0.120 Total estimated model params size (MB) Sanity Checking DataLoader 0: 0% 0/1 [00:01<?, ?it/s]