search

awslabs / gluonts

Probabilistic time series modeling in Python
https://ts.gluon.ai
Apache License 2.0
4.66k stars 756 forks source link

Can't seem to train a TFT with covariate influence #2162

Open strakehyr opened 2 years ago

strakehyr commented 2 years ago

Hi all, when training a TFT I use, as seen in other notebooks the following code:

train_ds = ListDataset([{FieldName.TARGET: df.iloc[:-horizon, :].loc[:, outputs].squeeze(),
                         FieldName.FEAT_DYNAMIC_REAL: df.iloc[:-horizon, 1:].values,
                         FieldName.START: start.tz_localize(None) }],
                         freq='H')

test_ds = ListDataset([{FieldName.TARGET: df.iloc[:, :].loc[:, outputs].squeeze(), 
                        FieldName.FEAT_DYNAMIC_REAL: df.iloc[:, 1:].values,
                        FieldName.START: start.tz_localize(None) }],
                        freq='H')

 estimator = TemporalFusionTransformerEstimator(
            freq = 'H',
            prediction_length = hyp['horizon'],
            context_length =  hyp['look_back'],
            trainer = Trainer(epochs =  hyp['num_epochs'], learning_rate = hyp['lr'], 
                              learning_rate_decay_factor = 0.9),
            hidden_dim = hyp['hidden_dim'],
            variable_dim = hyp['variable_dim'], # this is the dimension of variable encodings; in the original paper it is always `hidden_dim`
            num_heads = hyp['num_heads'],
            num_outputs = hyp['num_quantiles'], # number of quantiles to be predicted. E.g. [0.5,0.1,0.9] for `num_outputs=3` and [0.5,0.1,0.9,0.2,0.8] for `num_outputs=5`
            dropout_rate = hyp['dropout'])   
        predictor = estimator.train(train_ds)

Upon use of said model, I found out that changing all the covariates (dynamic real) in predictions changes absolutely nothing. Is this expected behaviour with the current code? Am I supposed to prepare data differently? TFT is just not making use of the GRNetwork between covariates when prepared like this.

When preparing with a long dataframe,

feat_dynamic_real = df.drop(feat_dynamic_cat+['Time']+outputs, axis = 1).columns.to_list()
        df_long = pd.melt(df, id_vars = 'Time')
        df.index = df['Time'] 
        for i in df_long.index:
            col = df_long.at[i, 'variable']
            if col in feat_dynamic_cat:
                cat = 1
            else: cat = 0
            pos = df.columns.get_loc(col)
            trgt = df.at[df_long.at[i, 'Time'], outputs[0]]
            df_long.loc[i, 'target'] = trgt
            df_long.loc[i, 'feat_dynamic_cat'] = cat
            df_long.loc[i, 'feat_static_cat'] = feat_static_cats[pos]

train_ds = PandasDataset.from_long_dataframe(df_long, freq = 'H', item_id = 'variable', 
                                                     target = outputs[0], 
                                                     timestamp = 'Time', feat_dynamic_real = ['feat_dynamic_real'],
                                                     feat_dynamic_cat = ['feat_dynamic_cat'])

 estimator = TemporalFusionTransformerEstimator(
            freq = 'H',
            prediction_length = hyp['horizon'],
            context_length =  hyp['look_back'],
            trainer = Trainer(epochs =  hyp['num_epochs'], learning_rate = hyp['lr'], 
                              learning_rate_decay_factor = 0.9),
            hidden_dim = hyp['hidden_dim'],
            variable_dim = hyp['variable_dim'], # this is the dimension of variable encodings; in the original paper it is always `hidden_dim`
            num_heads = hyp['num_heads'],
            num_outputs = hyp['num_quantiles'], # number of quantiles to be predicted. E.g. [0.5,0.1,0.9] for `num_outputs=3` and [0.5,0.1,0.9,0.2,0.8] for `num_outputs=5`
            dropout_rate = hyp['dropout'])   
        predictor = estimator.train(train_ds)

I currently get the error:


The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "C:\Users\user\AppData\Local\Temp/ipykernel_50220/464750682.py", line 12, in <module>
    predictor = estimator.train(train_ds)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\mx\model\estimator.py", line 220, in train
    return self.train_model(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\mx\model\estimator.py", line 177, in train_model
    training_data_loader = self.create_training_data_loader(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\model\tft\_estimator.py", line 342, in create_training_data_loader
    with env._let(max_idle_transforms=maybe_len(data) or 0):

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\itertools.py", line 45, in maybe_len
    return len(obj)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 100, in __len__
    return sum(1 for _ in self)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 100, in <genexpr>
    return sum(1 for _ in self)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 103, in __iter__
    yield from self.transformation(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\dataset\pandas.py", line 145, in __iter__
    dataentry = self.process(self._dataentry(item_id, df))

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\dataset\pandas.py", line 130, in _dataentry
    dataentry = as_dataentry(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\dataset\pandas.py", line 266, in as_dataentry
    dataentry[FieldName.TARGET] = data.loc[:, target].to_list()

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 961, in __getitem__
    return self._getitem_tuple(key)

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1140, in _getitem_tuple
    return self._getitem_lowerdim(tup)

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 867, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1202, in _getitem_axis
    return self._get_label(key, axis=axis)

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1153, in _get_label
    return self.obj.xs(label, axis=axis)

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\generic.py", line 3861, in xs
    return self[key]

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\frame.py", line 3505, in __getitem__
    indexer = self.columns.get_loc(key)

  File "C:\Users\user\Anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 3623, in get_loc
    raise KeyError(key) from err

KeyError: 'target'
lostella commented 2 years ago

@strakehyr looking at the TemporalFusionTransformerEstimator code, I can see that it requires one to explicitly set the names (and dimensions) of additional feature fields in the data. For example, for dynamic numerical features, this is where the feature fields are added: https://github.com/awslabs/gluon-ts/blob/b0d0c41cff7d8a8ab167fc16165477ba9e73329a/src/gluonts/mx/model/tft/_estimator.py#L256-L264

I think this kind of model configuration is not so bad, but it's very different from other models, which just silently assume certain field names to refer to this or that type of feature, and just use them in case they are present.

I believe this is something we will probably modify in the future, and make sure that all models are consistent in this kind of options.

cc @jaheba

lostella commented 2 years ago

@strakehyr you may want to try to configure the estimator as in the following example:

from typing import List
from gluonts.dataset.common import ListDataset
from gluonts.mx import TemporalFusionTransformerEstimator

dataset = ListDataset(
    [{
            "start": "2021-01-01 00",
            "target": [1.0] * 200,
            "feat_dynamic_real": [[1.0] * 200] * 3,
    }],
    freq="1H"
)

estimator = TemporalFusionTransformerEstimator(
    freq="1H",
    prediction_length=24,
    dynamic_feature_dims={"feat_dynamic_real": 3}
)

predictor = estimator.train(dataset)

(Note the dynamic_feature_dims option)

strakehyr commented 2 years ago

Hi @lostella, thanks for your help. I tried to run your code but ran into:

predictor = estimator.train(train_ds)
Traceback (most recent call last):

  File "C:\Users\user\AppData\Local\Temp/ipykernel_50220/3134606962.py", line 1, in <module>
    predictor = estimator.train(train_ds)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\mx\model\estimator.py", line 220, in train
    return self.train_model(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\mx\model\estimator.py", line 177, in train_model
    training_data_loader = self.create_training_data_loader(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\model\tft\_estimator.py", line 342, in create_training_data_loader
    with env._let(max_idle_transforms=maybe_len(data) or 0):

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\itertools.py", line 45, in maybe_len
    return len(obj)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 100, in __len__
    return sum(1 for _ in self)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 100, in <genexpr>
    return sum(1 for _ in self)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 103, in __iter__
    yield from self.transformation(

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 124, in __call__
    for data_entry in data_it:

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 128, in __call__
    raise e

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 126, in __call__
    yield self.map_transform(data_entry.copy(), is_train)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\_base.py", line 141, in map_transform
    return self.transform(data)

  File "C:\Users\user\Anaconda3\lib\site-packages\gluonts\transform\convert.py", line 209, in transform
    output = np.vstack(r) if not self.h_stack else np.hstack(r)

  File "<__array_function__ internals>", line 5, in vstack

  File "C:\Users\user\Anaconda3\lib\site-packages\numpy\core\shape_base.py", line 282, in vstack
    return _nx.concatenate(arrs, 0)

  File "<__array_function__ internals>", line 5, in concatenate

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 13345 and the array at index 1 has size 108

Do you have any advice on the long dataframe issue?

esbraun commented 2 years ago

@strakehyr I've run into issues using the train() method with covariates as well. I think it's because the TFT decoder expects future observations for future known covariates (vs ones known only in the past, which only need to be available to the encoder). I've found using make_evaluation_predictions() instead works since it already holds the prediction length of the target without dropping the future known covariates. So use the following for scoring your model:

forecast_it, ts_total_it = make_evaluation_predictions( dataset=ds_predict, predictor=predictor )

where ds_predict = training data + scoring data.

strakehyr commented 2 years ago

@strakehyr I've run into issues using the train() method with covariates as well. I think it's because the TFT decoder expects future observations for future known covariates (vs ones known only in the past, which only need to be available to the encoder). I've found using make_evaluation_predictions() instead works since it already holds the prediction length of the target without dropping the future known covariates. So use the following for scoring your model:

forecast_it, ts_total_it = make_evaluation_predictions( dataset=ds_predict, predictor=predictor )

where ds_predict = training data + scoring data.

Indeed that is how I run predictions.