DeepAR multiple features to predict one target and general questions

[jaeniale-fd]

First of all thank you for this great project/library. I want to use it in my master thesis but i have a few questions. I want to predict the future with the DeepAR Model. I have one target variable and multiple irregular features, for which i have values in the future. I have data for two years and want to predict 15 days.

My predition table is a pandas dataframe. My index consists of pandas timestamps. My target variable is an integer. My features are either one hot encoded categoricals or integers / floats.

To create my training set: start is the first day of my time_series target is my target variable to a given point in time features_x are the external features to that given point in time

To create my test set: start is the first day after the end of my training set target is my target variable from that point of time till the end features_x are the external features for all points in time (also the training data points)

this is my code:

estimator = DeepAREstimator(freq="d",
                            prediction_length=15
                            , context_length=7,
                            trainer=Trainer(epochs=10))

training_data = ListDataset(
    [{"start": df.index[0], "target": df.target[:pd.to_datetime('2019-05-01')],
      'feat_dynamic_real': df.feature_1[:pd.to_datetime('2019-05-01')],
      'feat_dynamic_real': df.feature_2[:pd.to_datetime('2019-05-01')],
      'feat_dynamic_real': df.feature_3[:pd.to_datetime('2019-05-01')],
      }],
    freq="d"
)
test_data = ListDataset(
    [
        {"start": df.index[df.index == pd.to_datetime('2019-08-01')][0], "target": df.target[pd.to_datetime('2019-05-02'):],
         'feat_dynamic_real': df.feature_1,
         'feat_dynamic_real': df.feature_2,
         'feat_dynamic_real': df.feature_3,
         }
    ],
    freq="d"
)

predictor = estimator.train(training_data=training_data)

def plot_forecasts(tss, forecasts, past_length, num_plots):
    for target, forecast in islice(zip(tss, forecasts), num_plots):
        ax = target[-past_length:].plot(figsize=(12, 5), linewidth=2)
        forecast.plot(color='g')
        plt.grid(which='both')
        plt.legend(["observations", "median prediction", "90% confidence interval", "50% confidence interval"])
        plt.show()

forecast_it, ts_it = make_evaluation_predictions(test_data, predictor=predictor, num_eval_samples=100)

forecasts = list(forecast_it)
tss = list(ts_it)
plot_forecasts(tss, forecasts, past_length=50, num_plots=3)

Questions: 1. Am i doing it right e.g. training set all data is from start till end of training set and in the test set my target variable and start is from end of training set till the end of my data set, while the features and put in completely.

2. How do i see if it used the external features properly? My forecast plots look almost exactly the same and evaluation does not change significantly as well.

3. Can i also add features, for which i don't have future values, to train the model? For example i want to train the model to predict the total count of ticket sales, and i have additional information, like marketing campaigns in the past ticket type and ticket seller, which might is an indicator for the future. Do i have to engeneer this feature myself, by for example shifitng the data or is there another way to include this into the training process? Ofcourse for marketing campaign i would have the information, but the effect might apply much later then 15 days. For ticket type and provider i don't have future information, but it might still have some influence on the model training.

4. Should i include time related internal features like quarter, day of week, half of month seperately or are those computed automatically?

5. I splitted the data up to build a daily model, normally my data would be hourly, e.g. i have data for every hour from 10 am to 8 pm and afterwards the store is closed, so the target to predict is always zero. How do i deal with this kind of data? Do i have to remove the "hourly" data points when the store is closed to prevent the Network from predicting values below zero? Do i have to set the freqency to hourly then? Is it even possible to set the frequency to hourly when it is not a full 24hour iteration(because of closing times)?

6. I was not able to to create the ListDataset in the way it is shown in the tutorial. When doing:

training_data = ListDataset([{FieldName.TARGET: target,
                              FieldName.START: start,
                              FieldName.FEAT_DYNAMIC_REAL: fdr
                              }`
                             for (target, start, fdr) in
                             zip(df.target[:pd.to_datetime('2019-05-01')],
                                 df.index[df.index <= pd.to_datetime('2019-05-01')],
                                 df.feature_1[:pd.to_datetime('2019-05-01')]
                                 )], freq='d')

It throws:

f"JSON array has bad shape - expected {self.req_ndim} " gluonts.core.exception.GluonTSDataError: JSON array has bad shape - expected 2 dimensions, got 2

Although the inputs are exactly the same as in the code which i made it work with.

[mbohlkeschneider]

Hi @jaeniale-fd ,

thank you for using GluonTS.

I'll try to go over your questions one by one:

1. Am i doing it right e.g. training set all data is from start till end of training set and in the test set my target variable and start is from end of training set till the end of my data set, while the features and put in completely.

Sounds right to me. If you want to compute the evaluation metrics with backtest_metrics the test time series should basically be on the entire horizon and the train time series should be test_ds[:-prediction_length]. You are correct, dynamic features should be given in the full test range.

2. How do i see if it used the external features properly? My forecast plots look almost exactly the same and evaluation does not change significantly as well.

The number of model parameters should go up if you include external features. If the forecast does not change a lot, this might mean that they have little impact on the forecast. It looks that you encoded them incorrectly if you want to use them all. They should be arranged in an 2D array.

So instead

         'feat_dynamic_real': df.feature_1,
         'feat_dynamic_real': df.feature_2,
         'feat_dynamic_real': df.feature_3,

try

         'feat_dynamic_real': [df.feature_1, df.feature_2, df.feature_3]

You can verify with the number of model parameters that they get picked up correctly.

3. Can i also add features, for which i don't have future values, to train the model?

No, you have to provide the future values for prediction. If you don't have them you can fill dummy values. This can be tricky to get right, though. For example, if you have a 0-1 promotion feature, you can assume that no promotion happens in the future. That would work, but I don't know whether this improves anything.

Do i have to engeneer this feature myself, by for example shifitng the data or is there another way to include this into the training process?

What exactly do you mean with "engineer features yourself"? Do you mean translating from strings to numerical values? If that is the case, then you have to do it yourself, yes.

4. Should i include time related internal features like quarter, day of week, half of month seperately or are those computed automatically?

We have some default features that we add all the time (like day of week). You can of course add whatever you like.

5. I splitted the data up to build a daily model, normally my data would be hourly, e.g. i have data for every hour from 10 am to 8 pm and afterwards the store is closed, so the target to predict is always zero. How do i deal with this kind of data? Do i have to remove the "hourly" data points when the store is closed to prevent the Network from predicting values below zero? Do i have to set the freqency to hourly then? Is it even possible to set the frequency to hourly when it is not a full 24hour iteration(because of closing times)?

Somebody raised this issue before. We currently do not support "business hours" frequency. One thing you could try to encode the target as nan for the time the store is closed. Please let us know what you find!

6. I was not able to to create the ListDataset in the way it is shown in the tutorial.

The code above worked out, no? Hard to diagnose the code it without seeing the data.

Hope that helped!

[jaeniale-fd]

Thank you for your fast answer.

Here is a synthetic example of my case:

from itertools import islice

import matplotlib.pyplot as plt
from gluonts.dataset.common import ListDataset
from gluonts.evaluation import Evaluator
from gluonts.evaluation.backtest import make_evaluation_predictions
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer
import pandas as pd
import numpy as np

date_rng = pd.date_range(start='21/09/2017', end='31/08/2019', freq='H')
df = pd.DataFrame(date_rng, columns=['date'])
df['date'] = pd.to_datetime(df['date'])
df['target'] = np.random.randint(0, 100, size=(len(date_rng)))
df['feature_1'] = np.random.randint(2, size=(len(date_rng)))
df['feature_2'] = np.random.randint(0, 20, size=(len(date_rng)))
df['feature_3'] = np.random.randint(2, size=(len(date_rng)))
df = df[(df['date'].dt.hour == 17)]

df = df.set_index('date')
prediction_length = 15

estimator = DeepAREstimator(freq="d",
                            prediction_length=prediction_length
                            , context_length=7,
                            trainer=Trainer(epochs=10), use_feat_dynamic_real=True)

training_data = ListDataset(
    [{"start": df.index[0], "target": df.target[:pd.to_datetime('2019-05-31 17:00')],
      'feat_dynamic_real': [df.feature_1[:pd.to_datetime('2019-05-31 17:00')],
                            df.feature_2[:pd.to_datetime('2019-05-31 17:00')],
                            df.feature_3[:pd.to_datetime('2019-05-31 17:00')]]
      }],
    freq="d"
)
test_data = ListDataset(
    [
        {"start": df.index[0],
         "target": df.target[pd.to_datetime('2019-06-01 17:00'):],
         'feat_dynamic_real': df.feature_1,
         'feat_dynamic_real': df.feature_2,
         'feat_dynamic_real': df.feature_3,
         }
    ],
    freq="d"
)

predictor = estimator.train(training_data=training_data)

def plot_forecasts(tss, forecasts, past_length, num_plots):
    for target, forecast in islice(zip(tss, forecasts), num_plots):
        ax = target[-past_length:].plot(figsize=(12, 5), linewidth=2)
        forecast.plot(color='g')
        plt.grid(which='both')
        plt.legend(["observations", "median prediction", "90% confidence interval", "50% confidence interval"])
        plt.show()

forecast_it, ts_it = make_evaluation_predictions(test_data, predictor=predictor, num_eval_samples=100)

forecasts = list(forecast_it)
tss = list(ts_it)
plot_forecasts(tss, forecasts, past_length=50, num_plots=3)

evaluator = Evaluator(quantiles=[0.5], seasonality=2018)

agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(test_data))

It throws:

ValueError: all the input array dimensions except for the concatenation axis must match exactly at forecasts = list(forecast_it)

Im probably still construction the splits wrong. In the tutorial the splits with dynamic features are created like that:

train_ds = ListDataset([{FieldName.TARGET: target,
                     FieldName.START: start,
                     FieldName.FEAT_DYNAMIC_REAL: fdr}
                    for (target, start, fdr) in zip(
target,
custom_ds_metadata['start'],
feat_dynamic_real)]

but i could not make it work, so i tried my approach.

Ideally i would like to do a rolling prediction on my test data e.g. predict 15 days from last day of training set, then add 15 days from testing set and predict another 15 days, repeat, repeat until the test set is empty. Do i have to implement that myself, or is it taken care of when the test period is longer then the prediction_length?

By "engineer features yourself" i meant finding out the influence and mapping it to the future, e.g. starting a marketing campaign now will increase sales by 0.2 in two weeks.

Thanks in advance

[davidlkl]

The fields in test_data should be set like this: {"start": df.index[0], "target": df.target[:pd.to_datetime('2019-06-01 17:00'):], 'feat_dynamic_real': [df.feature_1[:pd.to_datetime('2019-06-01 17:00')], df.feature_2[:pd.to_datetime('2019-06-01 17:00')], df.feature_3[:pd.to_datetime('2019-06-01 17:00')]] }

[mbohlkeschneider]

Ideally i would like to do a rolling prediction on my test data e.g. predict 15 days from last day of training set, then add 15 days from testing set and predict another 15 days, repeat, repeat until the test set is empty. Do i have to implement that myself, or is it taken care of when the test period is longer then the prediction_length?

We currently don't have functionality to do rolling evaluations, so you would have to construct the test set yourself.

By "engineer features yourself" i meant finding out the influence and mapping it to the future, e.g. starting a marketing campaign now will increase sales by 0.2 in two weeks.

Ok. So the model should learn the influence of a feature. You will have to come up with the numerical representation of the feature, though. Hope that was clear.

[jaeniale-fd]

The fields in test_data should be set like this: {"start": df.index[0], "target": df.target[:pd.to_datetime('2019-06-01 17:00'):], 'feat_dynamic_real': [df.feature_1[:pd.to_datetime('2019-06-01 17:00')], df.feature_2[:pd.to_datetime('2019-06-01 17:00')], df.feature_3[:pd.to_datetime('2019-06-01 17:00')]] }

When i use your code to construct the test data it throws the following error:

AttributeError: 'str' object has no attribute 'copy'

Also it does not really make sense, since it was stated that i need future values and full history for the dynamic_features, your code makes me use only the history of the dynamic features.

[jaeniale-fd]

@mbohlkeschneider by any chance, could you run my example and tell me what im doing wrong? Also where can i check the number of model parameters, will that be included in the return value of backtestmetrics?

Thanks in Advance

[mbohlkeschneider]

Hi @jaeniale-fd ,

@davidlkl is actually right. His code works fine. I have a snipplet below that puts his code into your snipplet (and fixed another bug in the backtest_metrics call. Maybe some confusion comes from that make_evaluation_predictions will remove the last prediction_length time points from the target. So giving target and the dynamic features in the full range will do the trick.

In this snipplet, I assume that the date range you are interested in is 2019-06-01 to 2019-06-15.

The number of models parameters is printed when you invoke training.

Hope that helps.

from itertools import islice

import matplotlib.pyplot as plt
from gluonts.dataset.common import ListDataset
from gluonts.evaluation import Evaluator
from gluonts.evaluation.backtest import make_evaluation_predictions
from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer
import pandas as pd
import numpy as np

date_rng = pd.date_range(start='21/09/2017', end='31/08/2019', freq='H')
df = pd.DataFrame(date_rng, columns=['date'])
df['date'] = pd.to_datetime(df['date'])
df['target'] = np.random.randint(0, 100, size=(len(date_rng)))
df['feature_1'] = np.random.randint(2, size=(len(date_rng)))
df['feature_2'] = np.random.randint(0, 20, size=(len(date_rng)))
df['feature_3'] = np.random.randint(2, size=(len(date_rng)))
df = df[(df['date'].dt.hour == 17)]

df = df.set_index('date')
prediction_length = 15

estimator = DeepAREstimator(freq="d",
                            prediction_length=prediction_length
                            , context_length=7,
                            trainer=Trainer(epochs=1), use_feat_dynamic_real=True)

training_data = ListDataset(
    [{"start": df.index[0], "target": df.target[:pd.to_datetime('2019-05-31 17:00')],
      'feat_dynamic_real': [df.feature_1[:pd.to_datetime('2019-05-31 17:00')],
                            df.feature_2[:pd.to_datetime('2019-05-31 17:00')],
                            df.feature_3[:pd.to_datetime('2019-05-31 17:00')]]
      }],
    freq="d"
)
test_data = ListDataset(
    [
        {"start": df.index[0],
        "target": df.target[:pd.to_datetime('2019-06-15 17:00'):],
        'feat_dynamic_real': [df.feature_1[:pd.to_datetime('2019-06-15 17:00')],
        df.feature_2[:pd.to_datetime('2019-06-15 17:00')],
        df.feature_3[:pd.to_datetime('2019-06-15 17:00')]]
        }
    ],
    freq="d"
)

predictor = estimator.train(training_data=training_data)

def plot_forecasts(tss, forecasts, past_length, num_plots):
    for target, forecast in islice(zip(tss, forecasts), num_plots):
        ax = target[-past_length:].plot(figsize=(12, 5), linewidth=2)
        forecast.plot(color='g')
        plt.grid(which='both')
        plt.legend(["observations", "median prediction", "90% confidence interval", "50% confidence interval"])
        plt.show()

forecast_it, ts_it = make_evaluation_predictions(test_data, predictor=predictor, num_samples=100)

forecasts = list(forecast_it)
tss = list(ts_it)
plot_forecasts(tss, forecasts, past_length=50, num_plots=3)

evaluator = Evaluator(quantiles=[0.5], seasonality=2018)

agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(test_data))

​

[mbohlkeschneider]

Hope that clarified your questions. Closing this for now, feel free to reopen if you still have questions.

[jaeniale-fd]

Thank you very much, that works like a charm now. Before i closes this topic i have a few more questions. @mbohlkeschneider i cannot reopen the issue if it has not been closed by myself, so i tag you here as you could hopefully answer my last questions.

About DeepAREstimator parameters: Does frequency only refer to the gap between two datapoints e.g. "D" or "H", or does it refer to the timeframe we see a pattern repeat, e.g. "7D". When i have a series of data points where 10 days lay in between, do i specify "10D" or "D"? How do i determine the value for the context_length parameter? Do i do this with grid search?

About the backtest (make_evaluation_prediction): The documentation says:

• Removes the final window of length prediction_length of the dataset.test that we want to predict
• The estimator uses the remaining data to predict (in the form of sample paths) the “future” window that was just removed

Does that mean it predicts my prediction_length at num_samples different random points in my test set to evaluate the model?

About Evaluator() It has a parameter "seasonality". I could not find any further information in the documentation. How do i determine the best value for this parameter, and what does it actually mean in regards to my dataset?

About feature importance: Is there anything implemented, that shows me how important which of my exogenous variables is?

About my specific hourly data problem: Since i actually have hourly data from 8am to 8pm, i think it is a total waste to look model each hour individually as daily data because i lose the influence of the previous hours. So what i did it i set my DeepAREstimator frequency to "H" and prediction length to 360 (15days x 24hours). Then i set the context length to 744 (31days x 4hours) because i want it to look at the last 312 days. I created a feature called store closed. The network should learn itself, that if that feature is 1, the sales are 0 as well, right?

It runs through, and the results make sense, i just want to know if i forgot something, that i did not think of.

Thanks in advance, Alex

[janrth]

If i have this:

 'feat_dynamic_real': [df.feature_1, df.feature_2, df.feature_3]

it means I have one target and 3 features, right (because you are selecting columns).

How would I be able to have multiple time-series for targets and many features. Because then my array would have a third dimension, right? And for some reason the estimator is complaining about this:

  Array 'feat_dynamic_real' has bad shape - expected 2 dimensions, got 3.

How would you make that work?

[shalinijaiswalsj09]

if for future prediction using gluonts:

training_data = ListDataset([{"start": df_train.index[0], "target": df_train.target[:pd.to_datetime('2022-05-31')],
      'feat_dynamic_real': [df_train.feature1[:pd.to_datetime('2022-05-31')],
                            finaldf_train.feature2[:pd.to_datetime('2022-05-31')],
                            finaldf_train.feature3[:pd.to_datetime('2022-05-31')]],
        'feat_dynamic_cat': [finaldf_train.feature4[:pd.to_datetime('2022-05-31')],
                            finaldf_train.feature5[:pd.to_datetime('2022-05-31')],
                             finaldf_train.feature6[:pd.to_datetime('2022-05-31')],
                             finaldf_train.feature7[:pd.to_datetime('2022-05-31')]]
      }],
    freq="d"
    )

what will be the format of test data when future target variable is unknown:

test_data = ListDataset(
            [
                {"start": pred_set.index[0],
                "target": ,
                'feat_dynamic_real': [pred_set.feature1,
                            pred_set.feature2,
                            pred_set.feature3,
                'feat_dynamic_cat': [pred_set.feature4,
                            pred_set.feature5,
                            pred_set.feature6,
                            pred_set.feature7]
                }
            ],
            freq="d"
        )

and defined estimator format is:

estimator =  DeepAREstimator(freq="d",
                            prediction_length=1
                            , context_length=7,
                            trainer=Trainer(epochs=1), use_feat_dynamic_real=True)

Is there any issue with defined components?