How do I include scaler for the target series when performing backtest or historical forecasts?

[joewong82]

Looking at the example in the quick start section, I understand that we need to scale the target series when there are multiple time series, e.g., train_air and train_milk. However, how do I implement this in the context of backtest and historical forecast? I am using N-BEATS. As it repeatedly builds the training set using the expanding window strategy, how do I construct a scaler using the respective training set and use it to transform the training and validation set? I am thinking about adding an encoder but it looks like that would scale the covariates instead. Any advice would be much appreciated. Thanks.

[dennisbader]

It's not possible to that on the individual train/test sets within historical_forecasts(). Do you use retrain=True for historical forecasts? If not, then it's totally fine to use the same scaled series that were used for training. As otherwise, the values for each train/test sets would not represent the initial scaling space anymore.

[joewong82]

Yeah I set retrain=True since I was using it for rolling cross validation. But not a big issue as I can easily implement a for loop to step through the data set myself. Thanks.

[bgonzalezfractal]

Hi, I was looking for exactly the same @joewong82 @dennisbader, this is easily possible doing a nested function in order to define Scaler and Metric context, an example:

def backtesting_model(model, train_series, val_series, transformer, p_covs, f_covs, cov_type, metric):
    def my_metric(actual, pred):
        return metric(transformer.inverse_transform(actual), transformer.inverse_transform(pred), reduction=lambda x: x)
    ts = train_series.append(val_series)
    backtest_metric = model.backtest(
            ts,
            past_covariates=p_covs if cov_type else None,
            future_covariates=f_covs if cov_type == 'future' else None,
            start=len(ts)-len(val_series),
            forecast_horizon=3,
            stride=1,
            metric=my_metric,
            retrain=False,
        )   
    return backtest_metric[0]

Then each time you are evaluating the scaler and metric will be defined by the macro function:

# Evaluation
        mapes_val = backtesting_model(model, train_tr, val_tr, transformer, p_covs, f_covs, cov_type, mape)
        mapes_test = mape(test[eval_col], preds_i[eval_col][-len(test):], n_jobs=-1, verbose=True)
        rmses_val = backtesting_model(model, train_tr, val_tr, transformer, p_covs, f_covs, cov_type, rmse)
        rmses_test = rmse(test[eval_col], preds_i[eval_col][-len(test):], n_jobs=-1, verbose=True)

In case anyone wants to implement this directly using darts.