Mixed past and future covariates

[Eslsamu]

Many problems have a mix of covariate time series which are known and unknown for the future. For example: We want to predict the hourly price of energy on the electricity market. The weather forecast, custom holidays, weekdays and specific events are known in the future. Demand or trading volume is unknown though.

The ideal solution would be a model that can forecast the values of the unknown covariates in the future and then treat all covariates as known in the future. Could this be integrated? Is there an easy workaround with the current setup?

And how would backtesting work?

[rmenzie]

if they are unknown in the future i believe the only method would be to treat them as exogenous and model them as multivariate. then all of your problems would be solved but it would significantly reduce the number of models for you to choose from.

[Eslsamu]

Yes I figured this out just right now as well and I am testing this to confirm it right now.

Maybe the documentation could mention that the multivariate variables used in the "target series" are used to forecast "each other" as well. Or perhaps this was obvious and I was not able to understand it. Either way, this thread can be closed.

[Eslsamu]

Can anyone actually show an example of this working? I am just getting completely random results when doing multivariate model in the scenario I was describing before.

[hrzn]

Hi @Eslsamu indeed this is an important question. Currently Darts is breaking this into 3 notions ) The "target" is the time series we are interested to forecast. It can be univariate or multivariate. Some (most) multivariate models will be able to jointly capture the way the dimensions interact, but some other models may be more naive ) The "past-observed" covariates, which are the (univariate or multivariate) covariate series that need to be observed (and hence are not known in the future. *) The "future-known" covariates, which are covariates that can be known in the future (e.g. weather forecasts, holidays, ...).

(In addition there might also be static covariates, which are not supported at the moment but we're working on it).

Not all models support everything, and we try to give a summary in the README here: https://github.com/unit8co/darts/

I think the approach you have in mind would consist in forecasting past-observed covariates (using any model) and then treat those as "future-known" covariates, and forecast the target based on these other covariates series using a RegressionModel.

We are still working on trying to generalise the combination of covariates supported in the different cases, and coming up with more examples (don't hesitate to let us know if you come up with an interesting example too).

[Eslsamu]

@hrzn

Thank you very much for this detailed explanation. I think you are doing great work here and I am looking forward to future extensions.

I read through the documentation multiple times, but am still not sure what exactly is happening when I am passing the covariate to the "fit" function.

Let's say I have a TCNModel (or any other DNN) and I have known and unknown future covariates. When I train the model, I am not specificying if these covariates will be known or unknown for future predictions. I basically treat them all the same.

If I am using "historical forecast" to backtest the model, how is prediction happening and the error calculated? Is it taking the the parallel-to-target covariate values (which may or not may in the actual future unknown) as an input or not?

Something else which is confusing to me (tested on the Energy Dataset): If I am using "predict" and I have trained the TCN model with covariates, then in theory I should have to provide the current covariates for all steps in the prediction. But this is not the case. In fact, if I provide them, the "predict" method generally performs with future covariates worse than without specifying covariates or just specifiying the ones which were used during training.

To visualize:

I am not sure how I should use "predict".

[hrzn]

Hi @Eslsamu , I agree that we still have some room for improvements with our documentation and the clarity with which some of this is exposed; it's also still (somewhat) moving as we are refining these concepts and still in the process of making everything as simple as possible.

To answer your question: the covariates you provide to predict() are automatically split behind the scenes into historic and future covariates, based on their time axis, as well as on the last timestamp of the target series you're trying to forecast. Basically, as long as you are making predictions for horizons n that are such that n <= output_chunk_length, there is no assumption that the covariates are known in advance. However, if you call predict() with n > output_chunk_length, then the forecast will only work if you provide covariates that are known sufficiently in advance (and hence these cases are assuming these covariates are future-known). The story is the same for backtesting, which just repeatedly calls predict(). RNNModels are basically a special case with output_chunk_length=1 and always requiring n future-known covariates in input.

Concerning your TCN example: the accuracy might very well drop when you provide covariates to the model. It either means that the covariates don't contain relevant signal or that the model is not able to capture it.

As explained above, you can expect some changes & improvements on this front in the upcoming releases as we'll try to better and more clearly capture the different kinds of covariates.

[Eslsamu]

Hello @hrzn Overall this project is really amazing. There is no other publicly available library with such a great support for forecasting using deep learning.

I switched last minute to this repository after not getting great result with AWS forecast (which was paid). And it is going much better with this so far.

"As explained above, you can expect some changes & improvements on this front in the upcoming releases as we'll try to better and more clearly capture the different kinds of covariates." -> Actually they are granger-causing and they overall improve the model performance when I use "historic backtest". But with "predict" it is somehow better to not use any series as a parameters than using the known future covariates. For which I don't have an explanation right now and which was the reason why I asked whether a model uses future covariates by default or not.

[pennfranc]

@Eslsamu Could you show us the code you used to train the model? If you only train on one time series (with corresponding covariates), the time series in the past and the covariates are saved, so they can be used at predict time without having to pass them to the predict function again. So if you trained your model on covariates, you are also using them at predict time, otherwise you would get an error. However, this doesn't explain why you get a different result when you pass them to the predict function explicitly. Could it be that they have somehow been altered between training and predicting?

Edit: Just to clarify, what gets saved after fitting the model are the past covariates. If your forecast horizon is not bigger than the output_chunk_length parameter of your TCN model, this is all you need to make a prediction. Future covariates only come into play when that is not the case as @hrzn said above.

[Eslsamu]

@pennfranc "However, this doesn't explain why you get a different result when you pass them to the predict function explicitly. Could it be that they have somehow been altered between training and predicting?" --> Yes in the first case they were corresponding to the training set (4/5th of the series) and during the prediction I passed the full series.

I will make a minimum example of this when I find time in the next days. For my project I am using a quite large dataset, so it is not optimal to showcase it here.

Edit: I understood the concept now. Future known covariates actually do not have to be given in parallel to the target in the forecast horizon n (output_chunk) except for the RNNModels. TCN/Transformer/NBeats will only take the past as an input as defined in "input_chunk_length". So if future covariates are known, then they should be given as extra columns beforehand. For example: weather now | weather +1 day | ... | weather + n days That is why we only need to provide an extended future covariate dataset if we predict more than n steps.

This is actually mentioned in the documentation, but it just took a while for me to grasp in practice.

[hrzn]

Hi @Eslsamu ,

We have refactored and improved the way past and future covariates are treated with Darts, starting in version 0.10.0. Now each model supporting covariates either supports past_covariates or future_covariates (or both). This way, you know that only past values of past_covariates are used, whereas future values of future_covariates will have to be provided at prediction time. We wrote a small article with an example to demonstrate how it works in practice: https://medium.com/unit8-machine-learning-publication/time-series-forecasting-using-past-and-future-external-data-with-darts-1f0539585993

Don't hesitate to let us know if you have any feedback. Hopefully it clarifies and somewhat simplifies how the covariates series are used.

[Eslsamu]

Hi @Eslsamu ,

We have refactored and improved the way past and future covariates are treated with Darts, starting in version 0.10.0. Now each model supporting covariates either supports past_covariates or future_covariates (or both). This way, you know that only past values of past_covariates are used, whereas future values of future_covariates will have to be provided at prediction time. We wrote a small article with an example to demonstrate how it works in practice: https://medium.com/unit8-machine-learning-publication/time-series-forecasting-using-past-and-future-external-data-with-darts-1f0539585993

Don't hesitate to let us know if you have any feedback. Hopefully it clarifies and somewhat simplifies how the covariates series are used.

That's amazing. I will actually try it out right away with my next project. Great work and thank you so much for considering the issue!