search

QData / spacetimeformer

Multivariate Time Series Forecasting with efficient Transformers. Code for the paper "Long-Range Transformers for Dynamic Spatiotemporal Forecasting."
https://arxiv.org/abs/2109.12218
MIT License
784 stars 185 forks source link

Strange predictions. #28

Open asmodaay opened 2 years ago

asmodaay commented 2 years ago

Hi! Do you have any ideas about strange shift in predictions? I mean the shape of prediction is pretty well and looks like target, but the distance between context and prediction is too big. media_images_test_prediction_plots_34_7 media_images_test_prediction_plots_29_3 media_images_test_prediction_plots_29_2

qiyanjun commented 2 years ago

@jakegrigsby can you check this?

jakegrigsby commented 2 years ago

Hmm, there's not a lot to go on here but possibly this is a normalization issue? Is that one of our datasets or are you trying your own? It doesn't seem normalized... or maybe it is normalized but is really non-stationary so that the validation data from the end of the timeseries is much larger than what was seen during training (common problem in domains where things generally go up over time... finance for example). That might lead to the correct shape but with predictions that are too low like you're seeing here. We do have an extra trick you can try to deal with this: https://github.com/QData/spacetimeformer/blob/1ee8c8143a218b5d70a688c4929ba9bca7441914/spacetimeformer/spacetimeformer_model/spacetimeformer_model.py#L48 We have this "linear_window" concept that we don't talk about in the paper draft because we don't use it on those datasets. Basically it uses a linear combination of linear_window points to create an offset for the rest of the predictions. This lets the Transformer learn the shape while the linear model learns the starting point. https://github.com/QData/spacetimeformer/blob/1ee8c8143a218b5d70a688c4929ba9bca7441914/spacetimeformer/forecaster.py#L108-L125

I put this in because LSTNet does something a lot like this and I wanted to see how much it mattered. The answer was not much, at least on the paper datasets. But it did help a lot on an exchange rates dataset that is non-stationary.

To summarize: make sure you are normalizing the data based on training set statistics, and then try --linear_window 12 or 24.

asmodaay commented 2 years ago

Thank a lot for your advice, i will try it and return with feedback. This is really non-stationary financial data with a high growth rate, but I had same issue with your exchange dataset but in a less degree.

ebuss81 commented 2 years ago

Hey everyone, first, many thanks to all your work. Its great! I probably have a comparable question. My data is highly non-stationary and noisy sucht that the predictions are only a straight line. Can we assume that the high non-stationarity and noise of the data is probably the main cause?

prediction_plots_2002_1 e7ac0f8b-5f10-4303-a0fb-27fa73723c69

Best Eddy

jakegrigsby commented 2 years ago

Hey @ebuss81 , that's possible. That's also much noisier data than I think I've ever tested on, but I would've hoped for a better result. Any chance you can email me this dataset so I can look at this?

meerahshivn commented 2 years ago

image Hello, i'm a beginner on this field and this project seems very interesting to explore for my research. Well, using asos dataset, I obtained result as shown in the image without the error computation MAPE, is it something hard-coded and required to change or is it related any other issues dependency of version...please do advise me...