search

JulesBelveze / time-series-autoencoder

PyTorch Dual-Attention LSTM-Autoencoder For Multivariate Time Series
Apache License 2.0
614 stars 63 forks source link

Question about model evaluation #41

Open myalos opened 10 months ago

myalos commented 10 months ago

Thanks for sharing the code, i learned a lot from it. I see that in eval.py evaluation is performed on the target scaled with StandardScaler I think that evaluation will decrease the actual mse. I'm new to time series forecasting and don't know if it's reasonable to evaluate it on standardized data.

JulesBelveze commented 10 months ago

Hey @myalos thanks for your interest! Well my idea behind using the scaled value to compute the MSE is that if you are trying to predict a target which takes values in the range [0, 0.1] you would get a way smaller MSE than if you are trying to predict a target which ranges between 0 and 1000, right? So by scaling the values you make sure that you end up with comparable values. What do you think?

myalos commented 10 months ago

Thanks for reply. I argree with what you said. I think that mae could be better metric to compare the models and rmse in original scale could make people better understand the power of the model, since the value between [0, 1] is not very intuitive.