search

yoshall / AirFormer

PyTorch implementation of AirFormer, AAAI-23
99 stars 22 forks source link

Model performance is very different from AirFormer paper. #2

Closed alien2327 closed 9 months ago

alien2327 commented 1 year ago

Hi, I just tested your code with data which you provided, but the average performance is very different from published paper.

In the paper, you mentioned that the average metrics of Airformer is,

0-7(1-24h) 8-15(25-48h) 16-23(49-72h) Sudden Changes
MAE 16.03 21.65 23.64 54.92
RMSE 32.36 44.67 50.22 90.15

but when I tried to run your code with hyperparameters which you also mentioned in the paper in Implementation Details section, the results of 25 run was,

0-7(1-24h) 8-15(25-48h) 16-23(49-72h) Sudden Changes
MAE 49.40 ± 12.51 61.10 ± 4.57 56.97 ± 1.71 68.39 ± 4.18
RMSE 62.45 ± 10.49 82.88 ± 3.54 83.19 ± 2.16 90.68 ± 4.11

I use the AIR_TINY dataset in this repository, and run main.py file as README instruction.

Could you explain why the model performance is so different?

yoshall commented 9 months ago

The dataset is just sample data for reproducibility, instead of the original one.