search

icannotnamemyself / FAN

Apache License 2.0
6 stars 3 forks source link

It seems the results are not performing well when adjusting the train/val/test ratio. #3

Open jackyue1994 opened 1 day ago

jackyue1994 commented 1 day ago

Specifically, after changing the train/val/test ratio to 6/2/2 in norm_experiments/experiments.py.NormExperiment._init_data_loader for the ETT datasets (to match the TimesNet configuration), the performance drops. Also, I have set scale_in_train to True. When I run the following command:

bash ./scripts/run_fan_wandb.sh "DLinear" "FAN" "ETTh1" "96 192 336 720" "cuda:0" 96 "{freq_topk:4}".

The resulting MSEs for predictions at horizons {96, 192, 336, 720} are 0.432, 0.480, 0.527, and 0.651, respectively. Is there something wrong with the setup or results?

wayne155 commented 23 hours ago

The reason why I use 7/2/1 is that some very early researches in multivariate time series muli-steps forecasting field followed this setting, e.g. MTGNN. Thanks for bringing the issue, I have not yet tried this setting (6/2/2). Regarding the performance, can you compare it with FAN or without FAN? also make sure DLinear is channel independent by setting independent=True.