tangjialiang97
commented
14 hours ago | We further conduct experiments on ETTh2 dataset and the LLM is RoBERTa-base. However, the results are still inconsistent with those reported in the paper. | Seq_len | Pred_len | Data | Mse | Mae |
|---|---|---|---|---|---|
| 96 | 96 | ETTh2 | 0.3143 | 0.3612 | |
| 96 | 192 | ETTh2 | 0.39 | 0.4062 | |
| 96 | 384 | ETTh2 | 0.4157 | 0.4326 | |
| 96 | 720 | ETTh2 | 0.4311 | 0.4456 | |
| Mean | 0.3877 | 0.4114 | |||
| Reported in the paper | 0.349 | 0.395 |
I encountered some issues while attempting to reproduce the results presented in the paper. I used the code and parameter settings provided in the public repository, but the results I obtained differ significantly from those reported in your paper. I would like to seek your guidance in understanding any potential reasons for this discrepancy and any specific settings or adjustments that might help me achieve comparable results
Training setting: model_name=LLMMixer
seq_len=96 e_layers=2 down_sampling_layers=3 down_sampling_window=2 learning_rate=0.0001 d_model=16 d_ff=32 train_epochs=20 patience=100 batch_size=128 learning_rate=0.001 pred_len=96 { python3 run.py \ --task_name long_term_forecast \ --is_training 1 \ --root_path /hss/giil/temp/data/all_six_datasets/ETT-small/ \ --data_path ETTh1.csv \ --modelid ETTh1$seqlen''96 \ --model $model_name \ --data ETTh1 \ --features M \ --seq_len $seq_len \ --label_len 0 \ --pred_len 96 \ --e_layers $e_layers \ --enc_in 7 \ --c_out 7 \ --des 'Exp' \ --itr 1 \ --d_model $d_model \ --d_ff $d_ff \ --learning_rate $learning_rate \ --train_epochs $train_epochs \ --patience $patience \ --batch_size $batch_size \ --learning_rate $learning_rate \ --down_sampling_layers $down_sampling_layers \ --down_sampling_method avg \ --down_sampling_window $down_sampling_window } sleep 1 wait { python3 run.py \ --task_name long_term_forecast \ --is_training 1 \ --root_path /hss/giil/temp/data/all_six_datasets/ETT-small/ \ --data_path ETTh1.csv \ --modelid ETTh1$seqlen''192 \ --model $model_name \ --data ETTh1 \ --features M \ --seq_len $seq_len \ --label_len 0 \ --pred_len 192 \ --e_layers $e_layers \ --enc_in 7 \ --c_out 7 \ --des 'Exp' \ --itr 1 \ --d_model $d_model \ --d_ff $d_ff \ --learning_rate $learning_rate \ --train_epochs $train_epochs \ --patience $patience \ --batch_size $batch_size \ --learning_rate $learning_rate \ --down_sampling_layers $down_sampling_layers \ --down_sampling_method avg \ --down_sampling_window $down_sampling_window } sleep 1 wait { python3 run.py \ --task_name long_term_forecast \ --is_training 1 \ --root_path /hss/giil/temp/data/all_six_datasets/ETT-small/ \ --data_path ETTh1.csv \ --modelid ETTh1$seqlen''384 \ --model $model_name \ --data ETTh1 \ --features M \ --seq_len $seq_len \ --label_len 0 \ --pred_len 384 \ --e_layers $e_layers \ --enc_in 7 \ --c_out 7 \ --des 'Exp' \ --itr 1 \ --d_model $d_model \ --d_ff $d_ff \ --learning_rate $learning_rate \ --train_epochs $train_epochs \ --patience $patience \ --batch_size $batch_size \ --learning_rate $learning_rate \ --down_sampling_layers $down_sampling_layers \ --down_sampling_method avg \ --down_sampling_window $down_sampling_window } sleep 1 wait { python3 run.py \ --task_name long_term_forecast \ --is_training 1 \ --root_path /hss/giil/temp/data/all_six_datasets/ETT-small/ \ --data_path ETTh1.csv \ --modelid ETTh1$seqlen''720 \ --model $model_name \ --data ETTh1 \ --features M \ --seq_len $seq_len \ --label_len 0 \ --pred_len 720 \ --e_layers $e_layers \ --enc_in 7 \ --c_out 7 \ --des 'Exp' \ --itr 1 \ --d_model $d_model \ --d_ff $d_ff \ --learning_rate $learning_rate \ --train_epochs $train_epochs \ --patience $patience \ --batch_size $batch_size \ --learning_rate $learning_rate \ --down_sampling_layers $down_sampling_layers \ --down_sampling_method avg \ --down_sampling_window $down_sampling_window } sleep 1 wait