nnzhan
commented
5 years ago Hi, I thinks it just a normal fluctuation. You can try repeat the experiments 10 times and take the average. It will you give pretty close results.
Closed sshleifer closed 4 years ago
nnzhan
commented
5 years ago Hi, I thinks it just a normal fluctuation. You can try repeat the experiments 10 times and take the average. It will you give pretty close results.
ICDI0906
commented
4 years ago I ran the Forward Backward Adaptive Command:
python train.py --device cuda:0 --gcn_bool --adjtype doubletransition --addaptadj --randomadj --epoch 100 $ep --expid $expidand got what I think are slightly worse results than Table 2 and 3 of the paper.
Table 3/METR-LA:
MAE, RMSE, MAPE = 3.04, 6.09, 8.23% My results: 3.0737, 6.1674, 8.30%
Does that sound like a normal amount of error, wrong command, or bug?
Training finished The valid loss on best model is 2.7565 Evaluate best model on test data for horizon 1, Test MAE: 2.2372, Test MAPE : 0.0533, Test RMSE: 3.8697 Evaluate best model on test data for horizon 2, Test MAE: 2.5196, Test MAPE : 0.0626, Test RMSE: 4.6753 Evaluate best model on test data for horizon 3, Test MAE: 2.7171, Test MAPE : 0.0695, Test RMSE: 5.2287 Evaluate best model on test data for horizon 4, Test MAE: 2.8760, Test MAPE : 0.0754, Test RMSE: 5.6681 Evaluate best model on test data for horizon 5, Test MAE: 3.0037, Test MAPE : 0.0803, Test RMSE: 6.0149 Evaluate best model on test data for horizon 6, Test MAE: 3.1157, Test MAPE : 0.0844, Test RMSE: 6.3154 Evaluate best model on test data for horizon 7, Test MAE: 3.2154, Test MAPE : 0.0882, Test RMSE: 6.5706 Evaluate best model on test data for horizon 8, Test MAE: 3.3002, Test MAPE: 0.0913, Test RMSE: 6.7903 Evaluate best model on test data for horizon 9, Test MAE: 3.3777, Test MAP$: 0.0941, Test RMSE: 6.9856 Evaluate best model on test data for horizon 10, Test MAE: 3.4449, Test MA$E: 0.0965, Test RMSE: 7.1507 Evaluate best model on test data for horizon 11, Test MAE: 3.5081, Test MA$E: 0.0989, Test RMSE: 7.2993 Evaluate best model on test data for horizon 12, Test MAE: 3.5691, Test MA$E: 0.1011, Test RMSE: 7.4404On average over 12 horizons, Test MAE: 3.0737, Test MAPE: 0.0830, Test RMS$: 6.1674 Total time spent: 4299.2252
Have you Replicated the result described in the paper, I run the code several times, but the result is not as good as the result in the paper? look forward to your reply~
sshleifer
commented
4 years ago I also get a bit worse. 3 runs: 3.0544 | 0.0823 | 6.0684 3.0638 | 0.0824 | 6.0895 3.0737 | 0.0832 | 6.1674
I'm working on a few improvements that I'll PR and publish on Dec. 10
ICDI0906
commented
4 years ago look forward to your work~
yufann
commented
3 years ago I ran the Forward Backward Adaptive Command:
python train.py --device cuda:0 --gcn_bool --adjtype doubletransition --addaptadj --randomadj --epoch 100 $ep --expid $expidand got what I think are slightly worse results than Table 2 and 3 of the paper. Table 3/METR-LA: MAE, RMSE, MAPE = 3.04, 6.09, 8.23% My results: 3.0737, 6.1674, 8.30% Does that sound like a normal amount of error, wrong command, or bug?
Training finished The valid loss on best model is 2.7565 Evaluate best model on test data for horizon 1, Test MAE: 2.2372, Test MAPE : 0.0533, Test RMSE: 3.8697 Evaluate best model on test data for horizon 2, Test MAE: 2.5196, Test MAPE : 0.0626, Test RMSE: 4.6753 Evaluate best model on test data for horizon 3, Test MAE: 2.7171, Test MAPE : 0.0695, Test RMSE: 5.2287 Evaluate best model on test data for horizon 4, Test MAE: 2.8760, Test MAPE : 0.0754, Test RMSE: 5.6681 Evaluate best model on test data for horizon 5, Test MAE: 3.0037, Test MAPE : 0.0803, Test RMSE: 6.0149 Evaluate best model on test data for horizon 6, Test MAE: 3.1157, Test MAPE : 0.0844, Test RMSE: 6.3154 Evaluate best model on test data for horizon 7, Test MAE: 3.2154, Test MAPE : 0.0882, Test RMSE: 6.5706 Evaluate best model on test data for horizon 8, Test MAE: 3.3002, Test MAPE: 0.0913, Test RMSE: 6.7903 Evaluate best model on test data for horizon 9, Test MAE: 3.3777, Test MAP$: 0.0941, Test RMSE: 6.9856 Evaluate best model on test data for horizon 10, Test MAE: 3.4449, Test MA$E: 0.0965, Test RMSE: 7.1507 Evaluate best model on test data for horizon 11, Test MAE: 3.5081, Test MA$E: 0.0989, Test RMSE: 7.2993 Evaluate best model on test data for horizon 12, Test MAE: 3.5691, Test MA$E: 0.1011, Test RMSE: 7.4404On average over 12 horizons, Test MAE: 3.0737, Test MAPE: 0.0830, Test RMS$: 6.1674 Total time spent: 4299.2252
Have you Replicated the result described in the paper, I run the code several times, but the result is not as good as the result in the paper? look forward to your reply~
I ran the Forward Backward Adaptive Command:
python train.py --device cuda:0 --gcn_bool --adjtype doubletransition --addaptadj --randomadj --epoch 100 $ep --expid $expidand got what I think are slightly worse results than Table 2 and 3 of the paper.
Table 3/METR-LA:
MAE, RMSE, MAPE = 3.04, 6.09, 8.23% My results: 3.0737, 6.1674, 8.30%
Does that sound like a normal amount of error, wrong command, or bug?
Training finished The valid loss on best model is 2.7565 Evaluate best model on test data for horizon 1, Test MAE: 2.2372, Test MAPE : 0.0533, Test RMSE: 3.8697 Evaluate best model on test data for horizon 2, Test MAE: 2.5196, Test MAPE : 0.0626, Test RMSE: 4.6753 Evaluate best model on test data for horizon 3, Test MAE: 2.7171, Test MAPE : 0.0695, Test RMSE: 5.2287 Evaluate best model on test data for horizon 4, Test MAE: 2.8760, Test MAPE : 0.0754, Test RMSE: 5.6681 Evaluate best model on test data for horizon 5, Test MAE: 3.0037, Test MAPE : 0.0803, Test RMSE: 6.0149 Evaluate best model on test data for horizon 6, Test MAE: 3.1157, Test MAPE : 0.0844, Test RMSE: 6.3154 Evaluate best model on test data for horizon 7, Test MAE: 3.2154, Test MAPE : 0.0882, Test RMSE: 6.5706 Evaluate best model on test data for horizon 8, Test MAE: 3.3002, Test MAPE: 0.0913, Test RMSE: 6.7903 Evaluate best model on test data for horizon 9, Test MAE: 3.3777, Test MAP$: 0.0941, Test RMSE: 6.9856 Evaluate best model on test data for horizon 10, Test MAE: 3.4449, Test MA$E: 0.0965, Test RMSE: 7.1507 Evaluate best model on test data for horizon 11, Test MAE: 3.5081, Test MA$E: 0.0989, Test RMSE: 7.2993 Evaluate best model on test data for horizon 12, Test MAE: 3.5691, Test MA$E: 0.1011, Test RMSE: 7.4404On average over 12 horizons, Test MAE: 3.0737, Test MAPE: 0.0830, Test RMS$: 6.1674 Total time spent: 4299.2252
hello, I have a question: why table2's graph wavenet result is more lower than table3, but this is not the same result?
I ran the Forward Backward Adaptive Command:
and got what I think are slightly worse results than Table 2 and 3 of the paper.
Table 3/METR-LA:
MAE, RMSE, MAPE = 3.04, 6.09, 8.23% My results: 3.0737, 6.1674, 8.30%
Does that sound like a normal amount of error, wrong command, or bug?
On average over 12 horizons, Test MAE: 3.0737, Test MAPE: 0.0830, Test RMS$: 6.1674 Total time spent: 4299.2252