search

GestaltCogTeam / D2STGNN

Code for our VLDB'22 paper Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting.
158 stars 16 forks source link

reproducing PEMS08 #4

Open naivegillian opened 1 year ago

naivegillian commented 1 year ago

Hi, I am trying to reproduce the results. METR-LA and PEMS-BAY looks fine. But on PEMS08 the results is not good as expected. Is the adjacency matrix used for PEMS08 can be downloaded from https://drive.google.com/drive/folders/1H3nl0eRCVl5jszHPesIPoPu1ODhFMSub so the first 5 nodes will be

        1. 0
        1. 0
        1. 1
        1. 0
        1. 0 Also, I find there are more 'inh_layer.pos_encoder' in my trained model than the one on https://drive.google.com/drive/folders/18nkluGajYET2F9mxz3Kl6jcFVAAUGfpc Thanks
huangst21 commented 1 year ago

Hi, I am trying to reproduce the results. METR-LA and PEMS-BAY looks fine. But on PEMS08 the results is not good as expected. Is the adjacency matrix used for PEMS08 can be downloaded from https://drive.google.com/drive/folders/1H3nl0eRCVl5jszHPesIPoPu1ODhFMSub so the first 5 nodes will be 0. 0. 0. 0. 0 0. 0. 1. 0. 0 0. 1. 0. 0. 1 0. 0. 0. 0. 0 0. 0. 1. 0. 0 Also, I find there are more 'inh_layer.pos_encoder' in my trained model than the one on https://drive.google.com/drive/folders/18nkluGajYET2F9mxz3Kl6jcFVAAUGfpc Thanks

I had the same problem with the PEMS08 dataset, the experimental results were a long way from the reported results.

zezhishao commented 1 year ago

Hi guys. Thank you for your interest in D2STGNN. This repository is currently not under maintenance. You can reproduce D2STGNN based on BasicTS, which ensures the fairness of results based on a unified training/evaluation pipeline. You can also find more baselines and datasets in BasicTS. Here are the results of D2STGNN in BasicTS:

2023-08-31 12:09:26,902 - easytorch-training - INFO - Epoch 100 / 100
2023-08-31 12:11:16,473 - easytorch-training - INFO - Result <train>: [train_time: 109.57 (s), lr: 6.25e-05, train_MAE: 13.613422, train_RMSE: 22.730559, train_MAPE: 0.089726, train_WAPE: 0.059244, train_MSE: 519.687916]
2023-08-31 12:11:16,475 - easytorch-training - INFO - Start validation.
2023-08-31 12:11:26,387 - easytorch-training - INFO - Result <val>: [val_time: 9.91 (s), val_MAE: 14.225937, val_RMSE: 23.114646, val_MAPE: 0.111527, val_WAPE: 0.066082, val_MSE: 592.654739]
2023-08-31 12:11:26,438 - easytorch-training - INFO - Checkpoint checkpoints/D2STGNN_100/8407cac278902dac451f6fd1891a2471/D2STGNN_best_val_MAE.pt saved
2023-08-31 12:11:35,333 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 12.352158, Test RMSE: 19.550905, Test MAPE: 0.081585, Test WAPE: 0.053134, Test MSE: 382.237885
2023-08-31 12:11:35,364 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 12.845043, Test RMSE: 20.666191, Test MAPE: 0.085214, Test WAPE: 0.055268, Test MSE: 427.091431
2023-08-31 12:11:35,394 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 13.215744, Test RMSE: 21.467710, Test MAPE: 0.088085, Test WAPE: 0.056878, Test MSE: 460.862610
2023-08-31 12:11:35,424 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 13.593189, Test RMSE: 22.360163, Test MAPE: 0.089042, Test WAPE: 0.058517, Test MSE: 499.976868
2023-08-31 12:11:35,461 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 13.862035, Test RMSE: 22.866516, Test MAPE: 0.091356, Test WAPE: 0.059690, Test MSE: 522.877563
2023-08-31 12:11:35,485 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 14.095591, Test RMSE: 23.339945, Test MAPE: 0.093465, Test WAPE: 0.060712, Test MSE: 544.752991
2023-08-31 12:11:35,509 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 14.365138, Test RMSE: 23.942684, Test MAPE: 0.093750, Test WAPE: 0.061890, Test MSE: 573.252136
2023-08-31 12:11:35,533 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 14.566394, Test RMSE: 24.324722, Test MAPE: 0.095804, Test WAPE: 0.062775, Test MSE: 591.692139
2023-08-31 12:11:35,556 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 14.769111, Test RMSE: 24.696266, Test MAPE: 0.097954, Test WAPE: 0.063666, Test MSE: 609.905579
2023-08-31 12:11:35,580 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 14.968579, Test RMSE: 25.066189, Test MAPE: 0.098992, Test WAPE: 0.064545, Test MSE: 628.313782
2023-08-31 12:11:35,603 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 15.147799, Test RMSE: 25.366919, Test MAPE: 0.101015, Test WAPE: 0.065337, Test MSE: 643.480591
2023-08-31 12:11:35,627 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 15.400373, Test RMSE: 25.736530, Test MAPE: 0.103332, Test WAPE: 0.066446, Test MSE: 662.368958
2023-08-31 12:11:36,137 - easytorch-training - INFO - Result <test>: [test_time: 9.70 (s), test_MAE: 14.098426, test_RMSE: 23.357384, test_MAPE: 0.093299, test_WAPE: 0.060734, test_MSE: 545.567383]
2023-08-31 12:11:36,194 - easytorch-training - INFO - Checkpoint checkpoints/D2STGNN_100/8407cac278902dac451f6fd1891a2471/D2STGNN_100.pt saved
2023-08-31 12:11:36,201 - easytorch-training - INFO - The training finished at 2023-08-31 12:11:36

Kindly note that considering the randomness, we can not always get extactly the same results as reported in the paper. Reproduced results might be slightly different, better or worse.

naivegillian commented 12 months ago

Is there any difference in settings such that you can finish in 100 epoches ? I see 'standard_transform' in basicts. Thank you.