Kaimaoge
commented
3 years ago Hi,
For METR-LA dataset, the error is caused by the fact that the missing node in our paper is not consistent with the missing data in the github repository. Would you mind trying the script (https://github.com/Kaimaoge/IGNNK/blob/master/IGNNK_train.py) and missing node (https://github.com/Kaimaoge/IGNNK/blob/master/data/metr/unknow_infer.npy) in the attached files?
For NREL, it is caused by the inconsistent missing version of python 3.6 and 3.8 (some bugs in our code). Please follow our new demo (IGNNK/Demo_NREL.ipynb at master · Kaimaoge/IGNNK (github.com) https://github.com/Kaimaoge/IGNNK/blob/master/Demo_NREL.ipynb), which contains a pre-trained model that archives better results than the ones in our paper.
For USHCN, it is normal. We found IGNNK can not beat Okriging in this task, and we just pick up a very good result after several rounds of experiments.
For PeMS-Bay and Sedata, it should be noted that we use all the time periods to train our model and all the time period to evaluate our model in the transfer learning experiments.
A friend asks the same question, and the above ones are the answer. We did not check whether the problem for PeMS-Bay can be solved; we will check it if you still can not reproduce similar results by setting all-time periods as test data.
In addition, please also check whether the hidden layer uses ReLU as activation function. If not, please download the code again.
On Mon, Jul 12, 2021 at 6:00 AM zhengchuanpan @.***> wrote:
A great work, and thanks the authors for sharing the code. However, I have some questions about the model performance. I trained this model using the code and got the following results: on the METR-LA dataset, RMSE: 10.376, MAE: 6.829, R2: 0.806 on the NREL dataset, RMSE: 5.697, MAE: 3.975, R2: 0.729 on the USHCN dataset, RMSE: 3.494, MAE: 2.123, R2: 0.736 on the PeMS-BAY dataset, RMSE: 6.608, MAE: 3.826, R2: 0.525, MAPE: 8.9% Those results are much worse than the results reported in the paper. Even in the Demo_METR-LA.ipynb provided by the authors, the best results are RMSE: [10.40374277], MAE: [6.7628467], R2: [0.80512759], which are much worse than the paper reported. While in the SeData dataset, I got the results as RMSE: 6.664, MAE: 4.222, R2: 0.720, which are similar and even better than the results in the paper. So, how to train this model on the METR-LA, NREL, USHCN, and PeMS-BAY datasets to achieve similar results as reported in the paper.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Kaimaoge/IGNNK/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSYV3KJZ4ORNFGNM3HKB3DTXK4LFANCNFSM5AGQCNCA .
-- Kind regards
Wu Yuankai
zhengchuanpan
A great work, and thanks the authors for sharing the code. However, I have some questions about the model performance. I trained this model using the code and got the following results: on the METR-LA dataset, RMSE: 10.376, MAE: 6.829, R2: 0.806 on the NREL dataset, RMSE: 5.697, MAE: 3.975, R2: 0.729 on the USHCN dataset, RMSE: 3.494, MAE: 2.123, R2: 0.736 on the PeMS-BAY dataset, RMSE: 6.608, MAE: 3.826, R2: 0.525, MAPE: 8.9% Those results are much worse than the results reported in the paper. Even in the Demo_METR-LA.ipynb provided by the authors, the best results are RMSE: [10.40374277], MAE: [6.7628467], R2: [0.80512759], which are much worse than the paper reported. While in the SeData dataset, I got the results as RMSE: 6.664, MAE: 4.222, R2: 0.720, which are similar and even better than the results in the paper. So, how to train this model on the METR-LA, NREL, USHCN, and PeMS-BAY datasets to achieve similar results as reported in the paper.