METR-LA results is different than the paper

[aprbw]

Hi,

Thank you for publishing the code. I am trying to reproduce the results for the METR-LA dataset. The loss I get is larger than the one reported in Table 1 of the paper. I just pulled the repository, added tqdm() wrapper, and ran the code with the provided commands. Following I copy the log at epoch 149 (lowest val loss):

 74%|███████▍  | 149/200 [4:50:29<1:38:08, 115.46s/it]Num of epoch: 149 
2021-10-10 04:01:59,910 - INFO - epoch complete 
2021-10-10 04:01:59,912 - INFO - evaluating now! 
2021-10-10 04:02:05,843 - INFO - Epoch [149/200] (56250) train_mae: 8.1466, val_mae: 4.0258 
2021-10-10 04:02:17,088 - INFO - Test: mae: 4.2664, mape: 0.1018, rmse: 7.6282 
2021-10-10 04:02:17,089 - INFO - Horizon 15mins: mae: 2.9517, mape: 0.0785, rmse: 6.1336 
2021-10-10 04:02:17,089 - INFO - Horizon 30mins: mae: 3.5871, mape: 0.1015, rmse: 7.6120 
2021-10-10 04:02:17,090 - INFO - Horizon 60mins: mae: 4.4443, mape: 0.1349, rmse: 9.3053 
2021-10-10 04:02:17,093 - INFO - Epoch [149/200] (56250) train_mae: 8.1466, test_mae: 4.2664, lr: 0.000005, 110.3s, 116.3s

and 153 ( when early stopping kicks in):

2021-10-10 04:09:59,141 - INFO - Epoch [153/200] (57750) train_mae: 8.1453, val_mae: 3.9499 
2021-10-10 04:09:59,141 - WARNING - Early stopping at epoch: 153 
 76%|███████▋  | 153/200 [5:00:18<1:32:15, 117.77s/it]

If there are issues with the hyperparameters, it would be great if you could update the .yaml files. Thank you.

[chaoshangcs]

Hi, thanks for your question. I quickly tested our code and run the model using the current yaml files. I could get the better results as shown below.

2021-10-11 02:17:18,755 - INFO - Epoch [55/200] (21000) train_mae: 2.8157, val_mae: 3.4930 2021-10-11 02:17:33,013 - INFO - Test: mae: 3.6760, mape: 0.0814, rmse: 6.1649 2021-10-11 02:17:33,013 - INFO - Horizon 15mins: mae: 2.6431, mape: 0.0678, rmse: 5.2074 2021-10-11 02:17:33,014 - INFO - Horizon 30mins: mae: 3.0388, mape: 0.0827, rmse: 6.2265 2021-10-11 02:17:33,014 - INFO - Horizon 60mins: mae: 3.4591, mape: 0.0992, rmse: 7.2568

I believe you could get better one after tuning the parameters about learning rate: base_lr, lr_decay_ratio and steps. If you have any question, please let me know. : )

Best.

[aprbw]

Hi, thank you for your reply.

Can you please tell us the base_lr, lr_decay_ratio, and steps, you were using to get the results in the paper?

[chaoshangcs]

Hi, thanks for your message. I used the same parameters to get the results in the paper. Due to multiple rounds of updates, the performance might be influenced a little bit. However, as shown, the performance is still similar to the original performance. If I could spare more time to retune the model, I'd like to update the parameters later. BTW, please note that we have updated the results in table 4, 5 and 6 with corresponding explanation in the appendix.

[aprbw]

BTW, please note that we have updated the results in table 4, 5 and 6 with corresponding explanation in the appendix.

Ah I missed Appendix D. Thanks for pointing that out.

[chaoshangcs]

My pleasure!