[Bug]: RMSE and MAE are the same in the outputted test metric

[YuanbinLiu]

Email (Optional)

Version

1.0.0

Which OS(es) are you using?

• [X] MacOS
• [ ] Windows
• [ ] Linux

What happened?

I am using trainer.test(dataloaders=test_loader) to output the accuracy of the m3gnet model, but there seems to be an issue with the output, as the RMSE is exactly the same as the MAE. I have calculated this using an ASE calculator, and confirm that the output should correspond to the MAE, which means there is a problem with the RMSE output.

Code snippet

No response

Log output

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│      test_Energy_MAE      │     8.89860725402832      │
│     test_Energy_RMSE      │     8.89860725402832      │
│      test_Force_MAE       │     9.88778305053711      │
│      test_Force_RMSE      │    13.259322166442871     │
│    test_Site_Wise_MAE     │            0.0            │
│    test_Site_Wise_RMSE    │            0.0            │
│      test_Stress_MAE      │            0.0            │
│     test_Stress_RMSE      │            0.0            │
│      test_Total_Loss      │    263.05181884765625     │
└───────────────────────────┴───────────────────────────┘

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[kenko911]

Hi @YuanbinLiu, I don't see this happening in my unit tests. How do you test it? Are you using multiple identical structures in the dataloader for testing? If that's the case, it makes sense to have the same MAE and RMSE for energy.

[YuanbinLiu]

Hi, @kenko911, this issue might be related to the batch size. I set the batch size to 1, so the MAE and RMSE are necessarily equal. However, if I set the batch size to exceed the size of the dataset, then I can achieve a mathematically strict RMSE. Sounds sensible?

[kenko911]

Hi @YuanbinLiu, I think your second statement is not fully correct. As long as you have completely different energies in your dataset and the batchsize >1, you can also achieve "a mathematically strict RMSE" even if the batch size is not larger than the size of the dataset. I hope it helps.