search

divelab / DIG

A library for graph deep learning research
https://diveintographs.readthedocs.io/
GNU General Public License v3.0
1.82k stars 281 forks source link

Standardized MAE Calculation #151

Closed fmocking closed 1 year ago

fmocking commented 1 year ago

Hello! I'm trying to reproduce the standardized MAE values reported in SphereNet paer. The values I'm getting are slightly off on QM9 and MD17 datasets. I was wondering if it is possible to clarify the calculation (with the std value used to calculate it).

Thanks!

limei0307 commented 1 year ago

Hi @fmocking,

Thanks for your interest. We follow DimeNet to compute the standard MAE (equation 10 and 11). The most important thing is computing the standard deviation. I computed the standard deviation by myself and found that the std.MAE and logMAE values are consistent with the DimeNet paper.

Here are the computed standard deviation and steps to calculate std.MAE and logMAE. Please take a look and help us check it.

Thanks! Limei Wang

fmocking commented 1 year ago

Hi @limei0307,

Thank you so much for sharing the file to calculate the values, it is really helpful. After comparing with the values I have for std dev., there is a slight difference between the values. I assumed the std dev. was calculated over the test set. Now I see that it is calculated over all datasets rather than train or test sets. Thanks again.

Best,