search

anthony-wang / CrabNet

Predict materials properties using only the composition information!
https://doi.org/10.1038/s41524-021-00545-1
MIT License
87 stars 27 forks source link

CrabNet matbench data, possible mismatch between submission notebook and results #23

Open sgbaird opened 2 years ago

sgbaird commented 2 years ago

@MahamadSalah74 I ran the matbench notebook posted on the matbench GitHub page (https://github.com/materialsproject/matbench/pull/23), and got somewhat higher MAEs for a few repeat runs (see below) compared to what was reported.

0.3683
0.3661
0.3658

The reported matbench result (0.3463) seems a bit more in line with matbench_crabnet.py which uses 300 epochs and the full train/val dataset for training. It's not a huge difference, but I'm trying to figure out if/what the discrepancy is.

Maybe I'm missing something basic. Could you comment on this?

sgbaird commented 2 years ago

Also, the CrabNet paper reports an even lower error for the matbench_expt_gap task (0.338 eV) than what's shown in the matbench submission (0.3463 eV). Maybe it is just that I didn't run it for enough epochs. Ran it for 100 epochs and got 0.3485 eV.

sgbaird commented 1 year ago

I think I was using a fresh clone of CrabNet, but this would probably need to be verified.