new_benchmark

[robinruff]

Benchmark submissions

Brief description of your algorithm

We found that there is a strong interdependency between crystal preprocessing for GNNs and GNN architectures. Our model "coGN" was optimized with respect to both aspects.

The corresponding preprint can be found on arXiv.

The coGN model is based on the Graph Network (GN) framework and is comparably simple, as it only contains MLPs as update functions, mean or sum aggregation functions and no sophisticated message-passing scheme.

This is the official submission for coGN, which was preceded by a preliminary submission #238 (the files for the preliminary submission can also be removed, when this is merged) The code and the info.json changed, but the model should be equivalent. (Minor) changes in the results are caused by a stochastic training process.

[hrushikesh-s]

Hi @robinruff,

Thank you for submitting this PR. The CI tests are failing due to the identical names of both coGN models (this and this). Please consider renaming the algorithm names to make the CI tests pass.

Additionally, please note that the current version of Matbench benchmark tasks is v0.1. Therefore, kindly rename the directory from matbench_v0.6_coGN_official to matbench_v0.1_coGN_official.

Best, Hrushikesh

[robinruff]

Hi @hrushikesh-s,

thank you for the tips. I'll fix the issues. Are we allowed to remove the preliminary coGN benchmark submission directory? I think it would make the repository less cluttered and resolve the naming conflict. As stated before, the model is equivalent, only the code got refactored and the info.json is de-anonymized.

[hrushikesh-s]

Based on the evaluation of the 9 tasks, it appears that 8 of them exhibit similar performance to the old coGN model, with only a slight variation observed in the Ex 2D materials task. Since the new model is comparable to the previous one and there are no significant differences in the Scaled Error metrics for these tasks, replacing the old model with the new one should be fine. @robinruff

Are we allowed to remove the preliminary coGN benchmark submission directory? I think it would make the repository less cluttered and resolve the naming conflict. As stated before, the model is equivalent, only the code got refactored and the info.json is de-anonymized.