Call for new benchmarks

[laszewsk]

Call for new benchmarks

The goal is to develop, test, deploy and evaluate Machine Learning benchmarks on a wide range of computing platforms. These benchmarks will be a starting point for exploring new ML methods, they can be used for ranking computers and help towards a better understanding of the interaction between ML applications and the underlying hardware. The main components of each benchmark are the scientific value, dataset, implementation and documentation. Details on how to contribute a benchmark can be found in the Policy and Submission Rules documents.

In the current suite there are four benchmarks which are drawn from various scientific domains such as material, life and earth sciences. The benchmarks are written in Python and they use libraries such as TensorFlow and PyTorch. The datasets are coming from various sources which include meteorological satellites, scanning tunnelling microscopes, seismographs and DNA sequencers. The size of these datasets covers a range from tens to hundreds of GigaBytes.

[laszewsk]

Should that be send in behalf of mlcommons first and then we distribute, e.g. the text disclaimer may need to look different?

[juripapay]

The "Call for new benchmarks" is just a draft. We need to talk to Geoffrey and the rest of the group if they are happy with the text.