search

livecomsjournal / journal_information

Information and documents supporting the work of the Living Journal of Computational Molecular Sciences (LiveCoMS)
http://www.livecomsjournal.org
8 stars 4 forks source link

Develop a benchmark set (or sets) for realistic comparison of performance? #23

Open mrshirts opened 7 years ago

mrshirts commented 7 years ago

There's been some discussion, if we are going to compare performance, that we have a community organized set of simulations for benchmarks. This would probably actually be a paper developed by authors, but the journal editorial board might need to be involved to help it along.

davidlmobley commented 7 years ago

We should start with the density benchmark set from John Chodera and Kyle Beauchamp. I believe Kyle has also done some work on modernizing it/making it easy to repeat with protocol modifications, etc. As it is, they ran it with GAFF and then I ran it with SMIRNOFF, so there will be at least multiple force fields on the same set. Would be good to extend out for other codes and force fields. I can get Kyle to update us on where things stand if it's of interest.

This is on roughly 42 different pure solvents at a range of temperatures and pressures, IIRC. The data is from ThermoML. I can give a reference if needed, but not at the moment as I'm headed to a meeting.

mrshirts commented 7 years ago

I think there are actually multiple benchmarks. At least two: one for correctness, one for speed.