plumed / plumed2

Development version of plumed 2
https://www.plumed.org
GNU Lesser General Public License v3.0
324 stars 269 forks source link

Added possibility to compare multiple kernels #1031

Closed GiovanniBussi closed 4 months ago

GiovanniBussi commented 4 months ago

This is to add the possibility to simultaneously test multiple kernels with the benchmark application.

In short, one can do

plumed-runtime benchmark --kernel /path/to/libplumedKernel.so

to test a specific kernel. Alternatively

plumed-runtime benchmark --kernels /path/to/one/libplumedKernel.so:/path/to/other/libplumedKernel.so

to test two (or more) kernels. Test is alternating from one kernel to the other in random order at every step so as to make the timing homogeneous. The idea is that possible system overloading will affect equally both kernels, and the comparison will be more fair.

I attach a notebook where I show that in this manner timings are reproducible. To do so I use

plumed-runtime benchmark --kernels this:this

to compare the kernel of the current PLUMED version with itself. Since it's the same kernel, the speed should be the same. From the figure it can be seen that the timings are significantly more homogeneous when we use the multiple kernel option rather than if we compare two separate runs.

image

timings.zip

codecov-commenter commented 4 months ago

Codecov Report

Attention: Patch coverage is 1.38889% with 71 lines in your changes are missing coverage. Please review.

Project coverage is 84.12%. Comparing base (923d41b) to head (5a571e7).

:exclamation: Current head 5a571e7 differs from pull request most recent head 6358441. Consider uploading reports for the commit 6358441 to get more accurate results

Files Patch % Lines
src/cltools/Benchmark.cpp 1.38% 71 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #1031 +/- ## ========================================== - Coverage 84.18% 84.12% -0.06% ========================================== Files 614 614 Lines 56669 56710 +41 ========================================== + Hits 47709 47710 +1 - Misses 8960 9000 +40 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.