search

stan-dev / math

The Stan Math Library is a C++ template library for automatic differentiation of any order using forward, reverse, and mixed modes. It includes a range of built-in functions for probabilistic modeling, linear algebra, and equation solving.
https://mc-stan.org
BSD 3-Clause "New" or "Revised" License
737 stars 185 forks source link

Can we run the benchmarks multiple times and report mean and sd? #1345

Open SteveBronder opened 5 years ago

SteveBronder commented 5 years ago

Description

I noticed that when running the gold models like SIR multiple times it can can anywhere from 100 seconds to 150 seconds on my machine with the develop branch. That's a pretty big variance! I think for the performance benchmarks we report back via Jenkins it would be a good idea to run the models 15-20 times and then report the mean and variance of the performance benchmark for each model.

@serban-nicusor-toptal can you point me to the code that the Jenkins uses to set off the performance tests? I can also modify the compare-hash.py file to report the mean and sd. If I remember right you had something like this in the works?

Current Version:

v2.20.0

bob-carpenter commented 5 years ago

While you're at it, would you mind adding something to the output to signal whether larger or smaller numbers are better. Thanks.

serban-nicusor-toptal commented 5 years ago

@bob-carpenter There's a PR for that: https://github.com/stan-dev/performance-tests-cmdstan/pull/24

serban-nicusor-toptal commented 5 years ago

@SteveBronder From math perspective triggering a performance run goes in this order.

  1. Math triggers Stan build: https://github.com/stan-dev/math/blob/develop/Jenkinsfile#L271
  2. Stan triggers Cmdstan build: https://github.com/stan-dev/stan/blob/develop/Jenkinsfile#L194
  3. Cmdstan triggers Performance tests: https://github.com/stan-dev/cmdstan/blob/develop/Jenkinsfile#L140

Note the pr variables passed around.

Tests are running here in the Jenkinsfile:

  1. Compare against target branch
  2. Known-Good
  3. Shotgun

If I can help you with anything let me know

bob-carpenter commented 5 years ago

Thanks so much. I can never keep my signs or numerators/denominators straight in anything.