By default perf:library will not stop when it detects 10 runs that are
statistically significant. This provides an average that is not as
accurate, but decreases the time to run a benchmark from 60+ minutes per
run to as little as 3 minutes and results are roughly comparable. This
setting can also be tuned so that all benchmark iterations are forced to
run.
In addition, one SHA can be specified instead of 2. When this is
provided then derailed will assume you want this SHA run against the
most recent commit present. This could be useful if you are trying out
different changes on a branch and want to always compare the same
baseline to the latest commit in your branch.
By default perf:library will not stop when it detects 10 runs that are statistically significant. This provides an average that is not as accurate, but decreases the time to run a benchmark from 60+ minutes per run to as little as 3 minutes and results are roughly comparable. This setting can also be tuned so that all benchmark iterations are forced to run.
In addition, one SHA can be specified instead of 2. When this is provided then derailed will assume you want this SHA run against the most recent commit present. This could be useful if you are trying out different changes on a branch and want to always compare the same baseline to the latest commit in your branch.