search

samuelcolvin / pytest-speed

Modern benchmarking library for python with pytest integration.
https://pypi.org/project/pytest-speed/
51 stars 4 forks source link

Benchmark fails with "variance requires at least two data points" #5

Open levrik opened 1 year ago

levrik commented 1 year ago

I'm trying to collect more details but maybe you have an idea already. When running on a benchmark which according to pytest_benchmark takes ~0.32 ms per iteration, it fails with the following error:

statistics.StatisticsError: variance requires at least two data points

.venv/lib/python3.9/site-packages/pytest_speed/__init__.py:99: in run
    benchmark = benchmark_collection.run_benchmark(name, group, func, *args)
.venv/lib/python3.9/site-packages/pytest_speed/benchmark.py:118: in run_benchmark
    stddev_ns=stdev(times),
../../../.pyenv/versions/3.9.10/lib/python3.9/statistics.py:797: in stdev
    var = variance(data, xbar)

Another benchmark which is faster per iteration doesn't raise this error at all.

levrik commented 1 year ago

It looks like the benchmark simply times out, pushing only a single number to times list in run_benchmark method which isn't supported when being passed to stdev further down in the method. I guess that too many iterations are being executed per round. I'll be trying to dig deeper tomorrow.

samuelcolvin commented 1 year ago

Makes sense - I've never run a benchmark that takes the entire allowed time.

PR welcome to fix this.

levrik commented 1 year ago

I saw that pytest-benchmark only runs a single iteration per round for "slow" running tests.

https://github.com/ionelmc/pytest-benchmark/blob/728752d2976ef53fde7e40beb3e55f09cf4d4736/src/pytest_benchmark/fixture.py#L297-L301

WDYT about implementing this? I would be open to prepare a PR.

samuelcolvin commented 1 year ago

That's basically the same as fixing the above error I guess.

samuelcolvin commented 1 year ago

Happy to review a PR.