Open nstarman opened 1 month ago
As I very recently discovered the big feature asv has which pytest-benchmark does not is the ability to do memory usage benchmarking as well.
https://github.com/bloomberg/pytest-memray is a thin wrapper connecting https://github.com/bloomberg/memray (12K stars) to pytest.
Timing tests and memory benchmark tests are often two different tests, so IMO it's fine to use 2 popular tools, one for each.
Still not clear to me whether the pytest way can benchmark over a long period of time like asv can.
There is native capability: https://pytest-benchmark.readthedocs.io/en/latest/comparing.html, but this is where #117 also comes in to ingest that saved data from pytest-benchmark
and then present that data in a more granular and explorable way. Pretty much the same way we do code coverage where pytest
measure code coverage but we use a service (codecov.io) to better present and assess the data.
Given the dissatisfaction with Codecov since it was bought out, I am still unsure...
But with common open-source reports we can switch visualization methods. For codecov the tools use the same standard and we could switch to coveralls
if we want. pytest-benchmark
is similar in that people are building off of that format since it integrates with pytest
. We can use pytest-benchmark
out of the box or augment it. I find these concerns to be in favor of using more swappable tools and stuff that hooks into standard frameworks!
What other options for actually visualising the data exist? If there are already well maintained options I would feel a lot happier about it.
On the memory vs timing thing, while I agree the benchmarks are likely to be different, running two different pytest plugins, and (presumably?) two different visualisation / reporting tools feels like more effort? Maybe that's worth it, but I am not familiar with it all enough to know.
Just to put it on the table, if we wanted we could use pytest-benchmark to define/run the benchmarks, and use asv for visualization - see discussion here - basically 'just' need to convert pytest-benchmark json to asv json.
Pandas also has a good discussion about asv https://github.com/pandas-dev/pandas/issues/45049
Looks like speed.python.org uses codespeed, not to be confused with https://codspeed.io (which apparently must be to measure the speed of fish)
Is there a reason, beyond historical, that we use
ASV
overpytest-benchmark
? Looking at the two tools,pytest-benchmark
has 1.5x more stars and 8x greater usage (as measured by GH dependencies). Alsopytest-benchmark
integrates into our existing pytest framework so that this repo might pull tests directly fromastropy
's test suite, using apytest
Mark (e.g.pytest.mark.benchmark_only
).