search

wesm / pandas2

Design documents and code for the pandas 2.0 effort.
https://pandas-dev.github.io/pandas2/
306 stars 41 forks source link

DEV: Publishing benchmarks vs. trunk? #47

Open wesm opened 7 years ago

wesm commented 7 years ago

We'll want to develop a microbenchmark suite to compare pandas 2.0 perf versus 1.x, especially in microbenchmarks. What's the right tool for this? asv, vbench?

jorisvandenbossche commented 7 years ago

IMO certainly not vbench (unless someone wants to put time in developing, but even then). The benchmark suite already is ported over to asv, and I have a PR to clean them up (https://github.com/pandas-dev/pandas/pull/14099, once that is finished, I would remove the vbench ones, we don't use them anymore). But I have less of an idea how reliable it is for microbenchmarks (I think it is based on timeit, and I think there are arguments to be made that this is not really reliable for microbenchmarks)

wesm commented 7 years ago

I was thinking that ASV is the best choice -- as long as we can easily compare perf vs. a snapshot of master (e.g. we can use v0.19.0 as a baseline) so we can see how we're doing.