search

psf / pyperf

Toolkit to run Python benchmarks
http://pyperf.readthedocs.io/
MIT License
766 stars 74 forks source link

Having trouble while using pypy: pyperf is much slower than timeit #137

Open wangyi041228 opened 2 years ago

wangyi041228 commented 2 years ago

I'm learting Python and made some tests. I downloaded and unzipped pypy to a folder. I can't find the proper way to use pyperf with pypy. Can I get the result with JIT?

Code:

# global i, big loop of empty
## pyperf
py -3.10 -m pyperf timeit -s "global i" "for i in range(100_000_000): pass" --rigorous
Mean +- std dev: 1.95 sec +- 0.04 sec

pypy -m pyperf timeit -s "global i" "for i in range(100_000_000): pass" --rigorous
Mean +- std dev: 1.02 sec +- 0.02 sec

### The command
### Can I get the result with JIT?

## timeit
py -3.10 -m timeit -s "global i" "for i in range(100_000_000): pass"
1 loop, best of 5: 2.02 sec per loop

pypy -m timeit -s "global i" "for i in range(100_000_000): pass"
5 loops, average of 7: 46 +- 1.56 msec per loop

pypy --jit off -m timeit -s "global i" "for i in range(100_000_000): pass"
1 loops, average of 7: 6.31 +- 0.0155 sec per loop
vstinner commented 2 years ago

You should write the result into a JSON file, and then analyze the data: https://pyperf.readthedocs.io/en/latest/analyze.html

pyperf is now well tuned for JIT compilers. You may have to adjust parameters like --loops and --warmups manually to control if the JIT compiler optimized the code or not.

Measuring performance is a hard problem. Welcome to hell.

vstinner commented 2 years ago

cc @cfbolz @corona10

See also issues like #128

wangyi041228 commented 2 years ago

Thanks for your guide. I'll learn more about Python and hell.

cfbolz commented 2 years ago

I don't know, it does feel a little bit weird that pyperf gives a result that is a factor of 500x slower than timeit? --loops and --warmups shouldn't really come into it here, because the code that is being measured is a loop with 100 million iterations, so even if perf runs that exactly once, the JIT will compile things.

and if I put that code just directly in a file and run it, it takes about 200ms, just like timeit reports.

but in general @wangyi041228, I would try PyPy on some code that a) is not a test suite and b) not a microbenchmark and c) that you actually care about being fast.