search

jenstroeger / python-package-template

An opinionated Python package/application template repository, with SLSA and SBOM support built in, enabled for security scanners, code linters, typing, testing and code coverage monitoring, and release automation for reproducible builds.
MIT License
34 stars 11 forks source link

Consider tracking performance across package releases. #563

Open jenstroeger opened 1 year ago

jenstroeger commented 1 year ago

We currently have no way to measure the performance of some functions in the package, or to track performance regressions of said functions.

Packages like pytest-benchmark plug directly into pytest, but Python also comes with its own profiler and visualizer.

There’s a related discussion here: https://github.com/python/mypy/issues/14187

danirivas commented 1 year ago

At risk of getting repetitive, I strongly believe benchmarking is crucial to, at least, know where we're at. First rule of optimization: Measure, don't guess.

That being said, I LOVE those stats and histograms pytest-benchmark generates. However, AFAIU the call to benchmark has to be explicit for each test we'd want to benchmark. Would it make sense to expose a decorator in the template such that tests can be easily wrapped into a benchmark call?

Regarding the Python profilers, I have mixed feelings. I use them often and extensively but I've found it to not play particularly well with multiple processes. And I'm yet to find a way to easly profile memory in Python.