Document recommended layout for package authors

[Julian]

Hi. Thanks for writing this.

It'd be pretty helpful if the documentation recommended a layout / setup for package authors that want to write and/or ship a set of benchmarks with their package.

E.g., for jsonschema I went with this layout and have if __name__ == "__main__" blocks in a set of benchmark files, and a toxenv that runs them -- but it's not 100% ideal (for a few reasons).

Would be great if you have better suggestions for a recommended setup.

[vstinner]

doc/examples/ directory contains multiple examples. I didn't use if __name__ == "__main__" in the bench_xxx.py examples, but it should work no? What is your problem with if __name__ == "__main__"?

[Julian]

doc/examples/ directory contains multiple examples.

Cool yeah I saw those, but I meant more for shipping entire "suites" of multiple benchmarks rather than just a single one.

I didn't use if name == "main" in the bench_xxx.py examples, but it should work no? What is your problem with if name == "main"?

Nothing specifically for this case yeah, sorry, that wasn't the main point -- the main point was the above (about how/where to ship suites of benchmarks if you're writing a library) -- e.g., the kind of recommendation I meant was "if you have 5 of those, put 5 files in a benchmark directory, and here's how you run all 5 of them on an ongoing basis", etc. -- does that make more sense?

[Julian]

If not, maybe to elaborate on one of the "not ideal" reasons -- for me the way I set things up, every time I add a new benchmark I have to modify my setup, instead of e.g. automatically discovering the whole suite of benchmarks and running each one. Or more importantly I never bothered to set up tracking of multiple benchmark runs.

[vstinner]

about how/where to ship suites of benchmarks if you're writing a library

I have not idea on what is the best way to organize a benchmark suite nor how to run it. The largest benchmark suite I worked on was https://pyperformance.readthedocs.io/ which uses its own command line which uses pyperf internally.

[Julian]

Got it, ok, thanks for the pointer to pyperformance, will have a look at how you did it there --

Do you think then that documenting how to do so would be out of scope for pyperf and recommend someone make a tool on top of pyperf, or do you think that pyperf should gain a suite runner at some point?

On Fri, Dec 13, 2019, 10:52 Victor Stinner notifications@github.com wrote:

about how/where to ship suites of benchmarks if you're writing a library

I have not idea on what is the best way to organize a benchmark suite nor how to run it. The largest benchmark suite I worked on was https://pyperformance.readthedocs.io/ which uses its own command line which uses pyperf internally.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/vstinner/pyperf/issues/60?email_source=notifications&email_token=AACQQXUZWADR4YMNSZIA7ELQYNSN7A5CNFSM4J2KYJNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGZUIHQ#issuecomment-565396510, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACQQXQZXT2J7WLPP7NCCHTQYNSN7ANCNFSM4J2KYJNA .

[vstinner]

If you propose a documentation PR, I can review it.

[Julian]

Perfect, will put one together.

On Sat, Dec 14, 2019, 13:40 Victor Stinner notifications@github.com wrote:

If you propose a documentation PR, I can review it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/vstinner/pyperf/issues/60?email_source=notifications&email_token=AACQQXTESZJWIJHHMEAWTNDQYTO3RA5CNFSM4J2KYJNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG4C2EI#issuecomment-565718289, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACQQXWFRCDSH4ZRVFCXVNLQYTO3RANCNFSM4J2KYJNA .

[Julian]

This has sat in my email inbox for a few weeks and I've gotten snowed in with a bunch of other stuff :/

I still hopefully want to send a PR, but I guess going to close this ticket and will just send one over when I get a spare moment (who knows when that'll be but certainly before the next time I try to write a benchmark suite again...).

[vstinner]

That's ok, no problem ;-)