search

arbeitsgruppe-digitale-altnordistik / Sammlung-Toole

A new look on Handrit.is data
https://arbeitsgruppe-digitale-altnordistik.github.io/Sammlung-Toole/
MIT License
0 stars 0 forks source link

Add automated benchmarks to track tool performance #106

Open BalduinLandolt opened 2 years ago

BalduinLandolt commented 2 years ago

In order to be able to optimize things (e.g. #105 ), we need metrics on how well our tool performs. Otherwise optimizations are "hit or miss".

One option would be to directly integrate benchmarks into CI with github-action-benchmark.
This requires the benchmark to be set up with pytest-benchmark, the process is described here.

kraus-s commented 2 years ago

Sounds reasonable to me. I'd be happy to learn how to do it, but if you don't want to wait for me to figure out how to do it first, fair enough :D

On Sat, 30 Jul 2022, 14:07 Balduin Landolt, @.***> wrote:

In order to be able to optimize things (e.g. #105 https://github.com/arbeitsgruppe-digitale-altnordistik/Sammlung-Toole/issues/105 ), we need metrics on how well our tool performs. Otherwise optimizations are "hit or miss".

One option would be to directly integrate benchmarks into CI with github-action-benchmark https://github.com/benchmark-action/github-action-benchmark. This requires the benchmark to be set up with pytest-benchmark https://pytest-benchmark.readthedocs.io/en/stable/, the process is described here https://github.com/benchmark-action/github-action-benchmark/blob/master/examples/pytest/README.md .

— Reply to this email directly, view it on GitHub https://github.com/arbeitsgruppe-digitale-altnordistik/Sammlung-Toole/issues/106, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANZXABCUU2GIGTFLYSJMAHDVWULHJANCNFSM55DD7VKQ . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

BalduinLandolt commented 2 years ago

This issue has no rush IMO, and I will have to learn just as much... but I'd be happy to tackle it together. (But I think we should try to focus on getting an acceptable 1.0 out there before fall, so maybe we shouldn't let our work be guided by what's most interesting... even though it's tempting ^^)

epompeii commented 1 year ago

Another option is a tool I've been working on, Bencher: https://github.com/bencherdev/bencher It has all of the benefits of github-action-benchmark plus quite a bit more. I would be more than happy to help with any integration!

BalduinLandolt commented 1 year ago

Another option is a tool I've been working on, Bencher: https://github.com/bencherdev/bencher It has all of the benefits of github-action-benchmark plus quite a bit more. I would be more than happy to help with any integration!

Thank you for the hint! Looks promising indeed.
We are currently not progressing much (all have other things on our plates), so I'm not sure how soon we tackle this issue. But once we do, we'll definitely look into bencher - and possibly ask you for help with the integration ;-)