search

epiverse-trace / epidemics

A library of published compartmental epidemic models, and classes to represent demographic structure, non-pharmaceutical interventions, and vaccination regimes, to compose epidemic scenarios.
https://epiverse-trace.github.io/epidemics/
Other
8 stars 2 forks source link

Add a continuous benchmarking workflow #184

Closed pratikunterwegs closed 5 months ago

pratikunterwegs commented 5 months ago

This PR requests the addition of a GHA workflow for continuous benchmarking. This is expected to help avoid changes that reduce model run speed, which is important as the expected use case is 100s-1000s of runs.

An example can be found in {EpiNow2}. @sbfnk and @jamesmbaazam any feedback on your experience with using {touchstone} would be great - would you recommend using it?

sbfnk commented 5 months ago

Yes, I'd recommend it. A problem with EpiNow2 is that the benchmarks take quite long to run (1-5 minutes) and so doing repeated testing with multiple tests inflates run times pretty quickly. Because of this we have small sample sizes for the repeated tests (n=5) and get quite lot of uncertainty in the benchmarks. I think it would work really well for less computation intensive code such as in styler.

Perhaps worth noting that the package is looking for a maintainer https://github.com/lorenzwalthert/touchstone/issues/31#issuecomment-1949634987 although code contributions from others are still supported.

pratikunterwegs commented 5 months ago

Thanks @sbfnk - good to know - I'll look into adding this to the open PR.

pratikunterwegs commented 5 months ago

Just looking into this and something I should have realised earlier is that it won't be useful to add this workflow in the current PR as the function names are changing as well. I could add the old names as aliases to the functions, but that would have to be reversed eventually. Possibly best to add this workflow after the current PR is merged.