JuliaMolSim / DFTK.jl

Density-functional toolkit
https://docs.dftk.org
MIT License
434 stars 89 forks source link

Use BenchmarkCI to automatically detect major runtime regressions #923

Open mfherbst opened 10 months ago

mfherbst commented 10 months ago

Since this runs in the CI we need to be a little conservative on what we time, some ideas:

Systems:

Tasks:

Probably with timing an SCF step and a single application of ε^\dagger for response on a not too trivial system we already do the key 80%.

mfherbst commented 10 months ago

Related to #542

epolack commented 10 months ago

I've just started working on that framework before seeing this issue. I can propose something if you have not yet started. It would use PkgBenchmark, and I am also thinking about flexibility to run big humongous test-cases from your testproblems. PkgBenchmark seems to be able to do the job easily coupled to @testitems. However if you want to look at older regressions (before a PR using PkgBenchmark and even before TestItemRunner), it is a bit more delicate.

mfherbst commented 10 months ago

I have not yet worked on this, no. Feel free to propose something. I don't care so much about the backend. BenchmarkCI just looked like the least hassle and pretty neat to use in CI and locally.

I think most important is simplicity and flexibility, e.g. to just run basic tests like the ones I sketched above by dropping a julia runner script into a folder, but at the same time be able to only filter a subset for faster testing on github's CI.

I also bought a bigger machine recently, which we can use for larger periodic testing.

And of course regressions should be automatically annotated in PRs, so you don't miss them.