Add the ability to benchmark locally

MFlowCode / MFC

Exascale simulation of multiphase/physics fluid dynamics

https://mflowcode.github.io

MIT License

132 stars 56 forks source link

Add the ability to benchmark locally #416

Closed sbryngelson closed 1 month ago

sbryngelson commented 1 month ago

Add the ability to run ./mfc.sh bench (or a variant) from a local computer.

Runs locally and gets benchmark simulation execution times in .out or .yml file
Clones master and gets those times in .out or .yml file
Diffs them, just like current ./mfc.sh bench does for CI

I'm currently running ./mfc.sh bench —o name_here, which "benchmarks" the benchmark cases on your local copy and writes the output to name_here, but it doesn't clone the master branch and compare against that.

We would probably want to add an additional benchmark flag for local benchmarking because I think the above version is used in CI. So, something like ./mfc.sh bench --local would do the trick.

henryleberre commented 1 month ago

The intention was that you run ./mfc.sh bench on a reference branch/commit and use the resulting benchmark file thereafter. I am not sure that implementing this automation is worth the cost. It feels easier to use this method. Your automation script would have to use git merge-base between the current branch and upstream to find the "reference" official commit.

sbryngelson commented 1 month ago

This seems reasonable, though I'm not sure I understand fully what you mean. I see that doing a git call and re-running everything is a burden that most probably do not want. I guess you're saying that one should clone two branches or forks, then run ./mfc.sh bench on both and compare the output yourself?

henryleberre commented 1 month ago

This seems reasonable, though I'm not sure I understand fully what you mean. I see that doing a git call and re-running everything is a burden that most probably do not want. I guess you're saying that one should clone two branches or forks, then run ./mfc.sh bench on both and compare the output yourself?

Yes that is pretty much what I was saying. You checkout master once, run ./mfc.sh bench and keep the results in a file named something like "bench-master.yml". Thereafter, you use ./mfc.sh bench and ./mfc.sh bench to compare your tree against master (using the benchmark file generated earlier).

I was also trying to say that writing a script that automates this process would actually need to handle a few edge cases that might be burdensome to enumerate, discover, and handle.

sbryngelson commented 1 month ago

Got it - I agree. Closing.