Closed nawtrey closed 3 weeks ago
I added some benchmarks for generate_all_flux_diagrams
and calc_state_probs
. The results were pretty interesting as well --
$ asv run
Couldn't load asv.plugins._mamba_helpers because
No module named 'conda'
▒ Creating environments
▒ Discovering benchmarks
▒ Running 6 total benchmarks (1 commits * 1 environments * 6 benchmarks)
[ 0.00%] ▒ For kda commit 260ffa6d <master>:
[ 0.00%] ▒▒ Benchmarking conda-py3.9-pip+networkx-pip+numpy-pip+pytest-pip+sympy
[16.67%] ▒▒▒ Running (bench_calculations.StateProbs.time_calc_state_probs--).
[33.33%] ▒▒▒ Running (bench_diagrams.DirectionalDiagrams.time_generate_directional_diagrams--)..
[58.33%] ▒▒▒ ...ulations.StateProbs.peakmem_calc_state_probs ok
[58.33%] ▒▒▒ ============== ======= =======
-- output_strings
-------------- ---------------
graph True False
============== ======= =======
3-state 72.1M 70.5M
Hill-5-state 72M 70.2M
Hill-8-state 74.1M 71M
EmrE-8-state 80.7M 73.6M
============== ======= =======
[66.67%] ▒▒▒ ...alculations.StateProbs.time_calc_state_probs ok
[66.67%] ▒▒▒ ============== ============ ============
-- output_strings
-------------- -------------------------
graph True False
============== ============ ============
3-state 3.40▒1ms 1.60▒0.6ms
Hill-5-state 14.4▒1ms 7.14▒0.1ms
Hill-8-state 619▒20ms 65.5▒7ms
EmrE-8-state 11.0▒0.09s 393▒2ms
============== ============ ============
[75.00%] ▒▒▒ ...agrams.peakmem_generate_directional_diagrams ok
[75.00%] ▒▒▒ ============== ======= =======
-- return_edges
-------------- ---------------
graph True False
============== ======= =======
3-state 70.5M 70.3M
Hill-5-state 70.3M 70.7M
Hill-8-state 70.4M 76.2M
EmrE-8-state 73.3M 109M
============== ======= =======
[83.33%] ▒▒▒ ...lDiagrams.time_generate_directional_diagrams ok
[83.33%] ▒▒▒ ============== ============= =============
-- return_edges
-------------- ---------------------------
graph True False
============== ============= =============
3-state 1.20▒0.02ms 1.54▒0.02ms
Hill-5-state 6.20▒0.02ms 9.13▒0.08ms
Hill-8-state 54.1▒0.7ms 87.1▒1ms
EmrE-8-state 371▒4ms 621▒10ms
============== ============= =============
[91.67%] ▒▒▒ ....FluxDiagrams.peakmem_generate_flux_diagrams ok
[91.67%] ▒▒▒ ============== =======
graph
-------------- -------
3-state 70.4M
Hill-5-state 70.9M
Hill-8-state 71.4M
EmrE-8-state 76.3M
============== =======
[100.00%] ▒▒▒ ...ams.FluxDiagrams.time_generate_flux_diagrams ok
[100.00%] ▒▒▒ ============== =============
graph
-------------- -------------
3-state 832~2us
Hill-5-state 2.31▒0.02ms
Hill-8-state 16.6▒0.4ms
EmrE-8-state 203▒40ms
============== =============
The only case here that bothers me in terms of performance is the Emre-8-state
run with output_strings=True
for calc_state_probs
. It takes 27x (!!) as long as the output_strings=False
code path (11.03
s vs. 0.39
s), which is pretty surprising. I may open a performance-related issue to see if I can address that, there may yet be some performance to squeeze out of that function.
With https://github.com/Becksteinlab/kda/commit/e3505b13d698608f84e49ce019271e0a88334353 merged I've checked off the first task. I'll just note the typical workflow (locally):
Basic asv
usage:
asv run
-- run the benchmark suiteasv publish
-- generates HTML pages for viewing the resultsasv preview
-- uses HTML to allow you to view in a browserasv rm "benchmarks=*"
-- removes all benchmarks
Motivation
If any algorithmic improvements are to be made to KDA we will need some type of formal testing that includes both time-based tests and memory-based tests.
Previously with KDA I have used
kernprof
andmemory_profiler
(see GH-25), as well as ad-hoc testing (see GH-40, GH-60) to make performance comparisons. These have been very helpful in the past, but I've since become familiar with Airspeed Velocity (asv
), and it offers a good interface for measuring both peak memory and runtime performance across commits in one package.Solution
I've already done some work on this front, and locally I created a benchmark for
diagrams.generate_directional_diagrams
. Here are the initial results running onmaster
:The memory footprint when operating in terms of arrays vs.
NetworkX
graph objects is basically the same for all cases except the EmrE 8-state model, which is only marginally larger.I think moving forward this will be a very helpful tool for monitoring KDA performance over time. I'll try to open a PR for this in the next few days, I was just too excited not to post the results of the initial benchmark. I don't remember exactly how this hooks into the CI, but I don't plan on setting any requirements for PR approval regarding performance. It will only really be relevant for issues like #22.
Tasks
asv check
to the CI (handle separately)asv
tag to root README