search

Becksteinlab / kda

Python package used for the analysis of biochemical kinetic diagrams.
GNU General Public License v3.0
5 stars 1 forks source link

CI: Add Airspeed Velocity benchmarking to CI #108

Closed nawtrey closed 3 months ago

nawtrey commented 3 months ago

Motivation

If any algorithmic improvements are to be made to KDA we will need some type of formal testing that includes both time-based tests and memory-based tests.

Previously with KDA I have used kernprof and memory_profiler (see GH-25), as well as ad-hoc testing (see GH-40, GH-60) to make performance comparisons. These have been very helpful in the past, but I've since become familiar with Airspeed Velocity (asv), and it offers a good interface for measuring both peak memory and runtime performance across commits in one package.

Solution

I've already done some work on this front, and locally I created a benchmark for diagrams.generate_directional_diagrams. Here are the initial results running on master:

$ asv run
Couldn't load asv.plugins._mamba_helpers because
No module named 'conda'
▒ Creating environments
▒ Discovering benchmarks
▒ Running 2 total benchmarks (1 commits * 1 environments * 2 benchmarks)
[ 0.00%] ▒ For kda commit 260ffa6d <master>:
[ 0.00%] ▒▒ Benchmarking conda-py3.9-pip+networkx-pip+numpy-pip+pytest-pip+sympy
[50.00%] ▒▒▒ Running (directional_diagrams.GenerateDirectionalDiagrams.time_generate_directional_diagrams--).
[75.00%] ▒▒▒ ...agrams.peakmem_generate_directional_diagrams                ok
[75.00%] ▒▒▒ ============== ======= =======
             --               return_edges
             -------------- ---------------
               graph_name     True   False
             ============== ======= =======
                3-state      70.2M   70.6M
              Hill-5-state   70.4M   70.6M
              Hill-8-state   70.9M   76.1M
              EmrE-8-state   73.5M    109M
             ============== ======= =======

[100.00%] ▒▒▒ ...lDiagrams.time_generate_directional_diagrams                ok
[100.00%] ▒▒▒ ============== ============= ============
              --                    return_edges
              -------------- --------------------------
                graph_name        True        False
              ============== ============= ============
                 3-state      1.28▒0.02ms   1.69▒0.1ms
               Hill-5-state    7.20▒0.3ms   10.1▒0.2ms
               Hill-8-state     60.3▒3ms     107▒5ms
               EmrE-8-state     420▒20ms     719▒30ms
              ============== ============= ============

The memory footprint when operating in terms of arrays vs. NetworkX graph objects is basically the same for all cases except the EmrE 8-state model, which is only marginally larger.

I think moving forward this will be a very helpful tool for monitoring KDA performance over time. I'll try to open a PR for this in the next few days, I was just too excited not to post the results of the initial benchmark. I don't remember exactly how this hooks into the CI, but I don't plan on setting any requirements for PR approval regarding performance. It will only really be relevant for issues like #22.

Tasks

nawtrey commented 3 months ago

I added some benchmarks for generate_all_flux_diagrams and calc_state_probs. The results were pretty interesting as well --

$ asv run
Couldn't load asv.plugins._mamba_helpers because
No module named 'conda'
▒ Creating environments
▒ Discovering benchmarks
▒ Running 6 total benchmarks (1 commits * 1 environments * 6 benchmarks)
[ 0.00%] ▒ For kda commit 260ffa6d <master>:
[ 0.00%] ▒▒ Benchmarking conda-py3.9-pip+networkx-pip+numpy-pip+pytest-pip+sympy
[16.67%] ▒▒▒ Running (bench_calculations.StateProbs.time_calc_state_probs--).
[33.33%] ▒▒▒ Running (bench_diagrams.DirectionalDiagrams.time_generate_directional_diagrams--)..
[58.33%] ▒▒▒ ...ulations.StateProbs.peakmem_calc_state_probs                ok
[58.33%] ▒▒▒ ============== ======= =======
             --              output_strings
             -------------- ---------------
                 graph        True   False
             ============== ======= =======
                3-state      72.1M   70.5M
              Hill-5-state    72M    70.2M
              Hill-8-state   74.1M    71M
              EmrE-8-state   80.7M   73.6M
             ============== ======= =======

[66.67%] ▒▒▒ ...alculations.StateProbs.time_calc_state_probs                ok
[66.67%] ▒▒▒ ============== ============ ============
             --                   output_strings
             -------------- -------------------------
                 graph          True        False
             ============== ============ ============
                3-state       3.40▒1ms    1.60▒0.6ms
              Hill-5-state    14.4▒1ms    7.14▒0.1ms
              Hill-8-state    619▒20ms     65.5▒7ms
              EmrE-8-state   11.0▒0.09s    393▒2ms
             ============== ============ ============

[75.00%] ▒▒▒ ...agrams.peakmem_generate_directional_diagrams                ok
[75.00%] ▒▒▒ ============== ======= =======
             --               return_edges
             -------------- ---------------
                 graph        True   False
             ============== ======= =======
                3-state      70.5M   70.3M
              Hill-5-state   70.3M   70.7M
              Hill-8-state   70.4M   76.2M
              EmrE-8-state   73.3M    109M
             ============== ======= =======

[83.33%] ▒▒▒ ...lDiagrams.time_generate_directional_diagrams                ok
[83.33%] ▒▒▒ ============== ============= =============
             --                     return_edges
             -------------- ---------------------------
                 graph           True         False
             ============== ============= =============
                3-state      1.20▒0.02ms   1.54▒0.02ms
              Hill-5-state   6.20▒0.02ms   9.13▒0.08ms
              Hill-8-state    54.1▒0.7ms     87.1▒1ms
              EmrE-8-state     371▒4ms       621▒10ms
             ============== ============= =============

[91.67%] ▒▒▒ ....FluxDiagrams.peakmem_generate_flux_diagrams                ok
[91.67%] ▒▒▒ ============== =======
                 graph
             -------------- -------
                3-state      70.4M
              Hill-5-state   70.9M
              Hill-8-state   71.4M
              EmrE-8-state   76.3M
             ============== =======

[100.00%] ▒▒▒ ...ams.FluxDiagrams.time_generate_flux_diagrams                ok
[100.00%] ▒▒▒ ============== =============
                  graph
              -------------- -------------
                 3-state        832~2us
               Hill-5-state   2.31▒0.02ms
               Hill-8-state    16.6▒0.4ms
               EmrE-8-state     203▒40ms
              ============== =============

The only case here that bothers me in terms of performance is the Emre-8-state run with output_strings=True for calc_state_probs. It takes 27x (!!) as long as the output_strings=False code path (11.03 s vs. 0.39 s), which is pretty surprising. I may open a performance-related issue to see if I can address that, there may yet be some performance to squeeze out of that function.

nawtrey commented 3 months ago

With https://github.com/Becksteinlab/kda/commit/e3505b13d698608f84e49ce019271e0a88334353 merged I've checked off the first task. I'll just note the typical workflow (locally):

Basic asv usage: