holoviz / datashader

Quickly and accurately render even the largest data.
http://datashader.org
BSD 3-Clause "New" or "Revised" License
3.3k stars 366 forks source link

Add benchmarking framework using asv #1120

Closed ianthomas23 closed 1 year ago

ianthomas23 commented 2 years ago

This PR adds a benchmarking framework to datashader using asv. There are two reasons for doing this. Firstly it allows us to quantify the performance differences within any particular commit of different values of parameters such as antialiased line_width and canvas size. Secondly it allows us to monitor the impact on performance of code changes over time, whether those code changes are intended to improve performance or not.

There is a README file in the benchmarks directory which explains how to use it. So far the benchmarks included are canvas.line for LinesAxis1XConstant, and shade both for categorical and non-categorical data. It is expected that more benchmarks will be added in due course.

Results on my dev machine today are as follows:

$ asv show master
Commit: f1fbae82 <master>

line.Line.time_LinesAxis1XConstant [onion/virtualenv-py3.8-pyct]
  2/12 failed
  ============ =========== ============ ============ ============ ============ ============
  --                                   line_width / self_intersect                         
  ------------ ----------------------------------------------------------------------------
   line_count   0 / False    0 / True    1 / False     1 / True    2 / False     2 / True  
  ============ =========== ============ ============ ============ ============ ============
      1000        failed    6.33±0.1ms    105±20ms    30.3±0.1ms    111±20ms    40.5±0.6ms 
     10000        failed    51.0±0.6ms   1.26±0.01s   273±0.5ms    1.38±0.01s    379±2ms   
  ============ =========== ============ ============ ============ ============ ============
  started: 2022-09-12 14:25:58, duration: 46.6s

shade.Shade.time_shade [onion/virtualenv-py3.8-pyct]
  ok
  ============= ============= ============ ============
  --                              how                  
  ------------- ---------------------------------------
   canvas_size      linear        log        eq_hist   
  ============= ============= ============ ============
       300       10.9±0.04ms   11.4±0.2ms   20.8±0.2ms 
       1000       126±0.6ms    134±0.3ms     231±2ms   
  ============= ============= ============ ============
  started: 2022-09-12 14:26:45, duration: 3.77s

shade.ShadeCategorical.time_shade_categorical [onion/virtualenv-py3.8-pyct]
  ok
  ============= ========= ============ ============ ============ ============
  --                                         category_count                  
  ----------------------- ---------------------------------------------------
   canvas_size     how         3            10           30          100     
  ============= ========= ============ ============ ============ ============
       300        linear   16.6±0.1ms   27.5±0.5ms   58.2±0.7ms    181±2ms   
       300         log     17.6±0.3ms   28.6±0.5ms   59.6±0.6ms   182±0.7ms  
       300       eq_hist   26.9±0.1ms   36.9±0.5ms   67.2±0.3ms   191±0.6ms  
       1000       linear   204±0.6ms    352±0.9ms     730±3ms     2.00±0.01s 
       1000        log     211±0.6ms     361±1ms      740±2ms     2.01±0.01s 
       1000      eq_hist    307±1ms      446±2ms      821±4ms     2.09±0.01s 
  ============= ========= ============ ============ ============ ============
  started: 2022-09-12 14:26:48, duration: 1.21m

Note that the two failed benchmarks are aborted on purpose as non-antialiased lines ignore the self_intersect keyword argument.