search

TommyCpp / opentelemetry-rust

OpenTelemetry API and SDK for Rust
https://docs.rs/opentelemetry
Apache License 2.0
0 stars 1 forks source link

Benchmarking batch span processor #6

Open TommyCpp opened 3 years ago

TommyCpp commented 3 years ago

Baseline

It's the current implementation, using Mutex

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [736.37 us 746.43 us 758.77 us]
                        change: [-12.015% -5.8075% +0.2269%] (p = 0.08 > 0.05)
                        No change in performance detected.
Found 2 outliers among 50 measurements (4.00%)
  1 (2.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.9227 s (6375                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [912.78 us 926.24 us 940.15 us]
                        change: [-26.831% -22.599% -18.845%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 5.1091 s (3825                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [1.3338 ms 1.3462 ms 1.3591 ms]
                        change: [-23.494% -15.897% -8.8477%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 50 measurements (6.00%)
  2 (4.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 6.5640 s (2550                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [2.7932 ms 2.8755 ms 2.9622 ms]
                        change: [-5.9656% -0.5466% +5.1050%] (p = 0.86 > 0.05)
                        No change in performance detected.
Found 4 outliers among 50 measurements (8.00%)
  3 (6.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.0s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 5.9593 s (1275                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [4.5020 ms 4.5346 ms 4.5782 ms]
                        change: [-36.961% -29.674% -21.758%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 50 measurements (8.00%)
  1 (2.00%) high mild
  3 (6.00%) high severe
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 5.4220 s (600                                                                                                           BatchSpanProcessor/with 32 concurrent task                        
                        time:   [9.0829 ms 9.1563 ms 9.2414 ms]
                        change: [-18.375% -15.488% -12.672%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high severe

Trace benchmark

Gnuplot not found, using plotters backend
start-end-span/always-sample                                                                             
                        time:   [3.3237 us 3.3370 us 3.3501 us]
                        change: [+517.41% +522.69% +527.56%] (p = 0.00 < 0.05)
                        Performance has regressed.
start-end-span/never-sample                                                                             
                        time:   [668.72 ns 670.24 ns 672.01 ns]
                        change: [+313.80% +316.24% +318.30%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 14 outliers among 100 measurements (14.00%)
  5 (5.00%) high mild
  9 (9.00%) high severe

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0298 s (682k iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [7.1765 us 7.2468 us 7.3096 us]
                        change: [+447.12% +454.10% +461.43%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
  2 (2.00%) low severe
  8 (8.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0016 s (11M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [213.42 ns 214.26 ns 215.30 ns]
                        change: [+18.254% +19.204% +20.178%] (p = 0.00 < 0.05)
                        Performance has regressed.

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0069 s (1.4M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [2.9509 us 2.9793 us 3.0101 us]
                        change: [+56.510% +58.479% +60.830%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (19M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [232.11 ns 232.98 ns 233.95 ns]
                        change: [+5.3952% +6.3716% +7.2540%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0126 s (1.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [2.4506 us 2.4971 us 2.5485 us]
                        change: [+55.970% +58.608% +61.277%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  2 (2.00%) high mild
  6 (6.00%) high severe
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0003 s (21M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [220.32 ns 220.54 ns 220.77 ns]
                        change: [-0.5896% -0.3117% -0.0442%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0129 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [3.8000 us 3.8261 us 3.8486 us]
                        change: [+17.408% +19.605% +21.895%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0012 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [273.85 ns 274.30 ns 274.79 ns]
                        change: [-73.278% -73.042% -72.679%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
TommyCpp commented 3 years ago

Async-channels

Using async-channels create's mpmc channels.

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [874.92 us 898.70 us 925.73 us]
                        change: [+11.302% +18.913% +31.195%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 50 measurements (6.00%)
  2 (4.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.4168 s (6375                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [750.50 us 767.57 us 781.88 us]
                        change: [-25.132% -21.966% -18.419%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 6.2837 s (5100                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [926.02 us 1.0427 ms 1.1733 ms]
                        change: [-32.265% -27.603% -21.226%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
  2 (4.00%) high mild
  6 (12.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 5.4831 s (3825                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [1.4492 ms 1.4608 ms 1.4696 ms]
                        change: [-51.020% -48.392% -45.599%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.3568 s (2550                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [2.4467 ms 2.4725 ms 2.5012 ms]
                        change: [-48.672% -47.217% -45.926%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) low mild
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 5.7869 s (1275                                                                                                          BatchSpanProcessor/with 32 concurrent task                        
                        time:   [4.5102 ms 4.5577 ms 4.6029 ms]
                        change: [-51.147% -50.440% -49.718%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high mild

Trace benchmark

start-end-span/always-sample                                                                             
                        time:   [470.40 ns 474.71 ns 480.32 ns]
                        change: [-86.105% -85.948% -85.778%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
  6 (6.00%) low severe
  7 (7.00%) low mild
  4 (4.00%) high severe
start-end-span/never-sample                                                                            
                        time:   [168.75 ns 169.04 ns 169.36 ns]
                        change: [-74.908% -74.832% -74.752%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low severe
  8 (8.00%) low mild

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0019 s (4.4M iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [1.1569 us 1.1708 us 1.1842 us]
                        change: [-84.375% -84.118% -83.883%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0005 s (26M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [193.12 ns 193.64 ns 194.18 ns]
                        change: [-14.138% -13.368% -12.622%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 18 outliers among 100 measurements (18.00%)
  3 (3.00%) low severe
  10 (10.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0014 s (2.9M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [1.7045 us 1.7198 us 1.7343 us]
                        change: [-44.939% -44.101% -43.218%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low severe
  5 (5.00%) low mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (21M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [231.13 ns 231.88 ns 232.99 ns]
                        change: [-4.0637% -3.0699% -1.8872%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  7 (7.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0061 s (3.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [1.5574 us 1.5925 us 1.6251 us]
                        change: [-39.649% -38.382% -37.079%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0007 s (23M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [213.96 ns 214.78 ns 215.60 ns]
                        change: [-3.8833% -3.5341% -3.1886%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 22 outliers among 100 measurements (22.00%)
  6 (6.00%) low severe
  3 (3.00%) low mild
  7 (7.00%) high mild
  6 (6.00%) high severe

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0003 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [2.7983 us 2.8120 us 2.8255 us]
                        change: [-26.867% -25.453% -23.972%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) low severe
  4 (4.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0011 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [273.13 ns 273.86 ns 274.60 ns]
                        change: [-2.9915% -1.7610% -0.8519%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) low mild
  2 (2.00%) high mild
TommyCpp commented 3 years ago

Relay message via thread

This implementation uses an unbounded crossbeam channel to send a message to a dedicated thread, where the thread then relays the message to the exporter in async.

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [563.76 us 589.51 us 618.06 us]
                        change: [-30.874% -26.045% -20.092%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 50 measurements (12.00%)
  2 (4.00%) low mild
  2 (4.00%) high mild
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.6841 s (7650                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [717.01 us 719.22 us 722.06 us]
                        change: [-26.555% -23.393% -20.053%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
  1 (2.00%) low mild
  2 (4.00%) high mild
  5 (10.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 5.9721 s (6375                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [960.43 us 965.29 us 970.95 us]
                        change: [-30.879% -28.904% -26.468%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  1 (2.00%) high mild
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 6.2065 s (3825                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [1.6212 ms 1.6277 ms 1.6350 ms]
                        change: [-45.144% -42.513% -39.689%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 50 measurements (10.00%)
  1 (2.00%) low mild
  2 (4.00%) high mild
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.5828 s (2550                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [2.5997 ms 2.6474 ms 2.7094 ms]
                        change: [-42.897% -41.019% -39.159%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.6s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 6.5905 s (1275                                                                                                          BatchSpanProcessor/with 32 concurrent task                        
                        time:   [5.5081 ms 5.5455 ms 5.5788 ms]
                        change: [-41.664% -40.795% -39.963%] (p = 0.00 < 0.05)
                        Performance has improved.

Trace benchmark

start-end-span/always-sample                                                                            
                        time:   [481.14 ns 485.30 ns 489.27 ns]
                        change: [-86.535% -86.248% -85.973%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low severe
  9 (9.00%) low mild
start-end-span/never-sample                                                                            
                        time:   [176.38 ns 176.95 ns 177.47 ns]
                        change: [-73.528% -73.442% -73.358%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0025 s (2.7M iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [1.5561 us 1.5714 us 1.5870 us]
                        change: [-78.917% -78.591% -78.258%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0008 s (23M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [199.82 ns 200.00 ns 200.21 ns]
                        change: [-10.304% -9.5578% -8.8123%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0051 s (2.0M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [2.3176 us 2.3313 us 2.3459 us]
                        change: [-23.575% -22.399% -21.305%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high severe
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0005 s (21M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [219.61 ns 220.01 ns 220.42 ns]
                        change: [-8.8478% -8.0600% -7.2765%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0018 s (2.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [1.8282 us 1.8375 us 1.8463 us]
                        change: [-29.704% -28.325% -26.824%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) low severe
  6 (6.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0001 s (21M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [223.30 ns 223.66 ns 224.02 ns]
                        change: [+0.9638% +1.1981% +1.4397%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0172 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [3.1789 us 3.1948 us 3.2104 us]
                        change: [-14.686% -13.092% -11.468%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0009 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [257.17 ns 257.75 ns 258.42 ns]
                        change: [-7.4291% -6.1687% -5.1378%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe
TommyCpp commented 3 years ago

Runtime channel(Tokio)

This implementation uses tokio's mpsc channel.

Benchmark

Batch span processor benchmark

BatchSpanProcessor/with 1 concurrent task                        
                        time:   [763.66 us 777.63 us 794.86 us]
                        change: [-5.5082% -0.8943% +3.4321%] (p = 0.72 > 0.05)
                        No change in performance detected.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.1869 s (7650                                                                                                           BatchSpanProcessor/with 2 concurrent task                        
                        time:   [683.31 us 689.33 us 695.83 us]
                        change: [-30.661% -28.072% -25.529%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
  1 (2.00%) low severe
  3 (6.00%) low mild
  2 (4.00%) high mild
  2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 6.0023 s (6375                                                                                                           BatchSpanProcessor/with 4 concurrent task                        
                        time:   [872.21 us 877.92 us 884.16 us]
                        change: [-35.918% -32.276% -27.417%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 50 measurements (12.00%)
  1 (2.00%) low mild
  2 (4.00%) high mild
  3 (6.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 5.6743 s (3825                                                                                                           BatchSpanProcessor/with 8 concurrent task                        
                        time:   [1.4650 ms 1.4771 ms 1.4894 ms]
                        change: [-50.676% -48.612% -46.793%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 50 measurements (6.00%)
  3 (6.00%) high mild
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.7458 s (2550                                                                                                          BatchSpanProcessor/with 16 concurrent task                        
                        time:   [2.6165 ms 2.6337 ms 2.6510 ms]
                        change: [-45.358% -43.865% -42.602%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
  1 (2.00%) high mild
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.8s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 6.7980 s (1275                                                                                                          BatchSpanProcessor/with 32 concurrent task                        
                        time:   [5.1859 ms 5.2923 ms 5.4093 ms]
                        change: [-43.492% -42.330% -41.255%] (p = 0.00 < 0.05)
                        Performance has improved.

Trace benchmark

start-end-span/always-sample                                                                            
                        time:   [439.83 ns 444.14 ns 448.29 ns]
                        change: [-86.724% -86.631% -86.544%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) low severe
  3 (3.00%) low mild
start-end-span/never-sample                                                                            
                        time:   [174.11 ns 174.33 ns 174.54 ns]
                        change: [-74.100% -74.022% -73.949%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low severe
  4 (4.00%) low mild
  1 (1.00%) high mild

Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0005 s (4.1M iter                                                                                                          start-end-span-4-attrs/always-sample                        
                        time:   [1.1029 us 1.1181 us 1.1347 us]
                        change: [-84.832% -84.602% -84.372%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (27M iterat                                                                                                          start-end-span-4-attrs/never-sample                        
                        time:   [187.30 ns 187.55 ns 187.77 ns]
                        change: [-16.726% -15.987% -15.251%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  5 (5.00%) low severe
  10 (10.00%) low mild

Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0036 s (2.7M iter                                                                                                          start-end-span-8-attrs/always-sample                        
                        time:   [1.7884 us 1.8112 us 1.8354 us]
                        change: [-42.353% -41.313% -40.264%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  8 (8.00%) high mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0006 s (22M iterat                                                                                                          start-end-span-8-attrs/never-sample                        
                        time:   [221.78 ns 222.11 ns 222.41 ns]
                        change: [-8.7823% -7.9412% -7.1224%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low severe
  4 (4.00%) low mild

Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0038 s (2.                                                                                                          start-end-span-all-attr-types/always-sample                        
                        time:   [1.6476 us 1.6681 us 1.6882 us]
                        change: [-36.034% -34.687% -33.448%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0004 s (25M                                                                                                          start-end-span-all-attr-types/never-sample                        
                        time:   [212.80 ns 213.34 ns 213.90 ns]
                        change: [-6.0818% -5.3116% -4.4650%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0070 s                                                                                                           start-end-span-all-attr-types-2x/always-sample                        
                        time:   [2.9687 us 2.9791 us 2.9889 us]
                        change: [-22.268% -20.748% -19.142%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) low severe
  3 (3.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0008 s (                                                                                                          start-end-span-all-attr-types-2x/never-sample                        
                        time:   [265.47 ns 266.35 ns 267.49 ns]
                        change: [-5.3977% -4.1198% -3.1305%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) low severe
  8 (8.00%) low mild
TommyCpp commented 3 years ago
Benchmark Baseline(Current implementation) mpmc channel new thread Tokio channel
BSP/1 task 746.43us 898.70us(+18%) 589.51us(-26%) 777.63us(-0.8%)
BSP/2 tasks 926.24us 767.57us(-21%) 719us(-23%) 689.33us(-28.072%)
BSP/4 tasks 1.3ms 1.04ms(-27%) 965us(-28%) 877.92us(-32.27%)
BSP/8 tasks 2.8ms 1.46ms(-48%) 1.62ms(-42%) 1.47ms(-48%)
BSP/16 tasks 4.5ms 2.47ms(-47%) 2.64ms(-41%) 2.6337ms(-43%)
BSP/32 tasks 9.1ms 4.5ms(-50%) 5.54ms(-40%) 5.2923ms(-42%)
start-end-span/always-sample 3.37us 474.71ns(-85%) 485.30ns(-86%) 444.14ns(-86%)
start-end-span/never-sample 670.24ns 169.04ns(-74%) 176.95ns(-73%) 174.33ns(-74%)
start-end-span-4-attrs/always-sample 7.24us 1.1708us(-84%) 1.57us(-78%) 1.1181us(-84%)
start-end-span-4-attrs/never-sample 214.26ns 193.64ns(-13%) 200.00ns(-9%) 187.55ns(-15.987%)
start-end-span-8-attrs/always-sample 2.97us 1.71us(-44%) 2.33us(-22%) 1.81us(-41.313%)
start-end-span-8-attrs/never-sample 232.98ns 231.88ns(-3.06%) 220.01ns(-8.06%) 222.11ns(-7.9%)
start-end-span-all-attr-types/always-sample 2.49us 1.5us(-38%) 1.8375us(-28.325%) 1.6681us(-34%)
start-end-span-all-attr-types/never-sample 220.54ns 214ns(-3%) 223.66ns(+1.19%) 213.34ns(-5.311%)
start-end-span-all-attr-types-2x/always-sample 3.82us 2.81us(-25%) 3.1948us(-13.092%) 2.9791us(-20%)
start-end-span-all-attr-types-2x/never-sample 274.30ns 273.86ns(-1.76%) 257.74ns(-6.1%) 266.35ns(-4.1198%)