Open TommyCpp opened 3 years ago
Using async-channels
create's mpmc channels.
BatchSpanProcessor/with 1 concurrent task
time: [874.92 us 898.70 us 925.73 us]
change: [+11.302% +18.913% +31.195%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 50 measurements (6.00%)
2 (4.00%) high mild
1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.4168 s (6375 BatchSpanProcessor/with 2 concurrent task
time: [750.50 us 767.57 us 781.88 us]
change: [-25.132% -21.966% -18.419%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 6.2837 s (5100 BatchSpanProcessor/with 4 concurrent task
time: [926.02 us 1.0427 ms 1.1733 ms]
change: [-32.265% -27.603% -21.226%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
2 (4.00%) high mild
6 (12.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 5.4831 s (3825 BatchSpanProcessor/with 8 concurrent task
time: [1.4492 ms 1.4608 ms 1.4696 ms]
change: [-51.020% -48.392% -45.599%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.3568 s (2550 BatchSpanProcessor/with 16 concurrent task
time: [2.4467 ms 2.4725 ms 2.5012 ms]
change: [-48.672% -47.217% -45.926%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
2 (4.00%) low mild
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 5.8s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 5.7869 s (1275 BatchSpanProcessor/with 32 concurrent task
time: [4.5102 ms 4.5577 ms 4.6029 ms]
change: [-51.147% -50.440% -49.718%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
1 (2.00%) high mild
start-end-span/always-sample
time: [470.40 ns 474.71 ns 480.32 ns]
change: [-86.105% -85.948% -85.778%] (p = 0.00 < 0.05)
Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
6 (6.00%) low severe
7 (7.00%) low mild
4 (4.00%) high severe
start-end-span/never-sample
time: [168.75 ns 169.04 ns 169.36 ns]
change: [-74.908% -74.832% -74.752%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) low severe
8 (8.00%) low mild
Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0019 s (4.4M iter start-end-span-4-attrs/always-sample
time: [1.1569 us 1.1708 us 1.1842 us]
change: [-84.375% -84.118% -83.883%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low severe
1 (1.00%) low mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0005 s (26M iterat start-end-span-4-attrs/never-sample
time: [193.12 ns 193.64 ns 194.18 ns]
change: [-14.138% -13.368% -12.622%] (p = 0.00 < 0.05)
Performance has improved.
Found 18 outliers among 100 measurements (18.00%)
3 (3.00%) low severe
10 (10.00%) low mild
4 (4.00%) high mild
1 (1.00%) high severe
Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0014 s (2.9M iter start-end-span-8-attrs/always-sample
time: [1.7045 us 1.7198 us 1.7343 us]
change: [-44.939% -44.101% -43.218%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
2 (2.00%) low severe
5 (5.00%) low mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (21M iterat start-end-span-8-attrs/never-sample
time: [231.13 ns 231.88 ns 232.99 ns]
change: [-4.0637% -3.0699% -1.8872%] (p = 0.00 < 0.05)
Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
7 (7.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
4 (4.00%) high severe
Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0061 s (3. start-end-span-all-attr-types/always-sample
time: [1.5574 us 1.5925 us 1.6251 us]
change: [-39.649% -38.382% -37.079%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0007 s (23M start-end-span-all-attr-types/never-sample
time: [213.96 ns 214.78 ns 215.60 ns]
change: [-3.8833% -3.5341% -3.1886%] (p = 0.00 < 0.05)
Performance has improved.
Found 22 outliers among 100 measurements (22.00%)
6 (6.00%) low severe
3 (3.00%) low mild
7 (7.00%) high mild
6 (6.00%) high severe
Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0003 s start-end-span-all-attr-types-2x/always-sample
time: [2.7983 us 2.8120 us 2.8255 us]
change: [-26.867% -25.453% -23.972%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
5 (5.00%) low severe
4 (4.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0011 s ( start-end-span-all-attr-types-2x/never-sample
time: [273.13 ns 273.86 ns 274.60 ns]
change: [-2.9915% -1.7610% -0.8519%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) low mild
2 (2.00%) high mild
This implementation uses an unbounded crossbeam channel to send a message to a dedicated thread, where the thread then relays the message to the exporter in async.
BatchSpanProcessor/with 1 concurrent task
time: [563.76 us 589.51 us 618.06 us]
change: [-30.874% -26.045% -20.092%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 50 measurements (12.00%)
2 (4.00%) low mild
2 (4.00%) high mild
2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.6841 s (7650 BatchSpanProcessor/with 2 concurrent task
time: [717.01 us 719.22 us 722.06 us]
change: [-26.555% -23.393% -20.053%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
1 (2.00%) low mild
2 (4.00%) high mild
5 (10.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 5.9721 s (6375 BatchSpanProcessor/with 4 concurrent task
time: [960.43 us 965.29 us 970.95 us]
change: [-30.879% -28.904% -26.468%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
1 (2.00%) high mild
1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 6.2065 s (3825 BatchSpanProcessor/with 8 concurrent task
time: [1.6212 ms 1.6277 ms 1.6350 ms]
change: [-45.144% -42.513% -39.689%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 50 measurements (10.00%)
1 (2.00%) low mild
2 (4.00%) high mild
2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.5828 s (2550 BatchSpanProcessor/with 16 concurrent task
time: [2.5997 ms 2.6474 ms 2.7094 ms]
change: [-42.897% -41.019% -39.159%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
1 (2.00%) high severe
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.6s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 6.5905 s (1275 BatchSpanProcessor/with 32 concurrent task
time: [5.5081 ms 5.5455 ms 5.5788 ms]
change: [-41.664% -40.795% -39.963%] (p = 0.00 < 0.05)
Performance has improved.
start-end-span/always-sample
time: [481.14 ns 485.30 ns 489.27 ns]
change: [-86.535% -86.248% -85.973%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
1 (1.00%) low severe
9 (9.00%) low mild
start-end-span/never-sample
time: [176.38 ns 176.95 ns 177.47 ns]
change: [-73.528% -73.442% -73.358%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
2 (2.00%) low mild
4 (4.00%) high mild
Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0025 s (2.7M iter start-end-span-4-attrs/always-sample
time: [1.5561 us 1.5714 us 1.5870 us]
change: [-78.917% -78.591% -78.258%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) low severe
3 (3.00%) low mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0008 s (23M iterat start-end-span-4-attrs/never-sample
time: [199.82 ns 200.00 ns 200.21 ns]
change: [-10.304% -9.5578% -8.8123%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) low mild
1 (1.00%) high mild
Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0051 s (2.0M iter start-end-span-8-attrs/always-sample
time: [2.3176 us 2.3313 us 2.3459 us]
change: [-23.575% -22.399% -21.305%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) low severe
2 (2.00%) low mild
1 (1.00%) high severe
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0005 s (21M iterat start-end-span-8-attrs/never-sample
time: [219.61 ns 220.01 ns 220.42 ns]
change: [-8.8478% -8.0600% -7.2765%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild
Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0018 s (2. start-end-span-all-attr-types/always-sample
time: [1.8282 us 1.8375 us 1.8463 us]
change: [-29.704% -28.325% -26.824%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
4 (4.00%) low severe
6 (6.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0001 s (21M start-end-span-all-attr-types/never-sample
time: [223.30 ns 223.66 ns 224.02 ns]
change: [+0.9638% +1.1981% +1.4397%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0172 s start-end-span-all-attr-types-2x/always-sample
time: [3.1789 us 3.1948 us 3.2104 us]
change: [-14.686% -13.092% -11.468%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0009 s ( start-end-span-all-attr-types-2x/never-sample
time: [257.17 ns 257.75 ns 258.42 ns]
change: [-7.4291% -6.1687% -5.1378%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe
This implementation uses tokio's mpsc
channel.
BatchSpanProcessor/with 1 concurrent task
time: [763.66 us 777.63 us 794.86 us]
change: [-5.5082% -0.8943% +3.4321%] (p = 0.72 > 0.05)
No change in performance detected.
Found 2 outliers among 50 measurements (4.00%)
2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 2 concurrent task: Collecting 50 samples in estimated 5.1869 s (7650 BatchSpanProcessor/with 2 concurrent task
time: [683.31 us 689.33 us 695.83 us]
change: [-30.661% -28.072% -25.529%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 50 measurements (16.00%)
1 (2.00%) low severe
3 (6.00%) low mild
2 (4.00%) high mild
2 (4.00%) high severe
Benchmarking BatchSpanProcessor/with 4 concurrent task: Collecting 50 samples in estimated 6.0023 s (6375 BatchSpanProcessor/with 4 concurrent task
time: [872.21 us 877.92 us 884.16 us]
change: [-35.918% -32.276% -27.417%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 50 measurements (12.00%)
1 (2.00%) low mild
2 (4.00%) high mild
3 (6.00%) high severe
Benchmarking BatchSpanProcessor/with 8 concurrent task: Collecting 50 samples in estimated 5.6743 s (3825 BatchSpanProcessor/with 8 concurrent task
time: [1.4650 ms 1.4771 ms 1.4894 ms]
change: [-50.676% -48.612% -46.793%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 50 measurements (6.00%)
3 (6.00%) high mild
Benchmarking BatchSpanProcessor/with 16 concurrent task: Collecting 50 samples in estimated 6.7458 s (2550 BatchSpanProcessor/with 16 concurrent task
time: [2.6165 ms 2.6337 ms 2.6510 ms]
change: [-45.358% -43.865% -42.602%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 50 measurements (2.00%)
1 (2.00%) high mild
Benchmarking BatchSpanProcessor/with 32 concurrent task: Warming up for 3.0000 s
Warning: Unable to complete 50 samples in 5.0s. You may wish to increase target time to 6.8s, enable flat sampling, or reduce sample count to 30.
Benchmarking BatchSpanProcessor/with 32 concurrent task: Collecting 50 samples in estimated 6.7980 s (1275 BatchSpanProcessor/with 32 concurrent task
time: [5.1859 ms 5.2923 ms 5.4093 ms]
change: [-43.492% -42.330% -41.255%] (p = 0.00 < 0.05)
Performance has improved.
start-end-span/always-sample
time: [439.83 ns 444.14 ns 448.29 ns]
change: [-86.724% -86.631% -86.544%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) low severe
3 (3.00%) low mild
start-end-span/never-sample
time: [174.11 ns 174.33 ns 174.54 ns]
change: [-74.100% -74.022% -73.949%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
2 (2.00%) low severe
4 (4.00%) low mild
1 (1.00%) high mild
Benchmarking start-end-span-4-attrs/always-sample: Collecting 100 samples in estimated 5.0005 s (4.1M iter start-end-span-4-attrs/always-sample
time: [1.1029 us 1.1181 us 1.1347 us]
change: [-84.832% -84.602% -84.372%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) low mild
3 (3.00%) high mild
Benchmarking start-end-span-4-attrs/never-sample: Collecting 100 samples in estimated 5.0001 s (27M iterat start-end-span-4-attrs/never-sample
time: [187.30 ns 187.55 ns 187.77 ns]
change: [-16.726% -15.987% -15.251%] (p = 0.00 < 0.05)
Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
5 (5.00%) low severe
10 (10.00%) low mild
Benchmarking start-end-span-8-attrs/always-sample: Collecting 100 samples in estimated 5.0036 s (2.7M iter start-end-span-8-attrs/always-sample
time: [1.7884 us 1.8112 us 1.8354 us]
change: [-42.353% -41.313% -40.264%] (p = 0.00 < 0.05)
Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
2 (2.00%) low severe
3 (3.00%) low mild
8 (8.00%) high mild
Benchmarking start-end-span-8-attrs/never-sample: Collecting 100 samples in estimated 5.0006 s (22M iterat start-end-span-8-attrs/never-sample
time: [221.78 ns 222.11 ns 222.41 ns]
change: [-8.7823% -7.9412% -7.1224%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
1 (1.00%) low severe
4 (4.00%) low mild
Benchmarking start-end-span-all-attr-types/always-sample: Collecting 100 samples in estimated 5.0038 s (2. start-end-span-all-attr-types/always-sample
time: [1.6476 us 1.6681 us 1.6882 us]
change: [-36.034% -34.687% -33.448%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low severe
2 (2.00%) low mild
Benchmarking start-end-span-all-attr-types/never-sample: Collecting 100 samples in estimated 5.0004 s (25M start-end-span-all-attr-types/never-sample
time: [212.80 ns 213.34 ns 213.90 ns]
change: [-6.0818% -5.3116% -4.4650%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking start-end-span-all-attr-types-2x/always-sample: Collecting 100 samples in estimated 5.0070 s start-end-span-all-attr-types-2x/always-sample
time: [2.9687 us 2.9791 us 2.9889 us]
change: [-22.268% -20.748% -19.142%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
6 (6.00%) low severe
3 (3.00%) low mild
Benchmarking start-end-span-all-attr-types-2x/never-sample: Collecting 100 samples in estimated 5.0008 s ( start-end-span-all-attr-types-2x/never-sample
time: [265.47 ns 266.35 ns 267.49 ns]
change: [-5.3977% -4.1198% -3.1305%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
2 (2.00%) low severe
8 (8.00%) low mild
Benchmark | Baseline(Current implementation) | mpmc channel | new thread | Tokio channel |
---|---|---|---|---|
BSP/1 task | 746.43us | 898.70us(+18%) | 589.51us(-26%) | 777.63us(-0.8%) |
BSP/2 tasks | 926.24us | 767.57us(-21%) | 719us(-23%) | 689.33us(-28.072%) |
BSP/4 tasks | 1.3ms | 1.04ms(-27%) | 965us(-28%) | 877.92us(-32.27%) |
BSP/8 tasks | 2.8ms | 1.46ms(-48%) | 1.62ms(-42%) | 1.47ms(-48%) |
BSP/16 tasks | 4.5ms | 2.47ms(-47%) | 2.64ms(-41%) | 2.6337ms(-43%) |
BSP/32 tasks | 9.1ms | 4.5ms(-50%) | 5.54ms(-40%) | 5.2923ms(-42%) |
start-end-span/always-sample | 3.37us | 474.71ns(-85%) | 485.30ns(-86%) | 444.14ns(-86%) |
start-end-span/never-sample | 670.24ns | 169.04ns(-74%) | 176.95ns(-73%) | 174.33ns(-74%) |
start-end-span-4-attrs/always-sample | 7.24us | 1.1708us(-84%) | 1.57us(-78%) | 1.1181us(-84%) |
start-end-span-4-attrs/never-sample | 214.26ns | 193.64ns(-13%) | 200.00ns(-9%) | 187.55ns(-15.987%) |
start-end-span-8-attrs/always-sample | 2.97us | 1.71us(-44%) | 2.33us(-22%) | 1.81us(-41.313%) |
start-end-span-8-attrs/never-sample | 232.98ns | 231.88ns(-3.06%) | 220.01ns(-8.06%) | 222.11ns(-7.9%) |
start-end-span-all-attr-types/always-sample | 2.49us | 1.5us(-38%) | 1.8375us(-28.325%) | 1.6681us(-34%) |
start-end-span-all-attr-types/never-sample | 220.54ns | 214ns(-3%) | 223.66ns(+1.19%) | 213.34ns(-5.311%) |
start-end-span-all-attr-types-2x/always-sample | 3.82us | 2.81us(-25%) | 3.1948us(-13.092%) | 2.9791us(-20%) |
start-end-span-all-attr-types-2x/never-sample | 274.30ns | 273.86ns(-1.76%) | 257.74ns(-6.1%) | 266.35ns(-4.1198%) |
Baseline
It's the current implementation, using
Mutex
Benchmark
Batch span processor benchmark
Trace benchmark