search

jaspervdj / websockets

A Haskell library for creating WebSocket-capable servers
http://jaspervdj.be/websockets
BSD 3-Clause "New" or "Revised" License
405 stars 112 forks source link

Add benchmark for roundtrip connections #235

Open domenkozar opened 6 months ago

domenkozar commented 6 months ago

What's really surprising is that once you add more cores to the benchmarks, it becomes orders of magnitude slower.

With -N1:

benchmarking connections/100
time                 14.78 ms   (14.66 ms .. 14.86 ms)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 15.11 ms   (14.99 ms .. 15.33 ms)
std dev              388.5 μs   (236.6 μs .. 611.0 μs)

benchmarking connections/1000
time                 146.5 ms   (143.9 ms .. 149.7 ms)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 150.5 ms   (148.9 ms .. 152.0 ms)
std dev              2.242 ms   (1.489 ms .. 3.410 ms)
variance introduced by outliers: 12% (moderately inflated)

benchmarking connections/10000
time                 1.523 s    (1.500 s .. 1.567 s)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.541 s    (1.532 s .. 1.551 s)
std dev              10.84 ms   (3.937 ms .. 14.65 ms)
variance introduced by outliers: 19% (moderately inflated)

With -N2:

benchmarking connections/100
time                 15.90 ms   (8.682 ms .. 29.22 ms)
                     0.395 R²   (0.345 R² .. 0.999 R²)
mean                 9.890 ms   (8.624 ms .. 14.89 ms)
std dev              6.383 ms   (164.5 μs .. 13.09 ms)
variance introduced by outliers: 96% (severely inflated)

benchmarking connections/1000
time                 86.76 ms   (83.65 ms .. 93.80 ms)
                     0.995 R²   (0.984 R² .. 1.000 R²)
mean                 88.68 ms   (87.43 ms .. 91.65 ms)
std dev              3.044 ms   (1.942 ms .. 4.282 ms)

benchmarking connections/10000
time                 860.6 ms   (636.3 ms .. 1.077 s)
                     0.987 R²   (0.987 R² .. 1.000 R²)
mean                 931.1 ms   (890.3 ms .. 971.8 ms)
std dev              52.23 ms   (24.33 ms .. 69.72 ms)
variance introduced by outliers: 19% (moderately inflated)

With -N4:

Benchmark bench-connections: RUNNING...
benchmarking connections/100
time                 80.88 ms   (1.093 ms .. 159.9 ms)
                     0.263 R²   (0.004 R² .. 0.799 R²)
mean                 75.93 ms   (34.91 ms .. 128.1 ms)
std dev              78.69 ms   (56.97 ms .. 108.9 ms)
variance introduced by outliers: 91% (severely inflated)

benchmarking connections/1000
time                 864.1 ms   (-959.9 ms .. 3.006 s)
                     0.580 R²   (0.000 R² .. 1.000 R²)
mean                 893.1 ms   (405.7 ms .. 1.064 s)
std dev              326.1 ms   (16.56 ms .. 394.2 ms)
variance introduced by outliers: 73% (severely inflated)

With -N8:

benchmarking connections/100
time                 1.331 s    (1.024 s .. 2.048 s)
                     0.966 R²   (0.947 R² .. 1.000 R²)
mean                 1.088 s    (1.024 s .. 1.216 s)
std dev              128.0 ms   (86.18 μs .. 148.1 ms)
variance introduced by outliers: 23% (moderately inflated)