search

Kitura / Kitura-NIO

A networking library for Kitura, based on SwiftNIO
Apache License 2.0
37 stars 25 forks source link

Poor performance with the web-frameworks benchmark #207

Open pushkarnk opened 5 years ago

pushkarnk commented 5 years ago

Kitura with Kitura-NIO seems to performs badly on the web-frameworks benchmarks

Reported over slack. Copied from there:

Language (Runtime) Framework (Middleware) Average 50th percentile 90th percentile 99th percentile 99.9th percentile Standard deviation
swift (4.2) perfect (3.1) 22.11 ms 21.81 ms 26.35 ms 33.46 ms 56.41 ms 3607.00
swift (5.0) vapor (3.3) 65.25 ms 27.59 ms 49.31 ms 1354.51 ms 3496.16 ms 228498.67
swift (5.0) kitura (2.6) 35.61 ms 33.10 ms 48.56 ms 74.07 ms 647.15 ms 26 433.67
swift (5.0) kitura (2.7) 62.88 ms 61.14 ms 83.65 ms 115.44 ms 533.98 ms 21219.00
swift (5.0) kitura-nio (2.7) 177.09 ms 65.25 ms 115.28 ms 3343.02 ms 6831.05 ms 555543.00
Language (Runtime) Framework (Middleware) Requests / s Throughput
swift (4.2) perfect (3.1) 44599.00 41.93 MB
swift (5.0) vapor (3.3) 32438.67 54.37 MB
swift (5.0) kitura (2.6) 28 626.67 53.09 MB
swift (5.0) kitura (2.7) 15597.00 29.04 MB
swift (5.0) kitura-nio (2.7) 13049.00 24.67 MB
weissi commented 5 years ago

@pushkarnk this looks like some kind of stall to me. On the 50th percentile (median) kitura/kitura are pretty much the same, but the standard deviation is huge! So either the machine that was running the tests was super busy at some point when running kitura-nio or something stalled/got stuck for a while?

waghanza commented 5 years ago

FYI -> docker is in use on above results, then metrics are not very useful (could vary by ~40% )

I'm working on https://github.com/the-benchmarker/web-frameworks/pull/632 to use an isolated droplet and therefore to reduce docker noise

PS : If I can also have sponsored machine from IBM, I'll be happy to run Kitura benchmarks on :heart:

weissi commented 5 years ago

oh docker with networking i/egress networking? Yeah, that'll give you a bunch of variance.

waghanza commented 5 years ago

In the future, it will be on dedicated VM with a 2 GB/s link

djones6 commented 5 years ago

FYI, we can replicate a gap between Kitura-NIO and Kitura-net, and that definitely needs investigation. Previous investigations have shown Kitura-NIO to scale better than Kitura-net, but it's more CPU hungry so slower at lower CPU counts - we need to find out where that CPU is being burnt.

However, @helenmasters and I couldn't replicate the general slowness of Kitura vs. Vapor running the benchmark on a bare-metal Ubuntu 16.04 server, once we removed the Ubuntu version penalty (https://github.com/the-benchmarker/web-frameworks/pull/1346). We limited the Docker containers to a subset of CPUs to emulate the 8-cpu environment, and Vapor and Kitura came out similar in performance (results summarised in https://github.com/IBM-Swift/Kitura/issues/1448).

Perfect did show a lead over the others in our environment, similar to the one reported here, so the puzzle is why our local results for Kitura are substantially better than the ones published above.