search

romange / helio

A modern framework for backend development based on io_uring Linux interface
Apache License 2.0
447 stars 50 forks source link

chore: improve uring performance #183

Closed romange closed 9 months ago

romange commented 11 months ago
  1. Use io_uring_submit_and_get_events to both submit events and also fetch completions in single syscall.
  2. Make the critical section around blocking I/O command wait_for_cqe as small as possible to increase the chances that another thread will notify the proactor in time. That should decrease the amount of false positives where proactor blocks and then immediately wakes up do to notifications via kernel.

Performance tests: dragonfly is running on a 8-cpu server (aarch64), memtier client with 16 cpus: memtier_benchmark -s 172.31.36.182 --command=PING -t 16 -c 18 -n 60000 --hide-histogram --distinct-client-seed PING because we test pure networking stuff.

Dragonfly v1.13

ALL STATS
==================================================================================================
Type         Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
--------------------------------------------------------------------------------------------------
Pings      883283.45         0.32349         0.31900         0.47100         0.56700     18114.21
Totals     883283.45         0.32349         0.31900         0.47100         0.56700     18114.21

Dragonfly with helio changes upto commit 6b3c2c5d249ccb0f64da80d168bc35c574724fb4 (https://github.com/romange/helio/pull/193)

ALL STATS
==================================================================================================
Type         Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
--------------------------------------------------------------------------------------------------
Pings      964019.96         0.31617         0.31900         0.46300         0.55900     19769.94
Totals     964019.96         0.31617         0.31900         0.46300         0.55900     19769.94

Dragonfly with helio changes including direct-fds and uring improvements

ALL STATS
==================================================================================================
Type         Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
--------------------------------------------------------------------------------------------------
Pings     1003896.90         0.29585         0.29500         0.41500         0.51900     20587.73
Totals    1003896.90         0.29585         0.29500         0.41500         0.51900     20587.73
codecov-commenter commented 11 months ago

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (9e64c47) 79.00% compared to head (188609d) 78.50%.

Files Patch % Lines
util/fibers/uring_socket.cc 58.33% 5 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #183 +/- ## ========================================== - Coverage 79.00% 78.50% -0.51% ========================================== Files 100 100 Lines 7527 7537 +10 ========================================== - Hits 5947 5917 -30 - Misses 1580 1620 +40 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

romange commented 11 months ago

For same 3 configurations with GETs on empty database (testing the networking + 1 single hop, noop command): memtier_benchmark -s 172.31.36.182 --ratio=0:1 -t 16 -c 18 -n 60000 --hide-histogram --distinct-client-seed

Dragonfly with all the improvements

============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
----------------------------------------------------------------------------------------------------------------------------
Gets       740704.48         0.00    740704.48         0.38514         0.38300         0.55100         0.66300     28853.47

Dragonfly upto #193

============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
----------------------------------------------------------------------------------------------------------------------------
Gets       716695.29         0.00    716695.29         0.40247         0.40700         0.59100         0.75100     27918.21

Dragonfly v1.13

============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
----------------------------------------------------------------------------------------------------------------------------
Gets       689701.00         0.00    689701.00         0.42315         0.43100         0.61500         0.74300     26866.67