search

cloudwego / netpoll-benchmark

38 stars 12 forks source link

The benchmark script doesn't consider NUMA, which compromised the performance #4

Closed innerr closed 2 years ago

innerr commented 2 years ago

With NUMA info:

$> numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
node 0 size: 98210 MB
node 0 free: 12270 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
node 1 size: 98304 MB
node 1 free: 104 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

The benchmark script use cpuset 0-3, 4-19:

...
benchmark mode=1
server net_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 100, total: 5000000
Info: took 21127 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
...

These setting compromised the performance, the result logs are posted bellow.

innerr commented 2 years ago

The original setting result:

benchmark mode=1
server net_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 100, total: 5000000
Info: took 21127 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 236661.70, TP99: 2.19ms, TP999: 11.46ms (b=1024 Byte, c=100, n=5000000)
[NET@Server] CPU Usage: MIN: 301.00%, TP50: 321.00%, TP90: 330.33%, TP99: 333.33%, MAX: 335.34%, AVG:321.56%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 170.73%, TP50: 745.00%, TP90: 764.00%, TP99: 775.00%, MAX: 775.78%, AVG:698.94%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 100, total: 5000000
Info: took 20115 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 248561.55, TP99: 1.11ms, TP999: 2.52ms (b=1024 Byte, c=100, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 335.34%, TP50: 360.44%, TP90: 368.37%, TP99: 369.00%, MAX: 370.89%, AVG:360.37%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 195.20%, TP50: 799.00%, TP90: 816.00%, TP99: 821.82%, MAX: 822.82%, AVG:746.16%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server gnet_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [gnet] start ..., concurrent: 100, total: 5000000
Info: took 23067 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [gnet]: TPS: 216756.84, TP99: 2.02ms, TP999: 10.02ms (b=1024 Byte, c=100, n=5000000)
[GNET@Server] CPU Usage: MIN: 316.32%, TP50: 332.00%, TP90: 338.00%, TP99: 342.69%, MAX: 344.00%, AVG:331.53%
[GNET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[GNET@Client] CPU Usage: MIN: 168.00%, TP50: 673.00%, TP90: 688.69%, TP99: 693.69%, MAX: 695.70%, AVG:632.23%
[GNET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server evio_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [evio] start ..., concurrent: 100, total: 5000000
Info: took 23425 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [evio]: TPS: 213443.96, TP99: 4.65ms, TP999: 9.80ms (b=1024 Byte, c=100, n=5000000)
[EVIO@Server] CPU Usage: MIN: 310.31%, TP50: 328.33%, TP90: 333.00%, TP99: 335.00%, MAX: 336.00%, AVG:325.25%
[EVIO@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[EVIO@Client] CPU Usage: MIN: 171.88%, TP50: 675.68%, TP90: 696.70%, TP99: 699.70%, MAX: 702.70%, AVG:628.17%
[EVIO@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server net_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 500, total: 5000000
Info: took 19622 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 254808.75, TP99: 15.92ms, TP999: 22.98ms (b=1024 Byte, c=500, n=5000000)
[NET@Server] CPU Usage: MIN: 336.99%, TP50: 352.00%, TP90: 359.48%, TP99: 360.00%, MAX: 360.00%, AVG:351.55%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 165.79%, TP50: 820.82%, TP90: 843.84%, TP99: 845.85%, MAX: 848.00%, AVG:753.19%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 500, total: 5000000
Info: took 19828 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 252162.09, TP99: 4.58ms, TP999: 8.03ms (b=1024 Byte, c=500, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 339.32%, TP50: 373.88%, TP90: 384.00%, TP99: 386.00%, MAX: 391.39%, AVG:373.18%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 161.16%, TP50: 832.00%, TP90: 859.00%, TP99: 871.00%, MAX: 885.89%, AVG:766.17%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server gnet_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [gnet] start ..., concurrent: 500, total: 5000000
Info: took 29512 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [gnet]: TPS: 169422.25, TP99: 10.94ms, TP999: 17.57ms (b=1024 Byte, c=500, n=5000000)
[GNET@Server] CPU Usage: MIN: 319.32%, TP50: 329.66%, TP90: 336.00%, TP99: 338.00%, MAX: 339.34%, AVG:330.37%
[GNET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[GNET@Client] CPU Usage: MIN: 145.15%, TP50: 561.00%, TP90: 569.57%, TP99: 571.57%, MAX: 572.00%, AVG:529.05%
[GNET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server evio_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [evio] start ..., concurrent: 500, total: 5000000
Info: took 28109 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [evio]: TPS: 177875.76, TP99: 10.28ms, TP999: 16.37ms (b=1024 Byte, c=500, n=5000000)
[EVIO@Server] CPU Usage: MIN: 299.30%, TP50: 310.00%, TP90: 322.65%, TP99: 323.65%, MAX: 325.00%, AVG:312.10%
[EVIO@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[EVIO@Client] CPU Usage: MIN: 158.00%, TP50: 591.00%, TP90: 610.61%, TP99: 612.00%, MAX: 613.61%, AVG:562.94%
[EVIO@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server net_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 1000, total: 5000000
Info: took 19173 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 260776.91, TP99: 23.66ms, TP999: 38.38ms (b=1024 Byte, c=1000, n=5000000)
[NET@Server] CPU Usage: MIN: 360.00%, TP50: 370.00%, TP90: 380.38%, TP99: 381.00%, MAX: 384.38%, AVG:371.19%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 166.67%, TP50: 870.87%, TP90: 896.90%, TP99: 899.00%, MAX: 902.00%, AVG:811.33%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 1000, total: 5000000
Info: took 19542 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 255849.90, TP99: 8.70ms, TP999: 12.30ms (b=1024 Byte, c=1000, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 327.00%, TP50: 389.39%, TP90: 394.87%, TP99: 395.00%, MAX: 395.40%, AVG:385.13%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 161.00%, TP50: 880.00%, TP90: 898.90%, TP99: 902.90%, MAX: 902.90%, AVG:792.62%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server gnet_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [gnet] start ..., concurrent: 1000, total: 5000000
Info: took 28974 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [gnet]: TPS: 172563.30, TP99: 17.74ms, TP999: 25.86ms (b=1024 Byte, c=1000, n=5000000)
[GNET@Server] CPU Usage: MIN: 289.71%, TP50: 343.66%, TP90: 352.35%, TP99: 355.36%, MAX: 356.71%, AVG:342.39%
[GNET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[GNET@Client] CPU Usage: MIN: 164.16%, TP50: 589.59%, TP90: 604.60%, TP99: 611.00%, MAX: 611.61%, AVG:557.19%
[GNET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server evio_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [evio] start ..., concurrent: 1000, total: 5000000
Info: took 29355 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [evio]: TPS: 170327.84, TP99: 17.93ms, TP999: 27.54ms (b=1024 Byte, c=1000, n=5000000)
[EVIO@Server] CPU Usage: MIN: 294.00%, TP50: 311.00%, TP90: 316.32%, TP99: 320.32%, MAX: 320.32%, AVG:310.01%
[EVIO@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[EVIO@Client] CPU Usage: MIN: 219.70%, TP50: 585.00%, TP90: 596.60%, TP99: 603.00%, MAX: 606.61%, AVG:555.17%
[EVIO@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB
innerr commented 2 years ago

The result with modified cpuset core-list, without NUMA node-crossing:

benchmark mode=1
server net_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [net] start ..., concurrent: 100, total: 5000000
Info: took 18395 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 271800.85, TP99: 1.68ms, TP999: 3.86ms (b=1024 Byte, c=100, n=5000000)
[NET@Server] CPU Usage: MIN: 147.00%, TP50: 376.00%, TP90: 379.00%, TP99: 379.38%, MAX: 379.38%, AVG:363.06%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 162.39%, TP50: 868.00%, TP90: 876.88%, TP99: 877.88%, MAX: 878.88%, AVG:774.60%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [netpoll] start ..., concurrent: 100, total: 5000000
Info: took 17623 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 283707.16, TP99: 0.87ms, TP999: 1.44ms (b=1024 Byte, c=100, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 377.00%, TP50: 391.39%, TP90: 393.00%, TP99: 393.39%, MAX: 394.39%, AVG:389.34%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 160.56%, TP50: 898.00%, TP90: 907.91%, TP99: 909.00%, MAX: 909.91%, AVG:807.66%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server gnet_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [gnet] start ..., concurrent: 100, total: 5000000
Info: took 19401 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [gnet]: TPS: 257715.62, TP99: 1.19ms, TP999: 4.75ms (b=1024 Byte, c=100, n=5000000)
[GNET@Server] CPU Usage: MIN: 367.37%, TP50: 375.00%, TP90: 377.00%, TP99: 378.38%, MAX: 379.00%, AVG:374.58%
[GNET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[GNET@Client] CPU Usage: MIN: 184.00%, TP50: 798.00%, TP90: 807.81%, TP99: 810.00%, MAX: 810.00%, AVG:722.99%
[GNET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server evio_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [evio] start ..., concurrent: 100, total: 5000000
Info: took 20314 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [evio]: TPS: 246124.24, TP99: 0.93ms, TP999: 5.12ms (b=1024 Byte, c=100, n=5000000)
[EVIO@Server] CPU Usage: MIN: 310.00%, TP50: 386.00%, TP90: 387.39%, TP99: 388.00%, MAX: 388.39%, AVG:381.57%
[EVIO@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[EVIO@Client] CPU Usage: MIN: 170.92%, TP50: 792.00%, TP90: 801.80%, TP99: 804.80%, MAX: 805.00%, AVG:735.53%
[EVIO@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server net_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [net] start ..., concurrent: 500, total: 5000000
Info: took 18297 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 273267.70, TP99: 8.99ms, TP999: 16.84ms (b=1024 Byte, c=500, n=5000000)
[NET@Server] CPU Usage: MIN: 380.00%, TP50: 384.38%, TP90: 389.00%, TP99: 389.00%, MAX: 389.39%, AVG:385.18%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 152.78%, TP50: 880.00%, TP90: 888.00%, TP99: 892.89%, MAX: 895.90%, AVG:783.29%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [netpoll] start ..., concurrent: 500, total: 5000000
Info: took 17783 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 281158.35, TP99: 3.56ms, TP999: 4.98ms (b=1024 Byte, c=500, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 366.37%, TP50: 396.62%, TP90: 398.00%, TP99: 398.40%, MAX: 398.40%, AVG:394.53%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 158.38%, TP50: 915.92%, TP90: 925.93%, TP99: 926.00%, MAX: 926.93%, AVG:824.71%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server gnet_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [gnet] start ..., concurrent: 500, total: 5000000
Info: took 23569 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [gnet]: TPS: 212137.89, TP99: 5.71ms, TP999: 9.22ms (b=1024 Byte, c=500, n=5000000)
[GNET@Server] CPU Usage: MIN: 355.36%, TP50: 368.00%, TP90: 371.37%, TP99: 372.37%, MAX: 373.00%, AVG:367.64%
[GNET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[GNET@Client] CPU Usage: MIN: 154.00%, TP50: 673.00%, TP90: 679.00%, TP99: 680.00%, MAX: 684.00%, AVG:621.38%
[GNET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server evio_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [evio] start ..., concurrent: 500, total: 5000000
Info: took 23130 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [evio]: TPS: 216167.60, TP99: 4.98ms, TP999: 7.85ms (b=1024 Byte, c=500, n=5000000)
[EVIO@Server] CPU Usage: MIN: 363.00%, TP50: 371.00%, TP90: 376.38%, TP99: 377.00%, MAX: 378.38%, AVG:372.32%
[EVIO@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[EVIO@Client] CPU Usage: MIN: 150.00%, TP50: 696.70%, TP90: 716.72%, TP99: 726.00%, MAX: 727.00%, AVG:641.59%
[EVIO@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server net_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [net] start ..., concurrent: 1000, total: 5000000
Info: took 18256 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 273870.78, TP99: 18.39ms, TP999: 27.30ms (b=1024 Byte, c=1000, n=5000000)
[NET@Server] CPU Usage: MIN: 385.39%, TP50: 389.39%, TP90: 391.39%, TP99: 391.39%, MAX: 396.25%, AVG:389.54%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 155.00%, TP50: 907.91%, TP90: 919.92%, TP99: 919.92%, MAX: 920.92%, AVG:840.56%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [netpoll] start ..., concurrent: 1000, total: 5000000
Info: took 17913 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 279124.20, TP99: 6.77ms, TP999: 8.81ms (b=1024 Byte, c=1000, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 390.00%, TP50: 394.18%, TP90: 394.82%, TP99: 395.00%, MAX: 395.19%, AVG:393.96%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 146.70%, TP50: 922.92%, TP90: 930.00%, TP99: 931.93%, MAX: 934.00%, AVG:841.01%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server gnet_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [gnet] start ..., concurrent: 1000, total: 5000000
Info: took 22663 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [gnet]: TPS: 220618.67, TP99: 9.23ms, TP999: 13.08ms (b=1024 Byte, c=1000, n=5000000)
[GNET@Server] CPU Usage: MIN: 373.00%, TP50: 375.38%, TP90: 379.00%, TP99: 379.00%, MAX: 380.38%, AVG:376.37%
[GNET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[GNET@Client] CPU Usage: MIN: 163.00%, TP50: 706.00%, TP90: 715.72%, TP99: 715.72%, MAX: 719.00%, AVG:654.46%
[GNET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server evio_reciever running with taskset -c 0,2,4,6
client net_bencher running with taskset -c 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Info: [evio] start ..., concurrent: 1000, total: 5000000
Info: took 23841 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [evio]: TPS: 209719.52, TP99: 9.01ms, TP999: 13.04ms (b=1024 Byte, c=1000, n=5000000)
[EVIO@Server] CPU Usage: MIN: 353.35%, TP50: 358.36%, TP90: 361.00%, TP99: 362.00%, MAX: 363.36%, AVG:358.07%
[EVIO@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[EVIO@Client] CPU Usage: MIN: 149.15%, TP50: 690.69%, TP90: 696.70%, TP99: 697.70%, MAX: 701.00%, AVG:643.83%
[EVIO@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB