P99 latency decreased, but throughput decreased too for netpool

[siddontang]

Run benchmark_server with 100 and 200 concurrents for net and netpool.

As we can see, the P99 and P999 latencies for netpool are very promising, but the throughput decreased too compared to net, any idea?

Linux: 3.10.0-1160.45.1.el7.x86_64
go version go1.17.6 linux/amd64
CPU: 48c Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz

benchmark mode=1
server net_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 100, total: 5000000
Info: took 20595 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 242775.61, TP99: 2.12ms, TP999: 11.51ms (b=1024 Byte, c=100, n=5000000)
[NET@Server] CPU Usage: MIN: 302.00%, TP50: 319.32%, TP90: 331.00%, TP99: 331.33%, MAX: 331.33%, AVG:321.03%
[NET@Client] CPU Usage: MIN: 168.54%, TP50: 768.77%, TP90: 795.00%, TP99: 801.00%, MAX: 806.81%, AVG:733.64%

server netpoll_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 100, total: 5000000
Info: took 20709 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 241429.79, TP99: 1.13ms, TP999: 2.63ms (b=1024 Byte, c=100, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 322.32%, TP50: 346.35%, TP90: 358.00%, TP99: 360.64%, MAX: 363.36%, AVG:346.11%
[NETPOLL@Client] CPU Usage: MIN: 184.18%, TP50: 777.00%, TP90: 797.80%, TP99: 811.81%, MAX: 818.00%, AVG:742.23%

server net_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 200, total: 5000000
Info: took 20549 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 243317.69, TP99: 4.89ms, TP999: 15.00ms (b=1024 Byte, c=200, n=5000000)
[NET@Server] CPU Usage: MIN: 312.31%, TP50: 324.32%, TP90: 335.34%, TP99: 337.34%, MAX: 337.65%, AVG:325.91%
[NET@Client] CPU Usage: MIN: 170.17%, TP50: 786.00%, TP90: 805.00%, TP99: 808.00%, MAX: 814.00%, AVG:742.77%

server netpoll_reciever running with taskset -c 0-3
client net_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 200, total: 5000000
Info: took 20328 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 245962.76, TP99: 2.16ms, TP999: 4.82ms (b=1024 Byte, c=200, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 319.44%, TP50: 354.48%, TP90: 373.00%, TP99: 373.99%, MAX: 376.62%, AVG:353.83%
[NETPOLL@Client] CPU Usage: MIN: 172.00%, TP50: 797.80%, TP90: 839.00%, TP99: 855.00%, MAX: 855.86%, AVG:750.29%

[siddontang]

I commented out Mem usage because of #2

[siddontang]

I use netpoll_bencher instead of origin net_bencher to do the benchmark again, now the result is promising:

benchmark mode=1
server net_reciever running with taskset -c 0-3
client netpoll_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 100, total: 5000000
Info: took 21234 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 235469.05, TP99: 2.28ms, TP999: 11.67ms (b=1024 Byte, c=100, n=5000000)
[NET@Server] CPU Usage: MIN: 310.00%, TP50: 322.65%, TP90: 329.33%, TP99: 331.99%, MAX: 334.00%, AVG:322.43%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 163.16%, TP50: 571.00%, TP90: 602.00%, TP99: 605.61%, MAX: 607.00%, AVG:541.38%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0-3
client netpoll_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 100, total: 5000000
Info: took 20433 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 244693.71, TP99: 1.11ms, TP999: 2.34ms (b=1024 Byte, c=100, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 346.00%, TP50: 364.64%, TP90: 378.00%, TP99: 379.34%, MAX: 380.00%, AVG:365.06%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 158.00%, TP50: 617.00%, TP90: 633.00%, TP99: 638.00%, MAX: 639.64%, AVG:580.85%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server net_reciever running with taskset -c 0-3
client netpoll_bencher running with taskset -c 4-19
Info: [net] start ..., concurrent: 200, total: 5000000
Info: took 21101 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [net]: TPS: 236952.38, TP99: 5.85ms, TP999: 14.56ms (b=1024 Byte, c=200, n=5000000)
[NET@Server] CPU Usage: MIN: 317.32%, TP50: 327.00%, TP90: 336.00%, TP99: 336.31%, MAX: 338.34%, AVG:326.91%
[NET@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NET@Client] CPU Usage: MIN: 158.16%, TP50: 559.00%, TP90: 592.00%, TP99: 598.60%, MAX: 601.00%, AVG:532.23%
[NET@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

server netpoll_reciever running with taskset -c 0-3
client netpoll_bencher running with taskset -c 4-19
Info: [netpoll] start ..., concurrent: 200, total: 5000000
Info: took 20020 ms for 5000000 requests
Info: requests total: 5000000, failed: 0
Info: [netpoll]: TPS: 249741.09, TP99: 2.15ms, TP999: 4.53ms (b=1024 Byte, c=200, n=5000000)
[NETPOLL@Server] CPU Usage: MIN: 356.07%, TP50: 373.00%, TP90: 381.57%, TP99: 382.83%, MAX: 386.00%, AVG:373.19%
[NETPOLL@Server] Mem Usage: AVG: 0 KB, MAX: 0 KB
[NETPOLL@Client] CPU Usage: MIN: 163.00%, TP50: 636.00%, TP90: 654.65%, TP99: 657.00%, MAX: 661.66%, AVG:591.41%
[NETPOLL@Client] Mem Usage: AVG: 0 KB, MAX: 0 KB

[Hchenn]

No idea for now, we'll follow up later. THANKS !