Closed tanjunchen closed 9 months ago
eBPF limitations make it challenging to fully implement all L7 parsing on the kernel side. The choice of a 1024-byte payload size has been made to provide sufficient data for parsing L7 protocols in userland.
I've implemented several performance optimizations using your benchmark approach.
without coroot-node-agent:
taskset -c 1-2 wrk -t 2 -c 4 -d 60s http://10.42.0.9:80/ --latency
Running 1m test @ http://10.42.0.9:80/
2 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 173.83us 114.66us 8.32ms 97.46%
Req/Sec 11.64k 1.10k 14.84k 72.46%
Latency Distribution
50% 158.00us
75% 185.00us
90% 229.00us
99% 365.00us
1391869 requests in 1.00m, 1.10GB read
Requests/sec: 23159.30
Transfer/sec: 18.77MB
with coroot-node-agent v1.14.0 (with optimizations):
taskset -c 1-2 wrk -t 2 -c 4 -d 60s http://10.42.0.9:80/ --latency
Running 1m test @ http://10.42.0.9:80/
2 threads and 4 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 189.95us 129.27us 9.60ms 96.94%
Req/Sec 10.62k 1.11k 15.71k 72.67%
Latency Distribution
50% 171.00us
75% 207.00us
90% 259.00us
99% 414.00us
1268200 requests in 1.00m, 1.00GB read
Requests/sec: 21136.58
Transfer/sec: 17.13MB
This 9% degradation in request throughput can be attributed to the agent's CPU consumption, which reaches 30% of one CPU core. I expect that if there is no competition for CPU time, the degradation will be much less. At the eBPF level, the kernel ensures that the observer program does not introduce significant additional latency
@tanjunchen thank you for bringing up this topic
We've added the benchmark results to the documentation: https://coroot.com/docs/coroot-community-edition/getting-started/performance-impact
May I ask what is the additional network latency for L4 and L7 caused by coroot-node-agent? What is the impact of eBPF application topology, trace, etc. on the application? Can official documents provide performance pressure test data?
After our testing, the coroot-node-agent network latency p90 increased by 6460us and QPS decreased by 50%, as shown in the flame diagram below.![image](https://github.com/coroot/coroot-node-agent/assets/19339970/566b2855-aea7-44ef-933d-82b9f29bbc82)
Why is the value size of #define MAX_PAYLOAD_SIZE 1024? Is it quite time-consuming to process the logic of L7 here, and why is it 1024 bytes?
the process of performance test:
taskset -c 0-1 wrk -t 2 -c 4 -d 60s http://ip --latency
.the result of with coroot environment: