Closed pirog-spb closed 10 months ago
Useful links:
sysctl -w kernel.bpf_stats_enabled=1
After applying new metrics appear: run_time_ns
run_cnt
sudo bpftool prog show id 425
425: xdp name upf_ip_entrypoi tag 24497ec2dee18e41 gpl run_time_ns 414676 run_cnt 153
loaded_at 2023-09-24T17:03:21+0300 uid 0
xlated 15336B jited 9520B memlock 16384B map_ids 76,77,78,79,80,81,82,83
btf_id 253
sudo perf list | grep xdp
xdp:mem_connect [Tracepoint event]
xdp:mem_disconnect [Tracepoint event]
xdp:mem_return_failed [Tracepoint event]
xdp:xdp_bulk_tx [Tracepoint event]
xdp:xdp_cpumap_enqueue [Tracepoint event]
xdp:xdp_cpumap_kthread [Tracepoint event]
xdp:xdp_devmap_xmit [Tracepoint event]
xdp:xdp_exception [Tracepoint event]
xdp:xdp_redirect [Tracepoint event]
xdp:xdp_redirect_err [Tracepoint event]
xdp:xdp_redirect_map [Tracepoint event]
xdp:xdp_redirect_map_err [Tracepoint event]
Nothing standing out of a row:
sudo perf record -a -e xdp:* sleep 10
sudo perf script
nr-gnb 53518 [007] 35279.562881: xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=150 to_ifindex=154 err=0 map_id=2147483647 map_index=0
irq/139-iwlwifi 476 [005] 35279.594733: xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=154 to_ifindex=150 err=0 map_id=2147483647 map_index=0
nr-gnb 53518 [007] 35280.563707: xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=150 to_ifindex=154 err=0 map_id=2147483647 map_index=0
irq/139-iwlwifi 476 [005] 35280.567961: xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=154 to_ifindex=150 err=0 map_id=2147483647 map_index=0
Good for comparing. Currently useless
sudo ./bpftool prog profile id 501 duration 10 cycles instructions dtlb_misses llc_misses
1901 run_cnt
12799802 cycles
4378593 instructions # 0.34 insns per cycle
50552 llc_misses # 11545.26 LLC misses per million insns
24845 dtlb_misses # 5674.20 dtlb misses per million insns
root@470fc2e4450f:/ueransim# iperf3 -c iperf --bind 10.60.0.1 -t 30
Connecting to host iperf, port 5201
[ 5] local 10.60.0.1 port 40275 connected to 10.100.200.15 port 5201
...
[ 5] 29.00-30.00 sec 41.2 MBytes 346 Mbits/sec 0 427 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 1.27 GBytes 364 Mbits/sec 1451 sender
[ 5] 0.00-30.01 sec 1.27 GBytes 363 Mbits/sec receiver
Nothing changed
root@4908d4dfee4f:/ueransim# iperf3 -c iperf --bind 10.60.0.1 -t 30
Connecting to host iperf, port 5201
[ 5] local 10.60.0.1 port 58679 connected to 10.100.200.15 port 5201
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 1.27 GBytes 362 Mbits/sec 1238 sender
[ 5] 0.00-30.01 sec 1.26 GBytes 361 Mbits/sec receiver
Performance has slightly been improved
iperf3 -c iperf --bind 10.60.0.1 -t 30
Connecting to host iperf, port 5201
[ 5] local 10.60.0.1 port 49887 connected to 10.100.200.15 port 5201
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 1.31 GBytes 375 Mbits/sec 1193 sender
[ 5] 0.00-30.00 sec 1.31 GBytes 375 Mbits/sec receiver
Seems like UERANSIM is a bottleneck. Go further without it.
Depending on repeat count we got different results.
sudo ./bpftool prog run id 2413 data_in packet.bin
Return value: 4, duration: 4842ns
sudo ./bpftool prog run id 2413 data_in packet.bin repeat 600000000
Return value: 2, duration (average): 56ns
sudo ./bpftool prog run id 254 data_in packet.bin repeat 1000000000
Return value: 2, duration (average): 72ns
packet.bin
-> 142 byte long.
Best case: 56ns per packet -> 17,9 Mpps -> 20 Gbps Worst case: 4842ns per packet -> 0,207 Mpps -> 0,234 Gbps
In docker-compose environment eUPF performance according to iperf is no more than 300Mbps while direct iperf measurement shows dozen of gigabites.
eUPF perfomance should as close to theoretical maximum(direct iperf measurement) as possible. Tune eUPF and host to acheive performance target.
Acceptance criteria: