search

edgecomllc / eupf

5G User Plane Function (UPF) based on eBPF
Apache License 2.0
99 stars 18 forks source link

Tune eupf perfomane in docker-compose environment #405

Closed pirog-spb closed 10 months ago

pirog-spb commented 12 months ago

In docker-compose environment eUPF performance according to iperf is no more than 300Mbps while direct iperf measurement shows dozen of gigabites.

eUPF perfomance should as close to theoretical maximum(direct iperf measurement) as possible. Tune eUPF and host to acheive performance target.

Acceptance criteria:

pirog-spb commented 11 months ago

Useful links:

Enable ebpf perf statistics

sysctl -w kernel.bpf_stats_enabled=1

After applying new metrics appear: run_time_ns run_cnt

sudo bpftool prog show id 425
425: xdp  name upf_ip_entrypoi  tag 24497ec2dee18e41  gpl run_time_ns 414676 run_cnt 153
    loaded_at 2023-09-24T17:03:21+0300  uid 0
    xlated 15336B  jited 9520B  memlock 16384B  map_ids 76,77,78,79,80,81,82,83
    btf_id 253

Check xdp events

sudo perf list | grep xdp
  xdp:mem_connect                                    [Tracepoint event]
  xdp:mem_disconnect                                 [Tracepoint event]
  xdp:mem_return_failed                              [Tracepoint event]
  xdp:xdp_bulk_tx                                    [Tracepoint event]
  xdp:xdp_cpumap_enqueue                             [Tracepoint event]
  xdp:xdp_cpumap_kthread                             [Tracepoint event]
  xdp:xdp_devmap_xmit                                [Tracepoint event]
  xdp:xdp_exception                                  [Tracepoint event]
  xdp:xdp_redirect                                   [Tracepoint event]
  xdp:xdp_redirect_err                               [Tracepoint event]
  xdp:xdp_redirect_map                               [Tracepoint event]
  xdp:xdp_redirect_map_err                           [Tracepoint event]

Nothing standing out of a row:

sudo perf record -a -e xdp:* sleep 10

sudo perf script
          nr-gnb 53518 [007] 35279.562881:         xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=150 to_ifindex=154 err=0 map_id=2147483647 map_index=0
 irq/139-iwlwifi   476 [005] 35279.594733:         xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=154 to_ifindex=150 err=0 map_id=2147483647 map_index=0
          nr-gnb 53518 [007] 35280.563707:         xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=150 to_ifindex=154 err=0 map_id=2147483647 map_index=0
 irq/139-iwlwifi   476 [005] 35280.567961:         xdp:xdp_redirect: prog_id=501 action=REDIRECT ifindex=154 to_ifindex=150 err=0 map_id=2147483647 map_index=0

Use bpftool profile

Good for comparing. Currently useless 

sudo ./bpftool prog profile id 501 duration 10 cycles instructions dtlb_misses llc_misses

              1901 run_cnt             
          12799802 cycles                                                       
           4378593 instructions        #     0.34 insns per cycle               
             50552 llc_misses          # 11545.26 LLC misses per million insns  
             24845 dtlb_misses         #  5674.20 dtlb misses per million insns

Return to iperf test

Baseline

root@470fc2e4450f:/ueransim# iperf3 -c iperf --bind 10.60.0.1 -t 30
Connecting to host iperf, port 5201
[  5] local 10.60.0.1 port 40275 connected to 10.100.200.15 port 5201
...
[  5]  29.00-30.00  sec  41.2 MBytes   346 Mbits/sec    0    427 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  1.27 GBytes   364 Mbits/sec  1451             sender
[  5]   0.00-30.01  sec  1.27 GBytes   363 Mbits/sec                  receiver

Disable QER handling

Nothing changed

root@4908d4dfee4f:/ueransim# iperf3 -c iperf --bind 10.60.0.1 -t 30
Connecting to host iperf, port 5201
[  5] local 10.60.0.1 port 58679 connected to 10.100.200.15 port 5201
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  1.27 GBytes   362 Mbits/sec  1238             sender
[  5]   0.00-30.01  sec  1.26 GBytes   361 Mbits/sec                  receiver

Disable policing in UERANSIM

Performance has slightly been improved

iperf3 -c iperf --bind 10.60.0.1 -t 30
Connecting to host iperf, port 5201
[  5] local 10.60.0.1 port 49887 connected to 10.100.200.15 port 5201
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  1.31 GBytes   375 Mbits/sec  1193             sender
[  5]   0.00-30.00  sec  1.31 GBytes   375 Mbits/sec                  receiver

Seems like UERANSIM is a bottleneck. Go further without it.

Use bpftool prog run

Depending on repeat count we got different results.

sudo ./bpftool prog run id 2413 data_in packet.bin 
Return value: 4, duration: 4842ns
sudo ./bpftool prog run id 2413 data_in packet.bin repeat 600000000
Return value: 2, duration (average): 56ns
sudo ./bpftool prog run id 254 data_in packet.bin repeat 1000000000
Return value: 2, duration (average): 72ns

packet.bin -> 142 byte long.

Best case: 56ns per packet -> 17,9 Mpps -> 20 Gbps Worst case: 4842ns per packet -> 0,207 Mpps -> 0,234 Gbps