DPDK Low Throughput Using Kubernetes

[infinitydon]

Hi,

I am currently trying to use upf with DPDK using Kubernetes for orchestration but the iperf results I am getting seems not to be realistic..

ubuntu@ip-10-0-2-51:~/UERANSIM/build$ ./nr-binder 12.1.1.2 iperf3 -c 10.0.7.167 -i 1 -t 10
Connecting to host 10.0.7.167, port 5201
[  5] local 12.1.1.2 port 45115 connected to 10.0.7.167 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.80 MBytes  65.4 Mbits/sec  362   31.6 KBytes
[  5]   1.00-2.00   sec  2.50 MBytes  21.0 Mbits/sec  227   27.6 KBytes
[  5]   2.00-3.00   sec  1.25 MBytes  10.5 Mbits/sec  277   25.0 KBytes
[  5]   3.00-4.00   sec  2.50 MBytes  21.0 Mbits/sec  293   11.8 KBytes
[  5]   4.00-5.00   sec  2.50 MBytes  21.0 Mbits/sec  233   40.8 KBytes
[  5]   5.00-6.00   sec  2.50 MBytes  21.0 Mbits/sec  187   23.7 KBytes
[  5]   6.00-7.00   sec  2.50 MBytes  21.0 Mbits/sec  263   11.8 KBytes
[  5]   7.00-8.00   sec  2.50 MBytes  21.0 Mbits/sec  230   50.0 KBytes
[  5]   8.00-9.00   sec  2.50 MBytes  21.0 Mbits/sec  299   22.4 KBytes
[  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec  229   3.95 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.8 MBytes  23.3 Mbits/sec  2600             sender
[  5]   0.00-10.00  sec  25.0 MBytes  21.0 Mbits/sec                  receiver

iperf Done.

I have enabled both HugePages and guaranteed CPU Qos, so no other process is sharing the CPU with the vpp-upf pod..

    State:          Running
      Started:      Tue, 30 Nov 2021 07:03:59 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                      4
      hugepages-1Gi:            4Gi
      intel.com/sriov_dpdk_n3:  1
      intel.com/sriov_dpdk_n4:  1
      intel.com/sriov_dpdk_n6:  1
      memory:                   4Gi
    Requests:
      cpu:                      4
      hugepages-1Gi:            4Gi
      intel.com/sriov_dpdk_n3:  1
      intel.com/sriov_dpdk_n4:  1
      intel.com/sriov_dpdk_n6:  1
      memory:                   4Gi

unix {
  nodaemon
  log /tmp/vpp.log
  full-coredump
  gid vpp
  interactive
  cli-listen /run/vpp/cli.sock
  exec /openair-upf/etc/init.conf
}

api-trace {
  on
}

dpdk {
 uio-driver vfio-pci
 no-multi-seg
 dev default {
   num-rx-queues 1
   num-rx-queues 1
   }
 dev 0000:00:06.0
 dev 0000:00:07.0
 dev 0000:00:08.0
}

api-segment {
  gid vpp
}

plugins {
    path /usr/lib/x86_64-linux-gnu/vpp_plugins/
    plugin dpdk_plugin.so { enable }
    plugin gtpu_plugin.so { disable }
    plugin upf_plugin.so { enable }
}

ip table add 1
ip table add 2

set interface ip table VirtualFunctionEthernet0/6/0 1
set interface mtu 9001 VirtualFunctionEthernet0/6/0
set interface ip address VirtualFunctionEthernet0/6/0 10.0.4.11/24
set interface state VirtualFunctionEthernet0/6/0 up

set interface ip table VirtualFunctionEthernet0/7/0 0
set interface mtu 9001 VirtualFunctionEthernet0/7/0
set interface ip address VirtualFunctionEthernet0/7/0 10.0.6.11/24
set interface state VirtualFunctionEthernet0/7/0 up

set interface ip table VirtualFunctionEthernet0/8/0 2
set interface mtu 9001 VirtualFunctionEthernet0/8/0
set interface ip address VirtualFunctionEthernet0/8/0 10.0.7.11/24
set interface state VirtualFunctionEthernet0/8/0 up

ip route add 0.0.0.0/0 table 2 via 10.0.7.167 VirtualFunctionEthernet0/8/0

trace add dpdk-input 100

upf pfcp endpoint ip 10.0.6.11 vrf 0

upf nwi name access.oai.org vrf 1
upf nwi name core.oai.org vrf 2

upf node-id fqdn gw1.vppupf.node.5gcn.mnc95.mcc208.3gppnetwork.org

upf specification release 16

upf gtpu endpoint ip 10.0.4.11 nwi access.oai.org teid 0x000004d2/2

My understanding is that the upf-vpp is independent of the DPDK logic but I have not been to know why the throughput test is so low, I have played around with the MTU settings but there is no improvement.

[infinitydon]

I discovered the SMF had QoS values hard-coded in the configuration

What was confusing is that vpp-upf does not currently implement QoS enforcement so I could not pin point where the throttling was taking place but apparently it seems this is been done at the gNB.

I increased the QoS in the SMF and now I am getting a more realistic throughput values:

ubuntu@ip-10-0-2-51:~/UERANSIM/build$ ./nr-binder 12.1.1.2 iperf3 -c 10.0.7.167 -i 1 -t 5
Connecting to host 10.0.7.167, port 5201
[  5] local 12.1.1.2 port 43895 connected to 10.0.7.167 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   231 MBytes  1.94 Gbits/sec    0   3.02 MBytes
[  5]   1.00-2.00   sec   225 MBytes  1.89 Gbits/sec  293    781 KBytes
[  5]   2.00-3.00   sec   232 MBytes  1.95 Gbits/sec    0    968 KBytes
[  5]   3.00-4.00   sec   211 MBytes  1.77 Gbits/sec    2    799 KBytes
[  5]   4.00-5.00   sec   230 MBytes  1.93 Gbits/sec    0    983 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  1.10 GBytes  1.90 Gbits/sec  295             sender
[  5]   0.00-5.00   sec  1.10 GBytes  1.89 Gbits/sec                  receiver

iperf Done.

[Salhi-K]

Hello, could you please share with me the helm chart you used to deploy UPF with DPDK ?

[mgumz]

@Salhi-K this is available as our (Travelping's) commercial offer.