UPF Kubernetes Deployment

infinitydon commented 3 years ago

Hi,

Please I think the k8s manifest file upf-k8s.yaml needs to be updated, I could not find some of the parameter in conf/upf.json file.

Also kindly change the manifest from POD to Deployment.

krsna1729 commented 3 years ago

The statefulset helm charts are ONF member-only at the moment. upf-k8s.yaml could use an update.

https://github.com/omec-project/upf-epc/blob/e9cd16cb94629bdadee363d0e68fa26b866e3dac/upf-k8s.yaml#L47-L52

I think updating this to be below should be enough?

       "access": { 
         "ifname": "s1u" 
       }, 
       "core": { 
         "ifname": "sgi" 
       },

What else is missing/failing? Can you send a PR?

krsna1729 commented 3 years ago

This can be used as a reference for the pod spec itself https://jenkins.opencord.org/job/omec_upf-epc_container_remote/203/artifact/omec/k8s-logs/upf-0-get.log

infinitydon commented 3 years ago

Hi @krsna1729 - Thanks for the response, I am still trying to get the deployment working, so below are my observations and errors I encountered.

The multus NetworkAttachmentDefinition is using type vfioveth and I could not find a reference to vfioveth in the normal CNI tools but doing a google search led me to https://github.com/opencord/omec-cni/blob/master/vfioveth, this also did not work out of the box as some of the procedures to get the interface properties are not working on Ubuntu 20.04 (not sure if it works with other linux distros), I was able to scale through this by modifying the script and the interfaces were able to be attached to the POD. Please let me know if the vfioveth is still the recommended CNI plugin to use or if there is some updated doc about this.
After sorting out the CNI, the deployment still failed to start up properly, kindly see the attached log

upf-bess-july-13-log.txt

Kindly assist in resolving the errors. Once I am able to get this deployed successfully I will open a PR to share the updated guide for k8s

krsna1729 commented 3 years ago

Yes @hyunsun maintains vfioveth these days at the above URL. Are you running this in a VM or baremetal? VM would require vfio_noiommu. You can temporarily comment out the post start hook to make it running, get logs of all the containers and from within the bessd container collect -

ip route
ip link
ls /dev/vfio

then run the command

./bessctl run up4

Share the output as well as the container logs.

infinitydon commented 3 years ago

Hi @krsna1729 -

root@upf-76cb6b4b88-99j7j:/opt/bess/bessctl# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 32:dd:68:53:0a:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.160.162/32 brd 192.168.160.162 scope global eth0
       valid_lft forever preferred_lft forever
4: n3-vdev@n3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 66:5d:a9:9d:87:29 brd ff:ff:ff:ff:ff:ff
5: n3@n3-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:28:83:68 brd ff:ff:ff:ff:ff:ff
    inet 198.18.0.13/24 scope global n3
       valid_lft forever preferred_lft forever
6: n6-vdev@n6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:da:83:15:f2:91 brd ff:ff:ff:ff:ff:ff
7: n6@n6-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:65:41:b1 brd ff:ff:ff:ff:ff:ff
    inet 198.19.0.6/24 scope global n6
       valid_lft forever preferred_lft forever

root@upf-76cb6b4b88-99j7j:/opt/bess/bessctl# ls /dev/vfio
vfio

root@upf-76cb6b4b88-99j7j:/opt/bess/bessctl# ip r
default via 169.254.1.1 dev eth0
default via 198.19.0.1 dev n6 metric 110
169.254.1.1 dev eth0 scope link
198.18.0.0/24 dev n3 proto kernel scope link src 198.18.0.13
198.19.0.0/24 dev n6 proto kernel scope link src 198.19.0.6

root@upf-76cb6b4b88-99j7j:/opt/bess/bessctl# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether 32:dd:68:53:0a:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
4: n3-vdev@n3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 66:5d:a9:9d:87:29 brd ff:ff:ff:ff:ff:ff
5: n3@n3-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:28:83:68 brd ff:ff:ff:ff:ff:ff
    alias 0000:09:00.0
6: n6-vdev@n6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 3e:da:83:15:f2:91 brd ff:ff:ff:ff:ff:ff
7: n6@n6-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:65:41:b1 brd ff:ff:ff:ff:ff:ff
    alias 0000:06:00.0

root@upf-76cb6b4b88-99j7j:/opt/bess/bessctl# ./bessctl run up4
max_ip_defrag_flows value not set. Not installing IP4Defrag module.
ip_frag_with_eth_mtu value not set. Not installing IP4Frag module.
gtppsc not set. Default: Not adding PDU Session Container extension header
hwcksum not set, using default software fallback
ddp not set, using default software fallback
Autodetecting network driver
Sim mode not selected.
Can't parse unix socket paths for notify! Setting it to default values (/tmp/notifycp)
Can't parse unix socket paths for end marker! Setting it to default values (/tmp/pfcpport)
Network Token Function disabled
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Setting up port n3 on worker ids [0]
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Unable to initialize n3 fastpath using alias 0000:09:00.0,                        falling back to scan
Registered dpdk ports do not exist.
*** Error: Unhandled exception in the configuration script (most recent call last)
  File "/opt/bess/bessctl/conf/up4.bess", line 131, in <module>
    p.init_port(idx, parser.mode)
  File "/opt/bess/bessctl/conf/ports.py", line 177, in init_port
    sys.exit()
SystemExit
  Command failed: run up4

I am running kvm on Ubuntu 20.04, kindly let me know what steps are required.

krsna1729 commented 3 years ago

Can you also paste the logs of bessd. The above contains the output of the script only.

Also since you confirmed you are running in a VM, vfio_noiommu is needed. Go through the link to make this work https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/master/docs/dpdk/README-virt.md

root@upf-76cb6b4b88-99j7j:/opt/bess/bessctl# ls /dev/vfio
vfio

This is the problem, no devices were actually provided to container due to which the initialization script failed to detect DPDK port. Even in the bessd log it would say somewhere 0 DPDK PMD ports detected.

So please go through the link above to configure sriov when operating in a virtual environment. If possible also test this in baremetal as that is how we test and operate. Lets see how far we can go in that.

infinitydon commented 3 years ago

I have checked the link you shared and it seems the HW stated there:

VF drivers
DPDK applications require devices to be attached with supported dpdk backend driver.

For Intel® x700 series NICs vfio-pci is required.
For Mellanox ConnectX®-4 Lx, ConnectX®-5 Adapters mlx5_core or mlx5_ib is required.

is different from the network interfaces that are available on my server:

05:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 10 GbE SFP+ [8086:15c4] (rev 11)
        Subsystem: Intel Corporation Ethernet Connection X553 10 GbE SFP+ [8086:0000]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
05:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 10 GbE SFP+ [8086:15c4] (rev 11)
        Subsystem: Intel Corporation Ethernet Connection X553 10 GbE SFP+ [8086:0000]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e4] (rev 11)
        Subsystem: Intel Corporation Ethernet Connection X553 1GbE [8086:0000]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
07:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X553 1GbE [8086:15e4] (rev 11)
        Subsystem: Intel Corporation Ethernet Connection X553 1GbE [8086:0000]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

So this means that my setup can't work with dpdk using sriov-dp (i.e. only vfio-pci can work)?

Alternatively is it possible for me to use af_xdp instead? If yes are there any specific parameters I need to adjust in the UPF configmap?

krsna1729 commented 3 years ago

X550 series should be supported in DPDK itself and the sriov device plugin

krsna1729 commented 3 years ago

You will have to create VFs out of it on the host and pass them into the VM. Bind the VFs to vfio-pci with noiommu option inside the VM. This will allow the device plugin to discover the devices and correctly plumb into the container.

infinitydon commented 3 years ago

I have sorted out the vfio-pci issue but I am getting the following errors:

root@upf-6d465f7795-fj685:/opt/bess/bessctl# ./bessctl run up4
max_ip_defrag_flows value not set. Not installing IP4Defrag module.
ip_frag_with_eth_mtu value not set. Not installing IP4Frag module.
gtppsc not set. Default: Not adding PDU Session Container extension header
hwcksum not set, using default software fallback
ddp not set, using default software fallback
Autodetecting network driver
Sim mode not selected.
Can't parse unix socket paths for notify! Setting it to default values (/tmp/notifycp)
Can't parse unix socket paths for end marker! Setting it to default values (/tmp/pfcpport)
Network Token Function disabled
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Setting up port s1u on worker ids [0]
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Unable to initialize s1u fastpath using alias 0000:0a:00.0,                        falling back to scan
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
*** Error: Unhandled exception in the configuration script (most recent call last)
  File "/opt/bess/bessctl/conf/up4.bess", line 131, in <module>
    p.init_port(idx, parser.mode)
  File "/opt/bess/bessctl/conf/ports.py", line 181, in init_port
    raise Exception(
Exception: Registered port for s1u not detected!
  Command failed: run up4

Logs from the bessd container:

chris@5gcore-master-01:~/open5gs-upf-bess$ kubectl -n 5gcore logs upf-5c9d59477-8lwdn -c bessd
I0717 21:25:52.328210   315 main.cc:62] Launching BESS daemon in process mode...
I0717 21:25:52.328302   315 main.cc:75] bessd unknown
I0717 21:25:52.333992   315 bessd.cc:456] Loading plugin (attempt 1): /bin/modules/sequential_update.so
I0717 21:25:52.335826   315 dpdk.cc:167] Initializing DPDK EAL with options: ["bessd", "--master-lcore", "127", "--lcore", "127@0-7", "--no-shconf", "--legacy-mem", "--socket-mem", "1024,1024", "--huge-unlink"]
EAL: Detected 8 lcore(s)
EAL: Detected 2 NUMA nodes
Option --master-lcore is deprecated use main-lcore
EAL: Selected IOVA mode 'PA'
EAL: 256 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
I0717 21:25:52.348242   315 dpdk.cc:66] EAL: Probing VFIO support...
I0717 21:25:52.348304   315 dpdk.cc:66] EAL: VFIO support initialized
I0717 21:25:52.562538   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562659   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562685   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562857   315 dpdk.cc:66] EAL:   using IOMMU type 8 (No-IOMMU)
I0717 21:25:52.641211   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:52.701705   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:07:00.0 (socket 0)
I0717 21:25:52.802676   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.873642   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:52.933761   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:08:00.0 (socket 0)
I0717 21:25:53.035089   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.109526   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:53.158639   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:09:00.0 (socket 0)
I0717 21:25:53.259377   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.259469   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.259493   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.260179   315 dpdk.cc:66] EAL: No legacy callbacks, legacy socket not created
Segment 0-0: IOVA:0x180000000, len:1073741824, virt:0x140000000, socket_id:0, hugepage_sz:1073741824, nchannel:0, nrank:0 fd:10
Segment 2-0: IOVA:0x300000000, len:1073741824, virt:0x11c0000000, socket_id:1, hugepage_sz:1073741824, nchannel:0, nrank:0 fd:11
I0717 21:25:53.260300   315 packet_pool.cc:49] Creating DpdkPacketPool for 262144 packets on node 0
I0717 21:25:53.260352   315 packet_pool.cc:70] PacketPool0 requests for 262144 packets
I0717 21:25:53.295562   315 packet_pool.cc:157] PacketPool0 has been created with 262144 packets
I0717 21:25:53.295593   315 packet_pool.cc:49] Creating DpdkPacketPool for 262144 packets on node 1
I0717 21:25:53.295601   315 packet_pool.cc:70] PacketPool1 requests for 262144 packets
I0717 21:25:53.318130   315 packet_pool.cc:157] PacketPool1 has been created with 262144 packets
I0717 21:25:53.318295   315 pmd.cc:68] 3 DPDK PMD ports have been recognized:
I0717 21:25:53.318338   315 pmd.cc:92] DPDK port_id 0 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:b6:3b:13  00000000:07:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318349   315 pmd.cc:92] DPDK port_id 1 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:be:a2:42  00000000:08:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318357   315 pmd.cc:92] DPDK port_id 2 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:28:83:68  00000000:09:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318387   315 vport.cc:318] vport: BESS kernel module is not loaded. Loading...
sh: 1: insmod: not found
W0717 21:25:53.319600   315 vport.cc:330] Cannot load kernel module /bin/kmod/bess.ko
I0717 21:25:53.319650   315 bessctl.cc:1931] Server listening on 0.0.0.0:10514
I0717 21:25:53.319709790     315 server_builder.cc:247]      Synchronous server. Num CQs: 1, Min pollers: 1, Max Pollers: 1, CQ timeout (msec): 1000
I0717 21:26:12.682690   334 bessctl.cc:487] *** All workers have been paused ***
I0717 21:26:13.098367   370 worker.cc:319] Worker 0(0x7fdf077fc700) is running on core 0 (socket 0)
I0717 21:26:13.099241   354 bessctl.cc:691] Checking scheduling constraints
I0717 21:26:13.099714   354 bessctl.cc:516] *** Resuming ***

krsna1729 commented 3 years ago

Can you paste the config upf.json you are using? It seems to be complaining about the mode selected.

infinitydon commented 3 years ago

UPF config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: upf-conf
data:
  upf.json: |
    {
      "access": {
        "ifname": "s1u"
      },
      "core": {
        "ifname": "sgi"
      },
      "measure": true,
      "workers": "1",
      "max_sessions": 50000
    }

infinitydon commented 3 years ago

@krsna1729 - Sorry I mixed up the logs (I made a mistake in the config by setting the mode to null), I have updated the logs with the current error I am getting now, you can also check below:

Unable to initialize s1u fastpath using alias 0000:0a:00.0,                        falling back to scan
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
*** Error: Unhandled exception in the configuration script (most recent call last)
  File "/opt/bess/bessctl/conf/up4.bess", line 131, in <module>
    p.init_port(idx, parser.mode)
  File "/opt/bess/bessctl/conf/ports.py", line 181, in init_port
    raise Exception(
Exception: Registered port for s1u not detected!
  Command failed: run up4

krsna1729 commented 3 years ago

can you show the output of

In the bessd container

ip address
ip route
ip link
ls /dev/vfio

On the host

ip link show <PF name from which VFs were created>

krsna1729 commented 3 years ago

0000:0a:00.0 is this the PCI ID of access interface? Since the output is saying it saw it in the alias but not able to find it.

Also why is DPDK seeing 3 e1000_em interfaces? I thought you were using X550 VFs

I0717 21:25:52.348242   315 dpdk.cc:66] EAL: Probing VFIO support...
I0717 21:25:52.348304   315 dpdk.cc:66] EAL: VFIO support initialized
I0717 21:25:52.562538   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562659   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562685   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562857   315 dpdk.cc:66] EAL:   using IOMMU type 8 (No-IOMMU)
I0717 21:25:52.641211   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:52.701705   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:07:00.0 (socket 0)
I0717 21:25:52.802676   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.873642   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:52.933761   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:08:00.0 (socket 0)
I0717 21:25:53.035089   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.109526   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:53.158639   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:09:00.0 (socket 0)
I0717 21:25:53.259377   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.259469   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.259493   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0

I0717 21:25:53.318338   315 pmd.cc:92] DPDK port_id 0 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:b6:3b:13  00000000:07:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318349   315 pmd.cc:92] DPDK port_id 1 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:be:a2:42  00000000:08:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318357   315 pmd.cc:92] DPDK port_id 2 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:28:83:68  00000000:09:00.00 8086:10d3   numa_node 0

infinitydon commented 3 years ago

On the server with X553 interface for some reasons which I don't know the POD is not able to start at all (with the postStart lifecycle disabled):

chris@5gcore-master-01:~/open5gs-upf-bess$ kubectl -n 5gcore logs upf-84bf64bfc8-sgphk -c routectl
/opt/bess/bessctl/conf/route_control.py:311: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if item.prefix_len is 0:
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Connecting to BESS daemon...
Error connecting to BESS daemon. Retrying in 2sec...
Error connecting to BESS daemon. Retrying in 2sec...
Error connecting to BESS daemon. Retrying in 2sec...
Error connecting to BESS daemon. Retrying in 2sec...
Error connecting to BESS daemon. Retrying in 2sec...
Traceback (most recent call last):
  File "/opt/bess/bessctl/conf/route_control.py", line 517, in <module>
    main()
  File "/opt/bess/bessctl/conf/route_control.py", line 483, in main
    connect_bessd()
  File "/opt/bess/bessctl/conf/route_control.py", line 438, in connect_bessd
    raise Exception('BESS connection failure.')
Exception: BESS connection failure.

I decided to use another system and the POD is able to start properly (with the postStart lifecycle disabled) that is the reason for the _e1000em driver from the logs.

In the bessd container:

root@upf-74c49648cd-hpzpq:/opt/bess/bessctl# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 72:3d:cf:9d:3d:8b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.160.186/32 brd 172.16.160.186 scope global eth0
       valid_lft forever preferred_lft forever
4: s1u-vdev@s1u: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 62:8d:ad:f6:f7:ab brd ff:ff:ff:ff:ff:ff
5: s1u@s1u-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:ee:57:99 brd ff:ff:ff:ff:ff:ff
    inet 198.18.0.24/24 scope global s1u
       valid_lft forever preferred_lft forever
6: sgi-vdev@sgi: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 26:58:a6:35:42:b1 brd ff:ff:ff:ff:ff:ff
7: sgi@sgi-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:45:a2:1d brd ff:ff:ff:ff:ff:ff
    inet 198.19.0.20/24 scope global sgi
       valid_lft forever preferred_lft forever
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl# ip r
default via 169.254.1.1 dev eth0
default via 198.19.0.1 dev sgi metric 110
169.254.1.1 dev eth0 scope link
198.18.0.0/24 dev s1u proto kernel scope link src 198.18.0.24
198.19.0.0/24 dev sgi proto kernel scope link src 198.19.0.20
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether 72:3d:cf:9d:3d:8b brd ff:ff:ff:ff:ff:ff link-netnsid 0
4: s1u-vdev@s1u: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 62:8d:ad:f6:f7:ab brd ff:ff:ff:ff:ff:ff
5: s1u@s1u-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:ee:57:99 brd ff:ff:ff:ff:ff:ff
    alias 0000:0a:00.0
6: sgi-vdev@sgi: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 26:58:a6:35:42:b1 brd ff:ff:ff:ff:ff:ff
7: sgi@sgi-vdev: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:45:a2:1d brd ff:ff:ff:ff:ff:ff
    alias 0000:0b:00.0
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl#
root@upf-74c49648cd-hpzpq:/opt/bess/bessctl# ls /dev/vfio
0  1  noiommu-0  noiommu-1  noiommu-2  vfio

On the host, I can only do ip link for the s1u and sgi PF (can't do this for the vfios since they are plumbed already into userspace):

For s1u:

chris@5gcore-worker-01:~$ ip link show dev enp10s0
7: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:ee:57:99 brd ff:ff:ff:ff:ff:ff

For sgi:

chris@5gcore-worker-01:~$ ip link show dev enp11s0
8: enp11s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:45:a2:1d brd ff:ff:ff:ff:ff:ff

krsna1729 commented 3 years ago

The scenario changes without notice is confusing. I don't know anymore what I am debugging. Please use docker_setup.sh to debug your setup and then migrate to kubernetes on the same setup.

infinitydon commented 3 years ago

Thanks for the support so far.. I will probably try it later.

subhajitchatu commented 2 years ago

I have sorted out the vfio-pci issue but I am getting the following errors:

root@upf-6d465f7795-fj685:/opt/bess/bessctl# ./bessctl run up4
max_ip_defrag_flows value not set. Not installing IP4Defrag module.
ip_frag_with_eth_mtu value not set. Not installing IP4Frag module.
gtppsc not set. Default: Not adding PDU Session Container extension header
hwcksum not set, using default software fallback
ddp not set, using default software fallback
Autodetecting network driver
Sim mode not selected.
Can't parse unix socket paths for notify! Setting it to default values (/tmp/notifycp)
Can't parse unix socket paths for end marker! Setting it to default values (/tmp/pfcpport)
Network Token Function disabled
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Setting up port s1u on worker ids [0]
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
Unable to initialize s1u fastpath using alias 0000:0a:00.0,                        falling back to scan
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
*** Error: Unhandled exception in the configuration script (most recent call last)
  File "/opt/bess/bessctl/conf/up4.bess", line 131, in <module>
    p.init_port(idx, parser.mode)
  File "/opt/bess/bessctl/conf/ports.py", line 181, in init_port
    raise Exception(
Exception: Registered port for s1u not detected!
  Command failed: run up4

Logs from the bessd container:

chris@5gcore-master-01:~/open5gs-upf-bess$ kubectl -n 5gcore logs upf-5c9d59477-8lwdn -c bessd
I0717 21:25:52.328210   315 main.cc:62] Launching BESS daemon in process mode...
I0717 21:25:52.328302   315 main.cc:75] bessd unknown
I0717 21:25:52.333992   315 bessd.cc:456] Loading plugin (attempt 1): /bin/modules/sequential_update.so
I0717 21:25:52.335826   315 dpdk.cc:167] Initializing DPDK EAL with options: ["bessd", "--master-lcore", "127", "--lcore", "127@0-7", "--no-shconf", "--legacy-mem", "--socket-mem", "1024,1024", "--huge-unlink"]
EAL: Detected 8 lcore(s)
EAL: Detected 2 NUMA nodes
Option --master-lcore is deprecated use main-lcore
EAL: Selected IOVA mode 'PA'
EAL: 256 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
I0717 21:25:52.348242   315 dpdk.cc:66] EAL: Probing VFIO support...
I0717 21:25:52.348304   315 dpdk.cc:66] EAL: VFIO support initialized
I0717 21:25:52.562538   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562659   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562685   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.562857   315 dpdk.cc:66] EAL:   using IOMMU type 8 (No-IOMMU)
I0717 21:25:52.641211   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:52.701705   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:07:00.0 (socket 0)
I0717 21:25:52.802676   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:52.873642   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:52.933761   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:08:00.0 (socket 0)
I0717 21:25:53.035089   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.109526   315 dpdk.cc:66] EAL: Ignore mapping IO port bar(2)
I0717 21:25:53.158639   315 dpdk.cc:66] EAL: Probe PCI driver: net_e1000_em (8086:10d3) device: 0000:09:00.0 (socket 0)
I0717 21:25:53.259377   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.259469   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.259493   315 dpdk.cc:66] EAL:   Invalid NUMA socket, default to 0
I0717 21:25:53.260179   315 dpdk.cc:66] EAL: No legacy callbacks, legacy socket not created
Segment 0-0: IOVA:0x180000000, len:1073741824, virt:0x140000000, socket_id:0, hugepage_sz:1073741824, nchannel:0, nrank:0 fd:10
Segment 2-0: IOVA:0x300000000, len:1073741824, virt:0x11c0000000, socket_id:1, hugepage_sz:1073741824, nchannel:0, nrank:0 fd:11
I0717 21:25:53.260300   315 packet_pool.cc:49] Creating DpdkPacketPool for 262144 packets on node 0
I0717 21:25:53.260352   315 packet_pool.cc:70] PacketPool0 requests for 262144 packets
I0717 21:25:53.295562   315 packet_pool.cc:157] PacketPool0 has been created with 262144 packets
I0717 21:25:53.295593   315 packet_pool.cc:49] Creating DpdkPacketPool for 262144 packets on node 1
I0717 21:25:53.295601   315 packet_pool.cc:70] PacketPool1 requests for 262144 packets
I0717 21:25:53.318130   315 packet_pool.cc:157] PacketPool1 has been created with 262144 packets
I0717 21:25:53.318295   315 pmd.cc:68] 3 DPDK PMD ports have been recognized:
I0717 21:25:53.318338   315 pmd.cc:92] DPDK port_id 0 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:b6:3b:13  00000000:07:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318349   315 pmd.cc:92] DPDK port_id 1 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:be:a2:42  00000000:08:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318357   315 pmd.cc:92] DPDK port_id 2 (net_e1000_em)   RXQ 1 TXQ 1  52:54:00:28:83:68  00000000:09:00.00 8086:10d3   numa_node 0
I0717 21:25:53.318387   315 vport.cc:318] vport: BESS kernel module is not loaded. Loading...
sh: 1: insmod: not found
W0717 21:25:53.319600   315 vport.cc:330] Cannot load kernel module /bin/kmod/bess.ko
I0717 21:25:53.319650   315 bessctl.cc:1931] Server listening on 0.0.0.0:10514
I0717 21:25:53.319709790     315 server_builder.cc:247]      Synchronous server. Num CQs: 1, Min pollers: 1, Max Pollers: 1, CQ timeout (msec): 1000
I0717 21:26:12.682690   334 bessctl.cc:487] *** All workers have been paused ***
I0717 21:26:13.098367   370 worker.cc:319] Worker 0(0x7fdf077fc700) is running on core 0 (socket 0)
I0717 21:26:13.099241   354 bessctl.cc:691] Checking scheduling constraints
I0717 21:26:13.099714   354 bessctl.cc:516] *** Resuming ***

Hi, can you please elaborate how did you solve the vfio-pci issue ? How to attach mlx5 vf in docker running bess up4? Thanks

omec-project / upf

UPF Kubernetes Deployment #304