[Edge Throughput] Provision and install software for CNFs and VPP vSwitch

lixuna commented 6 years ago

[x] Provision a m2.xlarge.x86 machine on Packet.net for running the CNFs
[x] Setup SSH access for team

* Setup base host system SR-IOV, VFS, IOMMU

[x] Enable SR-IOV, VFs and IOMMU
- [x] Reference comparison/box-by-box-kvm-docker/install_vpp_prereqs.sh
- [x] Reference: NSM Notes for configuring VPP with Mellanox Connectx-4 (PFs and VFs for Mellanox specific requirements and IOMMU information

Layer-2 network setup

[x] Setup Layer-2 networking
- [x] Packet.net configuration for hybrid network including VLANs
  - Reference https://help.packet.net/article/58-layer-2-configurations
- [x] Setup host for layer-2
- [x] document layer-2 networking
[x] Test layer-2 network setup

DPDK and VF support

[x] Install / configure Mellanox drivers if needed
- *Reference: NSM Notes for configuring VPP with Mellanox Connectx-4 (PFs and VFs for Mellanox specific requirements and IOMMU information

Install software for running CNFs

[x] Install Docker on Packet system for testing CNFs on Docker

Install software for running a vSwitch

[x] Install VPP software
[x] Configure DPDK support for VPP
[x] Setup virtual interfaces for VPP
[x] Create simple configuration for VPP
[x] Test using ping and arping from another host on the same VLAN

Update docs

[x] Update scripts for installing pre-reqs, if needed eg. install_vpp_prereqs.sh
[x] Update documentation with steps for software install. eg. comparison README

taylor commented 5 years ago

TBD / Deferred:

May not need to enable huge pages after boot:

  - $ echo 5120 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
  - $ echo 5120 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

taylor commented 5 years ago

Not enabling VFs in host. Issues with VPP seeing double packets and experiencing lower performance. Instead we will let VPP handle creating sub interfaces (equivalent to VF).

echo 8 > /sys/class/net/enp94s0f1/device/sriov_numvfs
# Check the interface name before calling command
#Not persistent through reboots

taylor commented 5 years ago

TODO: document parameters for kernel

iommu=pt intel_iommu=on hugepagesz=2M hugepages=10240 isolcpus=2,4,6 nohz_full=2,4,6 rcu_nocbs=2,4,6

taylor commented 5 years ago

https://help.packet.net/article/58-layer-2-configurations

https://packet.kayako.com/article/57-layer-2-overview

taylor commented 5 years ago

Find the second interface which is part of the bonded interface with:

awk '{print $2}' /sys/class/net/bond0/bonding/slaves

taylor commented 5 years ago

Update /etc/network/interfaces to remove the second device from the bond configuration.

Delete device from bond-slaves on the bond0 device: bond-slaves enp2s0 enp2s0d1

Delete the bond-master bond0 line from the iface configuration for the device.

Change:

iface enp2s0d1 inet manual
    pre-up sleep 4
    bond-master bond0

to

iface enp2s0d1 inet manual
    pre-up sleep 4
    bond-master bond0

taylor commented 5 years ago

Enable iommu and hugepages in the kernel

sed -i.bak 's/(GRUB_CMDLINE_LINUX=\")/\1iommu=pt intel_iommu=on hugepagesz=2M hugepages=10240 isolcpus=2,4,6 nohz_full=2,4,6 rcu_nocbs=2,4,6 /' /etc/default/grub

taylor commented 5 years ago

After grub and bond changes reboot.

Check second interface is not part of bond with cat /sys/class/net/bond0/bonding/slaves

taylor commented 5 years ago

If you are testing the L2 network with more than one vlan set vlan id on the interface and set an ip on that interface:

ip link add name enp2s0d1.1030 link enp2s0d1 type vlan id 1030 
ip addr add 172.16.99.31/24 dev enp2s0d1.1030
ifconfig enp2s0d1.1030 up

Alternatively test with one vlan in Packet.net web ui attached to the server.

taylor commented 5 years ago

Simple VPP configuration using host interface and IP set in VPP

https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/cnf_edge_router/vpp_vswitch/etc/vpp/examples/working_simple_setup.gate

taylor commented 5 years ago

Install the Mellanox drivers, libs, tools and dependencies:

cd /tmp
wget http://content.mellanox.com/ofed/MLNX_OFED-4.4-1.0.0.0/MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64.tgz
tar zxvf MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64.tgz
cd MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64
./mlnxofedinstall --dpdk --upstream-libs --force

taylor commented 5 years ago

New version of mellanox needed. http://content.mellanox.com/ofed/MLNX_OFED-4.4-2.0.7.0/MLNX_OFED_LINUX-4.4-2.0.7.0-ubuntu18.04-x86_64.tgz

Found from http://www.mellanox.com/page/products_dyn?product_family=26

cncf / cnf-testbed

[Edge Throughput] Provision and install software for CNFs and VPP vSwitch #53