Create a multi-node bridged network version of the GoEPC use case

[taylor]

Multi-node bridged network GoEPC userplane use case using Multus

This is a POC step in reaching the goal of a multi-node segmented and routed user plane

Tasks:

• [x] Create a single-node bridged network version #327
• [x] Create documentation for K8s deployment
• [x] Update Helm charts to deploy EPC components to separate nodes. See: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
  • [ ] Add config option for enabling multi-host deployment. e.g helm install -e multihost=true
• [x] Add a new node group to https://github.com/crosscloudci/infra-provisioning/blob/node_groups/terraform/packet.tf - So that we can support different worker node machine types in the same cluster.
• [x] Add a host interface to the Multus bridges. - So Pods will have connectivity over multiple hosts.
  • [x] Create automation code for adding the host interface to the Multus bridges.

---

Testing:

• [ ] Acceptance Criteria

[taylor]

Acceptance Criteria:

Prereqs:

• Deploy K8s cluster with multus support following the deployment instructions here
• Run automation code for adding host interface to the Multus bridge.
• Clone the cnf-testbed
• Switch to the tag k8s_bridged_network-v0.2 (git checkout k8s_bridged_network-v0.2)

Multus multi-node bridged network user plane is successfully deployed

• [x] Go to the folder examples/use_case/gogtp-k8s/k8s_multi_node
• [x] Add the host interfaces to the Multus bridge (Pre-Req) by following the steps here
• [x] Deploy the use case to K8s following the documentation
  • [x] set environment variable for KUBECONFIG (replace path to match your location): export KUBECONFIG=<path>/<to>/kubeconfig
  • [x] Configure the bridges to be used with Multus to connect the pods: kubectl apply -f gogtp-bridges.yml
  • [x] Deploy the CNFs, external end-point test server, and testing system acting as "user equipment": helm install ./gogtp/
• [x] Wait for approximately 30 seconds, then check that the eNB is running: kubectl logs gogtp-enb
• Expected results should look something like:

  [eNB] 2020/01/31 13:23:46 Established S1-MME connection with 172.21.1.12:36412
  [eNB] 2020/01/31 13:23:46 Started serving S1-U on 172.21.0.11:2152
  [eNB] 2020/01/31 13:23:46 Successfully established tunnel for 001010000000001

Test HTTP GET to the end-point server

• [x] Deploy use case and verify the eNB is running
• [x] Send traffic following https://github.com/cncf/cnf-testbed/tree/go-gtp/examples/use_case/gogtp-k8s/k8s_bridged_networks#testing-the-gogtp-service-chain
  • [x] Get to the CLI on the test system: kubectl exec -it gogtp-ue /bin/bash
  • [x] Test that HTTP requests work to the testing end-point server by running: wget http://10.0.1.201

Expected results: You should see a successful HTTP request (200 OK), followed by index.html being saved.

Test ICMP/ping to the end-point server

• [x] Deploy use case and verify the eNB is running
• [x] Send traffic following https://github.com/cncf/cnf-testbed/tree/go-gtp/examples/use_case/gogtp-k8s/k8s_bridged_networks#testing-the-gogtp-service-chain
  • [x] Get to the CLI on the test system: kubectl exec -it gogtp-ue /bin/bash
  • [x] Run: ping -c 5 10.0.1.201

Expected results:

• 5 packets transmitted, 5 received, 0% packet loss

Testing that there are 3 nodes with services split across them

Validate that the user equipment and eNB are on the same node

kubectl describe pods <enb-pod> | grep Node
kubectl describe pods <ue-pod> | grep Node

Validate that the user equipment and eNB are not running on the same node as the EPC user plane components

kubectl describe pods <mme-pod> | grep Node
kubectl describe pods <sgw-pod> | grep Node
kubectl describe pods <pgw-pod> | grep Node

Validate that the HTTP end point server is not running on the same node as the EPC user plane components

kubectl describe pods <http-pod> | grep Node

Validate that the HTTP end point server is not running on the same node as the user equipment and eNB

kubectl describe pods <http-pod> | grep Node

[denverwilliams]

Tasks:

• [x] Update Helm charts to deploy EPC components to separate nodes. See: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

  • [ ] Add config option for enabling multi-host deployment. e.g helm install -e multihost=true

• [x] Add a new node group to https://github.com/crosscloudci/infra-provisioning/blob/node_groups/terraform/packet.tf - So that we can support different worker node machine types in the same cluster.

• [x] Add a host interface to the Multus bridges. - So Pods will have connectivity over multiple hosts.

  • [x] Create automation code for adding the host interface to the Multus bridges.

[michaelspedersen]

The initial implementation for this has been added to the go-gtp branch (available in PR).

The Helm charts are in a separate directory, examples/use_case/gogtp-k8s/k8s_multi_node, so deployment is done using helm install ./gogtp/ instead of helm install -e multihost=true. This was a deliberate choice to avoid impacting existing code prior to having the new implementation added.

Also, the host environment must be configured prior to running the multi-node example. I have created an ansible playbook that works similar to how we have been configuring for use with vSwitch (VPP) previously. I have not yet updated the Makefile to simplify this step, but it should be easy to expand the existing vswitch case to also include the multinode config (e.g. using || (OR) and pointing towards the correct playbook). Similar to the Helm charts, I decided to push this initial version to make sure it is available in Github prior to integrating it with existing code.

[michaelspedersen]

Makefile has been update in go-gtp branch, and documentation has been updated to reference this when deploying the use-case

[michaelspedersen]

Example use-case for this is available in examples/use_case/gogtp-k8s/k8s_multi_node. Closing this ticket