RaulB16
commented
1 year ago Hello! Any update? I'm facing the same problem. Thanks.
Closed danielfurman closed 1 year ago
RaulB16
commented
1 year ago Hello! Any update? I'm facing the same problem. Thanks.
danielfurman
commented
1 year ago @RaulB16 No, I didn't proceed further with this setup. I ended up using AWS EKS with Ubuntu 20.04 node and it worked much better. I needed to only change network interfaces names (e.g. n2network.masterIf) in free5gc Helm values file (to ens5 in my case).
raoufkh
commented
1 year ago Hi @danielfurman and @RaulB16
Honestly, I don't know where to install the gtp kernel module when using Minikube. I think you have to do it at the host level since you are using the Docker driver to create your cluster, but it doesn't seem to work. I leave this issue open to see if there is already someone who managed to deploy the project on Minikube.
Can you still share the state of the UPF and SMF pods with the kubectl describe command please? just to check that the N3 and N4 interfaces are created.
Raouf
danielfurman
commented
1 year ago Hi, with a help of my colleague I managed to proceed further with Minikube deployment.
I don't have the first deployment on my machine, but the problem was that Macvlan interfaces (e.g. N3, N4) were not created due to problem with CNIs configuration. I'll show better approach below.
Start Minikube cluster with Docker driver
# Starting minikube with Multus CNI
minikube start --memory 4096 --cni=/home/daniel/projects/vmw/multus-cni/deployments/multus-daemonset.yml
# Fix bridge CNI IPv6 issue (see: https://github.com/cri-o/cri-o/issues/3555#issuecomment-1027615192)
minikube ssh
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0Install gtp5g kernel module on your linux host (same as before)
Create 5gc and ueran namespaces
kubectl create namespace 5gc
kubectl create namespace ueranOptional - deploy Istio
Deploy free5gc with Helm
helm install free5gc towards5gs/free5gc -n 5gc -f ./free5gc-k8s-cluster/free5gc-helm-mk.yamlglobal:
n2network:
masterIf: eth0 # eth0 is the interface you can find on cluster Node
n3network:
masterIf: eth0
n4network:
masterIf: eth0
n6network:
masterIf: eth0
n9network:
masterIf: eth0Register UE via free5gc webui
export NODE_IP=$(minikube ip)firefox ${NODE_IP}:30500 (user: admin, pass: free5gc)Deploy UERANSIM with Helm
helm install ueransim towards5gs/ueransim -n ueran -f ./free5gc-k8s-cluster/ueransim-helm-mk.yamlglobal:
n2network:
masterIf: eth0
n3network:
masterIf: eth0Free5gc pods are running but some (e.g. NSSF) are logging several NRF connection errors: pod-errors.log
Macvlan interfaces are created in pods
UE established PDU session and created uesimtun0 interface
UPF pod has ip forwarding enabled:
➜ 5g git:(master) ✗ kubectl -n 5gc exec -it pod/free5gc-free5gc-upf-upf-566cdc4f54-b7zn7 -- cat /proc/sys/net/ipv4/ip_forward
1UE ping does not work due to Destination Host Unreachable
➜ 5g git:(master) ✗ kubectl -n ueran exec -it $UE_POD_NAME -- ping -I uesimtun0 www.google.com
PING www.google.com (142.250.186.36) from 10.1.0.1 uesimtun0: 56(84) bytes of data.
From 10.100.100.12 (10.100.100.12) icmp_seq=1 Destination Host Unreachable
From 10.100.100.12 (10.100.100.12) icmp_seq=2 Destination Host UnreachableMore info: kubectl-get-all.txt upf-pod.txt ue.log ue-pod.txt minikube.md - ip address and route info on pods, node, and laptop
If anyone has an idea why UE-internet ping might not be working or how to debug that, please let me know.
danielfurman
commented
1 year ago I managed to fix UE-internet ping for the Minikube setup. The last problem was that the following condition was not fullfiled (https://github.com/Orange-OpenSource/towards5gs-helm/tree/main/charts/free5gc#networks-configuration):
In addition, please make sure global.n6network.subnetIP, global.n6network.gatewayIP and free5gc-upf.upf.n6if.ipAddress parameters will match the IP address of the toto interface in order to make the UPF able to reach the Data Network via its N6 interface.
To fulfill that condition the easiest way was to start Minikube with a subnet matching N6 network default parameters:
minikube start --memory 8192 --subnet=10.100.100.0/24 --cni=../multus-cni/deployments/multus-daemonset.ymlThe rest of steps are the same as in Setup 2
Since the connectivity test is working for me, I am closing the issue.
Context
I am trying to deploy free5gc on local k8s cluster, so that I can explore it. I decided to spin a Minikube single-node cluster with Docker driver on my local Manjaro Linux laptop. It has following kernel:
5.15.60-1-MANJAROSteps
Steps I performed:
Setup Minikube with Docker driver
Install gtp5g kernel module on your linux host
Install multus-cni on your linux host
git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cnicat ./deployments/multus-daemonset-thick-plugin.yml | kubectl apply -f -Setup persistent volume in k8s
minikube ssh->mkdir /home/docker/5gc-dataSetup physical network interfaces in k8s node (eth0, eth1)
ip a show eth0on k8s node shows an interfaceInstall helm chart:
K8s node has following eth0 interface:
K8s has no eth1 interface.
Problem
SMF and UPF services fail:
Adjusting several helm variables didn't bring an improvement:
Questions
upf.n3if.ipAddress?Thank you in advance.