search

Orange-OpenSource / towards5gs-helm

Helm charts for deploying 5G network services on Kubernetes
Other
167 stars 130 forks source link

Ping through "uesimtun0" not working. #62

Closed rveerama1 closed 1 year ago

rveerama1 commented 1 year ago

I am trying to test the free5gc with UERANSIM.

I have followed steps from https://github.com/Orange-OpenSource/towards5gs-helm/blob/main/docs/demo/Setup-free5gc-and-test-with-UERANSIM.md

I will post detailed steps about my cluster and some extra details (Note: I have single node cluster)

I have ubuntu 22.04, kubeadm and kubectl 1.25.3.

$ uname -r
5.15.0-53-generic

sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

gtp5g module

git clone -b v0.3.1 https://github.com/free5gc/gtp5g.git
cd gtp5g
make
sudo make install

helm charts

helm repo add towards5gs https://raw.githubusercontent.com/Orange-OpenSource/towards5gs-helm/main/repo/
helm repo update

cluster

sudo kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=.kube/config

kubectl taint nodes --all node-role.kubernetes.io/control-plane-
kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml
kubectl apply -f calico/custom-resources.yaml

calico custom resource

$ cat calico/custom-resources.yaml 
# This section includes base Calico installation configuration.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    # Note: The ipPools section cannot be modified post-install.
    ipPools:
    - blockSize: 26
      cidr: 192.168.0.0/16
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()
    containerIPForwarding: Enabled
---

# This section configures the Calico API server.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer 
metadata: 
  name: default 
spec: {}

multus kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml upf in data plane name space and rest are in control plane name space

kubectl create ns cp
kubectl create ns dp

now free5gc

helm upgrade --install test -n dp \
--set global.n4network.masterIf=eno0 \
--set global.n3network.masterIf=eno0 \
--set global.n6network.masterIf=eno0 \
--set global.n6network.subnetIP="192.168.0.0" \
--set global.n6network.gatewayIP="192.168.0.1" \
--set upf.n6if.ipAddress="192.168.0.3" \
towards5gs/free5gc-upf

enable ip_forward in UPF

kubectl  exec -ti -n dp test-free5gc-upf-upf-6485b99bf9-l6fzt -- bash
apt update
apt install nano tcpdump iptables
uncomment "#net.ipv4.ip_forward=1" in /etc/sysctl.conf

verify
root@test-free5gc-upf-upf-6485b99bf9-l6fzt:/free5gc/upf# cat /proc/sys/net/ipv4/ip_forward
1
helm upgrade --install test -n cp \
--set deployUPF=false \
--set deployWEBUI=false \
--set mongodb.persistence.enabled=false \
--set global.n2network.masterIf=eno0 \
--set global.n3network.masterIf=eno0 \
--set global.n4network.masterIf=eno0 \
--set global.n6network.masterIf=eno0 \
--set global.n9network.masterIf=eno0 \
towards5gs/free5gc

Edit web ui service change from to nodeport clusterIP kubectl edit service -n cp webui-service

Port forward kubectl port-forward -n cp services/webui-service 5000

add the subscriber in webui

install ueransim simulator on a newly created sim namespace.

helm install sim -n sim --create-namespace \
--set global.n2network.masterIf=ens3 \
--set global.n3network.masterIf=ens3 \
towards5gs/ueransim

export POD_NAME=$(kubectl get pods --namespace sim -l "component=ue" -o jsonpath="{.items[0].metadata.name}")

$ kubectl --namespace sim exec -it $POD_NAME -- ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 26:76:42:85:b5:5f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.47.211/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::2476:42ff:fe85:b55f/64 scope link 
       valid_lft forever preferred_lft forever
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.1.0.1/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
    inet6 fe80::7cf1:ec24:48e:12d4/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

But ping fails through uesimtun0

$ kubectl --namespace sim exec -it $POD_NAME -- ping -c 1 -I uesimtun0 www.google.com
PING www.google.com (172.217.169.36) from 10.1.0.1 uesimtun0: 56(84) bytes of data.

--- www.google.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

command terminated with exit code 1

Details about cluster

$ kubectl get pods -o wide -A
NAMESPACE          NAME                                        READY   STATUS    RESTARTS   AGE   IP            
calico-apiserver   calico-apiserver-8c7944fd9-tvh9k            1/1     Running   0          81m   192.168.47.196
calico-apiserver   calico-apiserver-8c7944fd9-vvmsg            1/1     Running   0          81m   192.168.47.197
calico-system      calico-kube-controllers-6b57db7fd6-7n4hj    1/1     Running   0          82m   192.168.47.195
calico-system      calico-node-mh5bx                           1/1     Running   0          82m   10.237.72.160 
calico-system      calico-typha-6bdddf499-t4zx2                1/1     Running   0          82m   10.237.72.160 
cp                 mongodb-0                                   1/1     Running   0          48m   192.168.47.207
cp                 test-free5gc-amf-amf-57f6cb85f9-rtdhc       1/1     Running   0          48m   192.168.47.206
cp                 test-free5gc-ausf-ausf-5c57578f4c-nlgbg     1/1     Running   0          48m   192.168.47.203
cp                 test-free5gc-nrf-nrf-79df6c49d-sfh87        1/1     Running   0          48m   192.168.47.201
cp                 test-free5gc-nssf-nssf-fd7b87cc4-nwbbs      1/1     Running   0          48m   192.168.47.209
cp                 test-free5gc-pcf-pcf-9b7fcc57c-x8n6f        1/1     Running   0          48m   192.168.47.208
cp                 test-free5gc-smf-smf-67cdd4d846-xnrbg       1/1     Running   0          48m   192.168.47.202
cp                 test-free5gc-udm-udm-874f96955-56m4n        1/1     Running   0          48m   192.168.47.205
cp                 test-free5gc-udr-udr-6946f7db57-v9882       1/1     Running   0          48m   192.168.47.204
cp                 test-free5gc-webui-webui-6d788974b4-rmqs9   1/1     Running   0          48m   192.168.47.210
dp                 test-free5gc-upf-upf-6485b99bf9-l6fzt       1/1     Running   0          50m   192.168.47.200
sim                sim-ueransim-gnb-854d9496b6-5q5fd           1/1     Running   0          42m   192.168.47.212
sim                sim-ueransim-ue-86d5fbfd99-k6sxh            1/1     Running   0          42m   192.168.47.211
tigera-operator    tigera-operator-6bb5985474-zgqfz            1/1     Running   0          82m   10.237.72.160

Some extra information from UPF

root@test-free5gc-upf-upf-6485b99bf9-l6fzt:/free5gc/upf# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 1a:54:67:60:7c:8e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.47.200/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1854:67ff:fe60:7c8e/64 scope link 
       valid_lft forever preferred_lft forever
4: n3@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether fe:fb:d7:bd:0d:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.50.233/29 brd 10.100.50.239 scope global n3
       valid_lft forever preferred_lft forever
    inet6 fe80::fcfb:d7ff:febd:de8/64 scope link 
       valid_lft forever preferred_lft forever
5: n6@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether f6:23:e2:d4:65:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.3/24 brd 192.168.0.255 scope global n6
       valid_lft forever preferred_lft forever
    inet6 fe80::f423:e2ff:fed4:650c/64 scope link 
       valid_lft forever preferred_lft forever
6: n4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:d0:92:d5:4d:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.50.241/29 brd 10.100.50.247 scope global n4
       valid_lft forever preferred_lft forever
    inet6 fe80::d0:92ff:fed5:4d13/64 scope link 
       valid_lft forever preferred_lft forever
7: upfgtp: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1464 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet6 fe80::7fca:b50d:c9df:de15/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

root@test-free5gc-upf-upf-6485b99bf9-l6fzt:/free5gc/upf# ip route
default via 169.254.1.1 dev eth0 
10.1.0.0/17 dev upfgtp proto static 
10.100.50.232/29 dev n3 proto kernel scope link src 10.100.50.233 
10.100.50.240/29 dev n4 proto kernel scope link src 10.100.50.241 
169.254.1.1 dev eth0 scope link 
192.168.0.0/24 dev n6 proto kernel scope link src 192.168.0.3 
root@test-free5gc-upf-upf-6485b99bf9-l6fzt:/free5gc/upf# tcpdump -nei any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:54:11.475623  In ethertype IPv4 (0x0800), length 100: 10.1.0.1 > 172.217.169.36: ICMP echo request, id 46, seq 1, length 64
11:54:11.475707 Out f6:23:e2:d4:65:0c ethertype IPv4 (0x0800), length 100: 192.168.0.3 > 172.217.169.36: ICMP echo request, id 46, seq 1, length 64
root@test-free5gc-upf-upf-6485b99bf9-l6fzt:/free5gc/upf# iptables -nvL POSTROUTING -t nat
Chain POSTROUTING (policy ACCEPT 89 packets, 17126 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2   168 MASQUERADE  all  --  *      n6      10.1.0.0/16          0.0.0.0/0

some details from the host system

$ sudo iptables -nvL POSTROUTING -t nat
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
25859 1434K cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
17725 1071K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a4:bf:01:6b:f8:f0 brd ff:ff:ff:ff:ff:ff
    altname enp5s0
    inet 10.237.72.160/24 metric 100 brd 10.237.72.255 scope global dynamic eno0
       valid_lft 40030sec preferred_lft 40030sec
    inet6 fe80::a6bf:1ff:fe6b:f8f0/64 scope link 
       valid_lft forever preferred_lft forever
5: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 66:ae:fe:7c:7a:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.47.192/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::64ae:feff:fe7c:7a02/64 scope link 
       valid_lft forever preferred_lft forever
6: calib60157fc38c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-8ef93e66-1599-cb33-cb25-dc17859f684f
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
7: caliac337299544@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f49ce13d-9504-d124-65c4-1972bd887d88
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
8: cali33187d472f0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-a6eaffa8-fbfc-9067-32a0-25dac7dd01a4
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
9: cali29d08354e8f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-b6107500-32b6-f145-f2e2-e9df01fa7298
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
10: cali0d3d1dd5b15@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-ef39e33a-590b-8ae9-1133-3559a5618eb3
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
13: cali76d4c0a4623@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f2d86246-ad94-0e8d-da79-ce1d5cd66b6b
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
14: cali9eab4134b24@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f6fefd35-361a-c6f2-605b-37878800dbc0
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
15: cali20dc5627902@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-311c0b22-890f-290d-7902-8958814f9f48
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
16: cali65bcbe4d3b4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c1a2defc-970f-07e5-d0bb-73294cc06ec5
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
17: cali4b25d95aa41@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f1b8f38d-5f22-b793-08a9-ddd3088a76dd
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
18: cali84bff32026a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-2e283c20-537c-d55a-247b-fe45f45175aa
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
19: cali78bbe3202b8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-0e54611c-ee5d-3177-2646-cafc9fe7f994
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
20: cali9a1938603a4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-b0f43190-61df-3dc8-3e06-b12e5d516fc9
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
21: calie9deb3528f1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-821a95d1-c019-5995-f07a-f70adb4afb04
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
22: cali21abca10974@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-0956cd05-aaa5-7076-b321-80a1b6a99a28
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
23: cali6dc13f1c0d9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-52d629a1-c299-568a-4c7e-58b77d7c773a
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
24: cali13cf85f9d1e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f7b13497-7640-6ef8-4224-47e63bb428b8
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
25: cali3887821ffef@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-d275e4c4-24ed-859c-e7bf-d5b8ead52a7b
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.237.72.1     0.0.0.0         UG    100    0        0 eno0
10.184.9.1      10.237.72.1     255.255.255.255 UGH   100    0        0 eno0
10.237.72.0     0.0.0.0         255.255.255.0   U     100    0        0 eno0
10.237.72.1     0.0.0.0         255.255.255.255 UH    100    0        0 eno0
10.248.2.1      10.237.72.1     255.255.255.255 UGH   100    0        0 eno0
163.33.253.68   10.237.72.1     255.255.255.255 UGH   100    0        0 eno0
192.168.47.192  0.0.0.0         255.255.255.192 U     0      0        0 *
192.168.47.193  0.0.0.0         255.255.255.255 UH    0      0        0 calib60157fc38c
192.168.47.194  0.0.0.0         255.255.255.255 UH    0      0        0 caliac337299544
192.168.47.195  0.0.0.0         255.255.255.255 UH    0      0        0 cali33187d472f0
192.168.47.196  0.0.0.0         255.255.255.255 UH    0      0        0 cali29d08354e8f
192.168.47.197  0.0.0.0         255.255.255.255 UH    0      0        0 cali0d3d1dd5b15
192.168.47.200  0.0.0.0         255.255.255.255 UH    0      0        0 cali76d4c0a4623
192.168.47.201  0.0.0.0         255.255.255.255 UH    0      0        0 cali9eab4134b24
192.168.47.202  0.0.0.0         255.255.255.255 UH    0      0        0 cali20dc5627902
192.168.47.203  0.0.0.0         255.255.255.255 UH    0      0        0 cali65bcbe4d3b4
192.168.47.204  0.0.0.0         255.255.255.255 UH    0      0        0 cali4b25d95aa41
192.168.47.205  0.0.0.0         255.255.255.255 UH    0      0        0 cali84bff32026a
192.168.47.206  0.0.0.0         255.255.255.255 UH    0      0        0 cali78bbe3202b8
192.168.47.207  0.0.0.0         255.255.255.255 UH    0      0        0 cali9a1938603a4
192.168.47.208  0.0.0.0         255.255.255.255 UH    0      0        0 calie9deb3528f1
192.168.47.209  0.0.0.0         255.255.255.255 UH    0      0        0 cali21abca10974
192.168.47.210  0.0.0.0         255.255.255.255 UH    0      0        0 cali6dc13f1c0d9
192.168.47.211  0.0.0.0         255.255.255.255 UH    0      0        0 cali13cf85f9d1e
192.168.47.212  0.0.0.0         255.255.255.255 UH    0      0        0 cali3887821ffef
$ sudo tcpdump -nei any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
14:00:20.973772 eno0  Out ifindex 2 f6:23:e2:d4:65:0c ethertype IPv4 (0x0800), length 104: 192.168.0.3 > 172.217.169.36: ICMP echo request, id 52, seq 1, length 64
rveerama1 commented 1 year ago

I tried with flannel too. same issue exists.

rveerama1 commented 1 year ago

@raoufkh can you please take a look and help me.

amrbaraka commented 1 year ago

@rveerama1 I think the problem is that you've configured your N6 network details different than your host network:

As per installation instructions: "we'll suppose that you have only one interface on each Kubernetes node and its name is toto. Then you have to set these parameters to toto: global.n2network.masterIf global.n3network.masterIf global.n4network.masterIf global.n6network.masterIf global.n9network.masterIf

In addition, please make sure global.n6network.subnetIP, global.n6network.gatewayIP and free5gc-upf.upf.n6if.ipAddress parameters will match the IP address of the toto interface in order to make the UPF able to reach the Data Network via its N6 interface"

So according to your host networking setup:

2: eno0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether a4:bf:01:6b:f8:f0 brd ff:ff:ff:ff:ff:ff altname enp5s0 inet 10.237.72.160/24 metric 100 brd 10.237.72.255 scope global dynamic eno0 valid_lft 40030sec preferred_lft 40030sec inet6 fe80::a6bf:1ff:fe6b:f8f0/64 scope link valid_lft forever preferred_lft forever

$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.237.72.1 0.0.0.0 UG 100 0 0 eno0

Your config should look like this:

--set global.n4network.masterIf=eno0 \ --set global.n3network.masterIf=eno0 \ --set global.n6network.masterIf=eno0 \ --set global.n6network.subnetIP="10.237.72.0" \ --set global.n6network.gatewayIP="10.237.72.1" \ --set upf.n6if.ipAddress="10.237.72.x" \ >>> pick a free IP within subnet 10.237.72.0/24

Br, Amr

rveerama1 commented 1 year ago

@amrbaraka thanks a lot. that worked. I will close the issue.