Data Network unreachable on the n6 interface

[hkerma]

Hello I am opening a second issue since I now have a different problem.

I am trying to deploy 5G core and UERANSIM on a Kubernetes cluster. I am not using microk8s or anything, I just deployed the cluster using kubeadm. I use Kubernetes v1.22, kernel version 5.4, gtp5g installed on all Nodes. The pod CIDR is 192.168.0.0/16 (default I believe). I have one master and 6 workers.

Each Node has 2 network interface: eth1 is on 192.168.56.1/24, it is the network used for inter-node communication (it is a host-only adapter on VirtualBox) and eth0 is a NAT interface with Internet access. I have enabled promiscuous mode in VirtualBox and I work with the AMD PCNet FAST III (Am79C973) NIC (I tried with the Intel Pro 1000 and I have the same issue).

Calico is working fine. IP forwarding in Pod is enabled and I have 1 when I run cat /proc/sys/net/ipv4/ip_forward.

The issue is that I cannot deploy the UPF: it is stuck at ContainerCreating because of the following Multus error:

Warning  FailedCreatePodSandBox  1s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc =
 failed to set up sandbox container "600e58049416ff53b4914a0ad4de474e33ad0992a8c35e9aa46217fbabd82396" network for
 pod "free5gc-free5gc-upf-upf-9dd5954bb-8wspc": networkPlugin cni failed to set up pod "free5gc-free5gc-upf-upf-9dd5954
bb-8wspc_free5gc" network: [free5gc/free5gc-free5gc-upf-upf-9dd5954bb-8wspc:n6network-free5gc-free5gc-upf]: error adding
 container to network "n6network-free5gc-free5gc-upf": failed to add route '{0.0.0.0 00000000} via 10.0.2.2 dev n6': network is 
unreachable

For the configuration, I modified the free5gc's values.yaml file of n2, n3, n4 and n9 to eth1, and n6 to eth0. N6 is configured as the following ( did not put an excluded IP):

  n6network:
    name: n6network
    masterIf: eth0
    subnetIP: 10.0.2.0
    cidr: 24
    gatewayIP: 10.0.2.2

since my NAT network eth0 is the following:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 08:00:27:73:60:cf brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 82655sec preferred_lft 82655sec
    inet6 fe80::a00:27ff:fe73:60cf/64 scope link
       valid_lft forever preferred_lft forever

and iproute:

default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.56.0/24 dev eth1 proto kernel scope link src 192.168.56.10
...

So apparently the network is unreachable. However, when I change the UPF configuration to whatever else, it is created normally and I can go check in the pod that 10.0.2.2 is the gateway. I start to believe it is a problem with the promiscuous mode. I find myself a bit in the same situation as this issue however there is no such thing as "Forged transmits" in VirtualBox.

Is there a way I use something else than macvlan for Multus, or is it really the best option?

Best,

[raoufkh]

Hello @hkerma

First of all, thank you for the detailed description of the problem.

I think you are missing upf.n6if.IpAddress in the UPF Helm chart. If you are installing all Free5GC components using the main Helm chart you should see free5gc-upf.upf.n6if.IpAddress.

Regards, Abderaouf

[hkerma]

Hello You are a saint. Thank you so much, that was my issue. I set it to 10.0.2.10 arbitrarily and it work. I didn't see it at first because it is not in the main values.yaml file. Is there a reason why all the parameters are not in one values.yaml file?

Thank you so much for the help! Best,

[raoufkh]

Because this parameter is internal to the UPF Helm chart. It's customization is possible in the main free5gc but optional. More details here about global and subchart parameters. The parameter is present in free5gc and free5gc-upf readme files.

Regards, Abderaouf

[raoufkh]

I will close the issue