gNB unable to reach AMF

Hello I am trying to deploy 5G core and UERANSIM on a Kubernetes cluster. I am not using microk8s or anything, I just deployed the cluster using kubeadm. I use Kubernetes v1.22, kernel version 5.4, gtp5g installed on all Nodes. The pod CIDR is 192.168.0.0/16 (default I believe). I have one master and 6 workers.

Each Node has 2 network interface: eth1 is on 192.168.50.1/24, it is the network used for inter-node communication (it is a host-only adapter on VirtualBox) and eth0 is a NAT interface with Internet access.

I use Calico as a CNI, as well as Multus. For Calico, I use kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml without any modification, and it seems to work. For Multus, I used kubectl create -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick-plugin.yml without any modification. It seems like Multus correctly detected Calico.

Everything in the cluster seems to be running fine.

For the configuration, I modified the free5gc's values.yaml file of n2, n3, n4 and n9 to eth1, and n6 to eth0. I did the same for ueransim's values.yaml file. I did not change any IP addresses, subnet or port, as I think they are already consistent with the rest of the deployment (but I might be totally wrong here).

When I deploy free5gc (with Helm), Pods are started normally and without error. Same goes when I deploy ueransim. However, kubectl logs -n free5gc ueransim-gnb-... output:

UERANSIM v3.1.3
[2021-11-24 18:01:46.591] [sctp] [info] Trying to establish SCTP connection... (10.100.50.249:38412)
[2021-11-24 18:06:50.724] [sctp] [error] Connecting to 10.100.50.249:38412 failed. SCTP could not connect: Connection timed out

I tried to get into the Pod and reach the AMF but it says host unreachable.

I have noticed something weird: kubectl -n free5gc describe network-attachment-definition n2network output:

<...>
Spec:
  Config:  { "cniVersion": "0.3.1", "plugins": [ { "type": "macvlan", "capabilities": { "ips": true }, "master": "eth1", "mode": "bridge", "ipam": { "type": "static", "routes": [ { "dst": "0.0.0.0/0", "gw": "10.100.50.254" } ] } }, { "capabilities": { "mac": true }, "type": "tuning" } ] }
Events:    <none>

routes.dst is equal to 0.0.0.0/0. Is it normal? I had a look at that issue and it is pretty different.

I also tried Flannel + Multus but without success. I am new to Multus, I am out of ideas to make that work.

EDIT: Here is the content of /etc/cni/net.d/00-multus.conf on the Node gnB is deployed on. Same goes for the node AMF is deployed on. I don't see anything wrong with that. The interface of correctly created.

{
   "capabilities":{
      "bandwidth":true,
      "portMappings":true
   },
   "cniVersion":"0.3.1",
   "delegates":[
      {
         "cniVersion":"0.3.1",
         "name":"k8s-pod-network",
         "plugins":[
            {
               "datastore_type":"kubernetes",
               "ipam":{
                  "type":"calico-ipam"
               },
               "kubernetes":{
                  "kubeconfig":"/etc/cni/net.d/calico-kubeconfig"
               },
               "log_file_path":"/var/log/calico/cni/cni.log",
               "log_level":"info",
               "mtu":0,
               "nodename":"k8s-node-4",
               "policy":{
                  "type":"k8s"
               },
               "type":"calico"
            },
            {
               "capabilities":{
                  "portMappings":true
               },
               "snat":true,
               "type":"portmap"
            },
            {
               "capabilities":{
                  "bandwidth":true
               },
               "type":"bandwidth"
            }
         ]
      }
   ],
   "logLevel":"verbose",
   "logToStderr":true,
   "kubeconfig":"/etc/cni/net.d/multus.d/multus.kubeconfig",
   "name":"multus-cni-network",
   "type":"multus"
}

Best, Hugo

Hello @hkerma

Can you provide logs of SMF and UPF please?

Hello @raoufkh Sure!

UPF logs

vagrant@k8s-master:~$ kubectl logs -n free5gc free5gc-free5gc-upf-upf-5775999ccc-q8gc2
2021-11-24T17:58:16Z [INFO][UPF][Util] Config: /free5gc/config/..2021_11_24_17_58_09.829101327/upfcfg.yaml
2021-11-24T17:58:16Z [INFO][UPF][Util] UPF config version [1.0.0]
2021-11-24T17:58:16Z [INFO][UPF][Util] Set log level: info
2021-11-24T17:58:16Z [INFO][UPF][Util] DNN routes added, main routing table:
2021-11-24T17:58:16Z [INFO][UPF][Util] DstIp               Gateway        Iface     Priority RtProto   Type
2021-11-24T17:58:16Z [INFO][UPF][Util] 169.254.1.1 /32     0.0.0.0        eth0      0        boot      unicast
2021-11-24T17:58:16Z [INFO][UPF][Util] 10.100.100.0 /24    0.0.0.0        n6        0        kernel    unicast
2021-11-24T17:58:16Z [INFO][UPF][Util] 10.100.50.240 /29   0.0.0.0        n4        0        kernel    unicast
2021-11-24T17:58:16Z [INFO][UPF][Util] 10.100.50.232 /29   0.0.0.0        n3        0        kernel    unicast
2021-11-24T17:58:16Z [INFO][UPF][Util] 10.1.0.0 /17        0.0.0.0        upfgtp    0        static    unicast
2021-11-24T17:58:16Z [INFO][UPF][Util] 0.0.0.0 /0          169.254.1.1    eth0      0        boot      unicast

SMF logs

vagrant@k8s-master:~$ kubectl logs -n free5gc free5gc-free5gc-smf-smf-586799c4bc-25jg4
smf
2021-11-24T17:58:52Z [INFO][SMF][App] SMF version:
        free5GC version: v3.0.6
        build time:      2021-11-16T15:26:02Z
        commit hash:     46644f0a
        commit time:     2021-09-13T13:23:12Z
        go version:      go1.14.4 linux/amd64
2021-11-24T17:58:52Z [INFO][SMF][Init] SMF Log level is set to [info] level
2021-11-24T17:58:52Z [INFO][LIB][NAS] set log level : info
2021-11-24T17:58:52Z [INFO][LIB][NAS] set report call : false
2021-11-24T17:58:52Z [INFO][LIB][NGAP] set log level : info
2021-11-24T17:58:52Z [INFO][LIB][NGAP] set report call : false
2021-11-24T17:58:52Z [INFO][LIB][Aper] set log level : info
2021-11-24T17:58:52Z [INFO][LIB][Aper] set report call : false
2021-11-24T17:58:52Z [INFO][LIB][Path] set log level : info
2021-11-24T17:58:52Z [INFO][LIB][Path] set report call : false
2021-11-24T17:58:52Z [INFO][LIB][OAPI] set log level : info
2021-11-24T17:58:52Z [INFO][LIB][OAPI] set report call : false
2021-11-24T17:58:52Z [INFO][LIB][PFCP] set log level : info
2021-11-24T17:58:52Z [INFO][LIB][PFCP] set report call : false
2021-11-24T17:58:52Z [INFO][SMF][CFG] SMF config version [1.0.2]
2021-11-24T17:58:52Z [INFO][SMF][CFG] UE-Routing config version [1.0.1]
2021-11-24T17:58:52Z [INFO][SMF][CTX] smfconfig Info: Version[1.0.2] Description[SMF initial local configuration]
2021-11-24T17:58:52Z [INFO][SMF][CTX] Endpoints: [10.100.50.233]
2021-11-24T17:58:52Z [INFO][SMF][Init] Server started
2021-11-24T17:58:53Z [INFO][SMF][Init] SMF Registration to NRF {6ef77bff-5f49-4e11-b1cf-22792c0aeaa4 SMF REGISTERED 0 0xc000327e00 0xc000327e40 [] []   [smf-nsmf] [] <nil> [] [] <nil> 0 0 0 area1 <nil> <nil> <nil> <nil> 0xc0002f3000 <nil> <nil> <nil> <nil> <nil> map[] <nil> false 0xc000327c80 false false []}
2021-11-24T17:58:53Z [INFO][SMF][PFCP] Listen on 10.100.50.244:8805
2021-11-24T17:58:53Z [INFO][SMF][App] Send PFCP Association Request to UPF[10.100.50.241]
2021-11-24T17:59:02Z [INFO][LIB][PFCP] Remove Request Transaction [1]

My problem seems to be between AMF and gNB. May I know why you are considering logs from SMF and UPF? Is there something that might have went wrong there and had impact on the N2 interface?

EDIT: I tried to disable the firewall (sudo systemctl stop ufw on all the Nodes) and I also enabled "promiscuous mode: allow-all" on my VMs host-only interface, following that issue but it still doesn't work.

Thanks a lot

UPDATE: I managed to make it work.

Using macvlan with Multus means that "promiscuous mode" needs to be set to "allow-all" in VirtualBox (ref) while it is "deny" by default. That is the first thing that I needed to change in my config.

Second thing, it turns out VirtualBox has issue managing the promiscuous mode with network network adapters (namely, Intel Pro 1000/MT Desktop, which is default). Following that issue (https://www.virtualbox.org/ticket/6519) docker, I tried to change it to another network adapter (I use Intel Pro 1000/MT Server now) and everything works.

Hello! I'm sorry of not being able to respond yesterday. For the resolution, that's great! I will add a section in troubleshooting to talk about Promiscuous mode because it appears in many issues. Otherwise, I'll not add commands because it depends on the technology or the tool used for virtualization.

Regards, Abderaouf

Hello @hkerma I met the same issue as you. Following your solution, I set the "promiscuous mode" as "allow-all" in VirtualBox, and changed the network adapter to Intel Pro 1000/MT Server. However, gNB still can not connect to AMF. The error is the same. [2021-12-24 13:25:41.425] [sctp] [error] Connecting to 10.100.50.249:38412 failed. SCTP could not connect: Connection timed out I used calico as the CNI and deployed it using kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml. I deployed the Multus the same way as you. As for configurations, I did not change anything. You said that you finally made it work. What's your final configuration? Did you still use calico as the CNI?

Orange-OpenSource / towards5gs-helm

gNB unable to reach AMF #12