search

Orange-OpenSource / towards5gs-helm

Helm charts for deploying 5G network services on Kubernetes
Other
167 stars 130 forks source link

error invoking ConflistDel - "n2network-amf": conflistDel #29

Closed josebeo2016 closed 1 year ago

josebeo2016 commented 2 years ago

Hello everyone.

I'm trying to install towards5gs-helm on my system. The specs: Kubernetes

Network configuration Because the names of network interfaces on your Kubernetes nodes are different from eth0 and eth1, I changed all these parameter into the name of my network interface by modifying the towards5gs-helm/charts/free5gc/values.yaml:

#Global network parametes
  n2network:
    name: n2network
    masterIf: enp0s31f6
    subnetIP: 10.100.50.248
    cidr: 29
    gatewayIP: 10.100.50.254
    excludeIP: 10.100.50.254
  n3network:
    name: n3network
    masterIf: enp0s31f6
    subnetIP: 10.100.50.232
    cidr: 29
    gatewayIP: 10.100.50.238
    excludeIP: 10.100.50.238
  n4network:
    name: n4network
    masterIf: enp0s31f6
    subnetIP: 10.100.50.240
    cidr: 29
    gatewayIP: 10.100.50.246
    excludeIP: 10.100.50.246
  n6network:
    name: n6network
    masterIf: enp0s31f6
    subnetIP: <my IP address>
    cidr: 24
    gatewayIP: <my IP gateway>
    excludeIP: 10.100.100.254
  n9network:
    name: n9network
    masterIf: enp0s31f6
    subnetIP: 10.100.50.224
    cidr: 29
    gatewayIP: 10.100.50.230
    excludeIP: 10.100.50.230

I also tried to change other similar config of the amf, smf and upf chart. But the problem still happens.

The error kubectl get pods -n free5gc

NAME                                              READY   STATUS              RESTARTS   AGE
free5gc-v1-free5gc-amf-amf-89966854-xq5r4         0/1     Init:0/1            0          2m6s
free5gc-v1-free5gc-amf-amf-bc46d5dfc-6qpf4        0/1     Init:0/1            0          2m7s
free5gc-v1-free5gc-ausf-ausf-7c568887cd-249vf     1/1     Running             0          2m6s
free5gc-v1-free5gc-nrf-nrf-bb98d64f8-88vrm        1/1     Running             0          2m7s
free5gc-v1-free5gc-nssf-nssf-87f467897-bnprz      1/1     Running             0          2m7s
free5gc-v1-free5gc-pcf-pcf-7b66cd6494-cmt4z       1/1     Running             0          2m5s
free5gc-v1-free5gc-smf-smf-7489648d4d-h6jds       0/1     Init:0/1            0          2m3s
free5gc-v1-free5gc-smf-smf-76df75cfcf-gljvc       0/1     Init:0/1            0          2m7s
free5gc-v1-free5gc-udm-udm-757d5b546c-xj825       1/1     Running             0          2m5s
free5gc-v1-free5gc-udr-udr-7c7cc97f68-xdmmp       1/1     Running             0          2m4s
free5gc-v1-free5gc-upf-upf-658c9d59-r6r9h         0/1     ContainerCreating   0          2m7s
free5gc-v1-free5gc-upf-upf-666b689578-jtvm7       0/1     ContainerCreating   0          2m4s
free5gc-v1-free5gc-webui-webui-7d9f769968-dfqkc   1/1     Running             0          2m6s
mongodb-0                                         1/1     Running             0          2m7s

kubectl describe pod -n free5gc free5gc-v1-free5gc-amf-amf-89966854-xq5r4

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               55m                  default-scheduler  Successfully assigned free5gc/free5gc-v1-free5gc-amf-amf-89966854-xq5r4 to mec-1
  Normal   AddedInterface          55m                  multus             Add eth0 [10.0.1.83/32] from cilium
  Warning  FailedCreatePodSandBox  55m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "65dd9a802b8b4dc491c4a73d6870ba6532dd20355317ec4bc517b341c9a3d559" network for pod "free5gc-v1-free5gc-amf-amf-89966854-xq5r4": networkPlugin cni failed to set up pod "free5gc-v1-free5gc-amf-amf-89966854-xq5r4_free5gc" network: [free5gc/free5gc-v1-free5gc-amf-amf-89966854-xq5r4/:n2network-amf]: error adding container to network "n2network-amf": Link not found, failed to clean up sandbox container "65dd9a802b8b4dc491c4a73d6870ba6532dd20355317ec4bc517b341c9a3d559" network for pod "free5gc-v1-free5gc-amf-amf-89966854-xq5r4": networkPlugin cni failed to teardown pod "free5gc-v1-free5gc-amf-amf-89966854-xq5r4_free5gc" network: delegateDel: error invoking ConflistDel - "n2network-amf": conflistDel: error in getting result from DelNetworkList: Link not found]
ritokispingvin commented 2 years ago

Hi @josebeo2016 , I'm having a similar issue. Have you managed to find the fix for this? Thank you and brs, Tamas

josebeo2016 commented 2 years ago

Hi @josebeo2016 , I'm having a similar issue. Have you managed to find the fix for this? Thank you and brs, Tamas

I installed both Calico and Cilium working together in my system and the above problem is over. However, I face new issue:

$ >> k -n free5gc logs free5gc-v1-free5gc-upf-upf-658c9d59-qprgg
Cannot find device "n6"
2022-03-08T02:25:45Z [INFO][UPF][Util] Config: /free5gc/config/..2022_03_08_02_25_01.1342527809/upfcfg.yaml
2022-03-08T02:25:45Z [INFO][UPF][Util] UPF config version [1.0.0]
2022-03-08T02:25:45Z [INFO][UPF][Util] Set log level: info
2022-03-08T02:25:45Z [ERRO][UPF][Util] Socket bind fail : Cannot assign requested address
2022-03-08T02:25:45Z [ERRO][UPF][Util] Socket -1 register event in epoll error : Bad file descriptor
2022-03-08T02:25:45Z [ERRO][UPF][Util] UPDK epoll register error
2022-03-08T02:25:46Z [ERRO][UPF][Util] Pool is full, it may not belong to this pool
2022-03-08T02:25:46Z [ERRO][UPF][Util] Gtp5gDeviceAdd failed
2022-03-08T02:25:46Z [ERRO][UPF][Util] Socket bind fail : Cannot assign requested address
2022-03-08T02:25:46Z [ERRO][UPF][Util] Socket -1 register event in epoll error : Bad file descriptor
2022-03-08T02:25:46Z [ERRO][UPF][Util] PFCP Sock Register to epoll error
2022-03-08T02:25:46Z [ERRO][UPF][Util] Create PFCP Server for IPv4 error
2022-03-08T02:25:46Z [ERRO][UPF][Util] UPF - PFCP error when UPF initializes
2022-03-08T02:25:46Z [ERRO][UPF][Util] UPF failed to initialize
2022-03-08T02:25:46Z [ERRO][UPF][Util] Pool is full, it may not belong to this pool
2022-03-08T02:25:46Z [INFO][UPF][Util] Removing DNN routes
2022-03-08T02:25:46Z [ERRO][UPF][Util] if_nametoindex
2022-03-08T02:25:46Z [ERRO][UPF][Util] Delete routing rule to device upfgtp failed: 10.1.0.0/17
2022-03-08T02:25:46Z [ERRO][UPF][Util] UPDK epoll deregister error

I'm working now to fix the GTP5g module in my worker node now. I will update soon

raoufkh commented 2 years ago

I'm sorry Cilium has never been tested as the only and primary CNI plugin on the cluster. For the GTP5g kernel module, I'm using the tag v0.3.2 of this repo https://github.com/free5gc/gtp5g.git and it works well

josebeo2016 commented 2 years ago

Thank you for your suggestion. I will try to install the gtp5g v0.3.2. Last work, I tried the latest gtp5g version with self-modification on the original gtp5g source code because I use the Linux kernel 5.11.0.

image
raoufkh commented 2 years ago

Okay let me know about the result of gtp5g v0.3.2

raoufkh commented 1 year ago

Hi

Did this work for you? I am closing the issue but don't hesitate to re-open it if necessary

Raouf

lyhcc commented 1 year ago

It is not because of gtp5g. Network Interface in Ubuntu 18.04 is not reprented by eth0, so some configuration about network in grub are required changing. 

GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

and then execute sudo grub-mkconfig -o /boot/grub/grub.cfg

lyhcc commented 1 year ago

The reason is that eth0 and eth1 are used to configure N3, N4, N6 etc.