[sctp] [error]: could not connect: Connection timed out

sam-sre commented 2 years ago

Hi,

I am getting the SCTP could not connect: Connection timed out Error after deploying the open5gsand ueransimcharts.

Chain of events :

I deployed the open5gschart
I created a subscriber in the GUI
I deployed the ueransimchart

All components are deployed in the same namespace called open5gsnamespace. Below are all needed info.

Open5gs This is thevalues.yaml file :

I changed the sd: "ffffff" to be sd: "1" to match the UERANSIM value

image:
  registry: docker.io
  repository: openverso/open5gs
  tag: 2.3.3
  pullPolicy: IfNotPresent
webui:
  image:
    registry: docker.io
    repository: openverso/open5gs-webui
    tag: 2.3.3
    pullPolicy: IfNotPresent
  enabled: true
  resources: {}
  ingress:
    enabled: true  
    certManager: false
    pathType: ImplementationSpecific
    hostname:
    path: /
    annotations: {}
    tls: false
hss:
  enabled: false
  resources: {}
  ### For reference, see `open5gs.yaml`
  logger:
    level: info
    # domain: core,diameter
mme:
  enabled: false
  resources: {}
  logger:
    level: info
  parameter: {}
  gummei:
    plmn_id:
      mcc: 100
      mnc: 10
    mme_gid: 2
    mme_code: 1
  tai:
    plmn_id:
      mcc: 100
      mnc: 10
    tac: 1

  externalService:
    enabled: false
    type: LoadBalancer
    loadBalancerSourceRanges: []
    loadBalancerIP:
    externalTrafficPolicy: Local

sgwc:
  enabled: false
  resources: {}
  logger:
    level: info
    # domain: core,diameter
  parameter:
    no_ipv6: true
  max: {}
  pool: {}

sgwu:
  enabled: false
  resources: {}
  logger:
    level: info
    # domain: core,diameter
  parameter:
    no_ipv6: true
  max: {}
  pool: {}
  externalService:
    enabled: false
    advertiseDomain: "ext.openverso.org"
    type: LoadBalancer
    loadBalancerSourceRanges: []
    loadBalancerIP:
    externalTrafficPolicy: Local

amf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}
  guami:
  - plmn_id:
      mcc: 901
      mnc: 70
    amf_id:
      region: 2
      set: 1
  tai:
  - plmn_id:
      mcc: 901
      mnc: 70
    tac: 1
  plmn_support:
  - plmn_id:
      mcc: 901
      mnc: 70
    s_nssai:
      - sst: "1"
        sd: "1"
  externalService:
    enabled: false
    advertiseDomain: "ext.openverso.org"
    type: LoadBalancer
    loadBalancerSourceRanges: []
    loadBalancerIP:
    externalTrafficPolicy: Local

smf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}
  pdn:
    addr: 10.45.0.1/16

upf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}
  pdn:
    addr: 10.45.0.1/16
  externalService:
    enabled: false
    advertiseDomain: "ext.openverso.org"
    type: LoadBalancer
    loadBalancerSourceRanges: []
    loadBalancerIP:
    externalTrafficPolicy: Local

nrf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}

udr:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}

udm:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}

ausf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}

nssf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}
  nsi:
  - sst: "1"
    sd: "1"

bsf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}

pcf:
  enabled: true
  resources: {}
  logger:
    level: info
  parameter: {}

pcrf:
  enabled: true
  resources: {}
  logger:
    level: info
    # domain: core,diameter
  parameter:

mongodb:
  # mongodb.enabled -- set to 'false' to disable automatically deploying dependent charts
  enabled: true
  auth:
    enabled: false

securityPolicy:
  enabled: false

UERANSIM

This is thevalues.yaml file : I changed these values:

tac: '0001' -> tac: '1' matching it with amf configuration from open5gs
amf: open5gs-amf -> amf: core5g-open5gs-amf because core5g is the my Helm release name and it is added to the K8s service name.

global:
  imageRegistry:

  imagePullSecrets: []
  storageClass:

kubeVersion:
nameOverride:
fullnameOverride:
commonLabels: {}
commonAnnotations: {}
clusterDomain: cluster.local
extraDeploy: []

image:
  registry: docker.io
  repository: openverso/ueransim
  tag: 3.2.2

  pullPolicy: IfNotPresent

  pullSecrets: []

  debug: false

mcc: '901'
mnc: '70'
tac: '1'

UECount: 4
initialMSISDN: '0000000001'
key: 465B5CE8B199B49FAA5F0A2EE238A6BC
op: E8ED289DEBA952E4283B54E88E6183CA
opType: OPC
apn: internet
sst: 1
sd: ffffff

amf: core5g-open5gs-amf

resources:
  limits: {}
  requests: {}

podSecurityContext:
  enabled: false

containerSecurityContext:
  enabled: false

podLabels: {}
podAnnotations: {}
affinity: {}
nodeSelector: {}
tolerations: []
updateStrategy:
  type: RollingUpdate

Subscriber configuration in GUI

Untitled

My Environemnt:

2 Vagrant Boxes (Debian GNU/Linux 10 (buster))
K8s v1.20.11
docker v20.10.10
Helm v3

NAME    STATUS   ROLES                  AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION    CONTAINER-RUNTIME
kube1   Ready    control-plane,master   140m   v1.20.12   172.16.16.100   <none>        Debian GNU/Linux 10 (buster)   4.19.0-16-amd64   docker://20.10.10
kube2   Ready    <none>                 140m   v1.20.12   172.16.16.101   <none>        Debian GNU/Linux 10 (buster)   4.19.0-16-amd64   docker://20.10.10

Logs from gnodeb container

k exec -it ueransim-0 -c gnodeb -n open5gs -- /bin/bash

NOTE: 10.105.160.161 is the IP of the AMF K8s service

Launching gnb: nr-gnb -c gnb.yaml
UERANSIM v3.2.2
[2021-10-28 13:17:55.872] [sctp] [info] Trying to establish SCTP connection... (10.105.160.161:38412)
[2021-10-28 13:17:56.846] [rrc] [debug] UE[1] new signal detected
[2021-10-28 13:17:56.846] [rrc] [debug] UE[2] new signal detected
[2021-10-28 13:17:56.848] [rrc] [debug] UE[3] new signal detected
[2021-10-28 13:17:56.852] [rrc] [debug] UE[4] new signal detected
[2021-10-28 13:23:00.171] [sctp] [error] Connecting to 10.105.160.161:38412 failed. SCTP could not connect: Connection timed out

Logs from ues container

k exec -it ueransim-0 -c ues -n open5gs -- /bin/bash

Launching ue: nr-ue -c ue.yaml
UERANSIM v3.2.2
[2021-10-28 13:17:55.842] [901700000000003|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-10-28 13:17:55.844] [901700000000001|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-10-28 13:17:55.845] [901700000000004|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-10-28 13:17:55.847] [901700000000002|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-10-28 13:17:56.846] [901700000000003|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-10-28 13:17:56.847] [901700000000003|nas] [info] Selected plmn[901/70]
[2021-10-28 13:17:56.847] [901700000000003|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.847] [901700000000003|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.847] [901700000000003|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-10-28 13:17:56.847] [901700000000004|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-10-28 13:17:56.847] [901700000000004|nas] [info] Selected plmn[901/70]
[2021-10-28 13:17:56.847] [901700000000004|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.847] [901700000000004|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.847] [901700000000004|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-10-28 13:17:56.848] [901700000000001|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-10-28 13:17:56.848] [901700000000001|nas] [info] Selected plmn[901/70]
[2021-10-28 13:17:56.848] [901700000000001|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.848] [901700000000001|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.848] [901700000000001|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-10-28 13:17:56.852] [901700000000002|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-10-28 13:17:56.853] [901700000000002|nas] [info] Selected plmn[901/70]
[2021-10-28 13:17:56.853] [901700000000002|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.853] [901700000000002|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:17:56.853] [901700000000002|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-10-28 13:18:01.348] [901700000000001|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-10-28 13:18:01.348] [901700000000004|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-10-28 13:18:01.348] [901700000000003|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-10-28 13:18:01.350] [901700000000002|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-10-28 13:18:28.362] [901700000000002|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:18:28.362] [901700000000002|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:18:28.362] [901700000000002|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-10-28 13:18:28.362] [901700000000003|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:18:28.362] [901700000000003|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-10-28 13:18:28.362] [901700000000003|rrc] [error] Cell selection failure, no suitable or acceptable cell found

gibson3659 commented 2 years ago

I had the same issue. I managed to work around it by using the amf nodeport in the gnb config instead of the service. However, the up data is not flowing from the uesim.

sam-sre commented 2 years ago

@gibson3659

The AMF service has 2 ports, one for SCTP and the other for TCP

core5g-open5gs-amf      ClusterIP    10.97.223.188    <none>     7777/TCP,38412/SCTP         3m15s

Did you changed the AMF K8s service type to NodePort and used that port number?

gibson3659 commented 2 years ago

@sam-sre

I was answering from memory, but my memory was wrong. I hardcoded the ClusterIP into the config.

amfConfigs:

address: ${AMF_ADDR}

to

amfConfigs:

address: 10.100.185.204

sam-sre commented 2 years ago

@gibson3659 The entrypoint.sh is already doing that for me. I have no connectivity problem, I can ping from within the gnodebcontainer to the AMFPod and vise versa, also from the ues container to AMF Pod.

What was your ueransim values.yml file? I am doubting the configurations are the cause.

Any Ideas? @cgiraldo

cgiraldo commented 2 years ago

SCTP feature in kubernetes enter stable in 1.20, so we must use kubernetes >= 1.20. This is ok for your environment. @avrodriguezgrad , any ideas?

sam-sre commented 2 years ago

@cgiraldo Thanks Yes I am using K8s v1.20.12 and SCTP is supported. Can you please share TAC, ST and SD values that you are using in your working environment ... Should those be matching in both the open5s config and the UERANSIM config?

avrodriguezgrad commented 2 years ago

Hi, @sam-sre your sd value is different in the open5gs and ueransim configs, having sd=1 and sd=ffffff. Also, can you share AMF logs? So we can know if there is packet exchange between the gNB and the AMF.

sam-sre commented 2 years ago

Hi @avrodriguezgrad I changed the sd value view times to check if it was the source of the connection problem but it wasnt, that's why I decided to leave the defulat values of sd in both open5gs and ueransim which were different. So now I set the sd=1 value in both the open5gs and ueransim and still having the same error. I am also using tac=1 and sst=1 values in both charts.

Logs from AMF:

k logs core5g-open5gs-amf-0 -n open5gs
Open5GS daemon v2.3.3

11/01 10:27:29.325: [app] INFO: Configuration: '/opt/open5gs/etc/open5gs/amf.yaml' (../lib/app/ogs-init.c:129)
11/01 10:27:29.334: [sbi] INFO: nghttp2_server() [192.168.2.73]:7777 (../lib/sbi/nghttp2-server.c:144)
11/01 10:27:29.334: [amf] INFO: ngap_server() [192.168.2.73]:38412 (../src/amf/ngap-sctp.c:53)
11/01 10:27:29.335: [sctp] INFO: AMF initialize...done (../src/amf/app.c:33)
11/01 10:27:30.352: [sbi] WARNING: [7] Failed to connect to core5g-open5gs-nrf port 7777: Connection refused (../lib/sbi/client.c:476)
11/01 10:27:40.335: [amf] WARNING: [50cbbeee-3afe-41ec-8e47-79f99e1e58fe] Retry to registration with NRF (../src/amf/nf-sm.c:161)
11/01 10:27:40.337: [amf] INFO: [50cbbeee-3afe-41ec-8e47-79f99e1e58fe] NF registred [Heartbeat:10s] (../src/amf/nf-sm.c:198)
11/01 10:27:43.281: [amf] INFO: [528c9262-3afe-41ec-bb81-53fcf2b0cd75] (NRF-notify) NF registered (../src/amf/nnrf-handler.c:182)
11/01 10:27:43.282: [amf] INFO: [528c9262-3afe-41ec-bb81-53fcf2b0cd75] (NRF-notify) NF Profile updated (../src/amf/nnrf-handler.c:201)
11/01 10:28:24.557: [amf] INFO: [71b54a62-3afe-41ec-b147-4bd9880b7f38] (NRF-notify) NF registered (../src/amf/nnrf-handler.c:182)
11/01 10:28:24.557: [amf] INFO: [71b54a62-3afe-41ec-b147-4bd9880b7f38] (NRF-notify) NF Profile updated (../src/amf/nnrf-handler.c:201)

Logs from gnodeb container:

k logs ueransim-0 -c gnodeb -n open5gs
Launching gnb: nr-gnb -c gnb.yaml
UERANSIM v3.2.2
[2021-11-01 10:40:27.263] [sctp] [info] Trying to establish SCTP connection... (10.105.87.15:38412)
[2021-11-01 10:40:28.236] [rrc] [debug] UE[1] new signal detected
[2021-11-01 10:40:28.237] [rrc] [debug] UE[2] new signal detected
[2021-11-01 10:40:28.237] [rrc] [debug] UE[3] new signal detected
[2021-11-01 10:40:28.238] [rrc] [debug] UE[4] new signal detected
[2021-11-01 10:45:31.400] [sctp] [error] Connecting to 10.105.87.15:38412 failed. SCTP could not connect: Connection timed out

Logs from ues container:

k logs ueransim-0 -c ues -n open5gs
Launching ue: nr-ue -c ue.yaml
UERANSIM v3.2.2
[2021-11-01 10:30:53.973] [901700000000003|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-11-01 10:30:53.974] [901700000000001|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-11-01 10:30:53.975] [901700000000002|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-11-01 10:30:53.976] [901700000000004|nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2021-11-01 10:30:54.982] [901700000000002|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-11-01 10:30:54.982] [901700000000003|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-11-01 10:30:54.983] [901700000000001|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-11-01 10:30:54.983] [901700000000004|rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2021-11-01 10:30:54.983] [901700000000003|nas] [info] Selected plmn[901/70]
[2021-11-01 10:30:54.983] [901700000000003|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.983] [901700000000003|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.983] [901700000000003|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:30:54.983] [901700000000002|nas] [info] Selected plmn[901/70]
[2021-11-01 10:30:54.984] [901700000000002|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.984] [901700000000002|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.984] [901700000000002|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:30:54.984] [901700000000004|nas] [info] Selected plmn[901/70]
[2021-11-01 10:30:54.984] [901700000000001|nas] [info] Selected plmn[901/70]
[2021-11-01 10:30:54.985] [901700000000001|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.985] [901700000000001|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.985] [901700000000001|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:30:54.985] [901700000000004|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.985] [901700000000004|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:30:54.985] [901700000000004|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:30:59.481] [901700000000002|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-11-01 10:30:59.481] [901700000000004|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-11-01 10:30:59.482] [901700000000003|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-11-01 10:30:59.482] [901700000000001|nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
[2021-11-01 10:31:26.498] [901700000000003|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000003|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000003|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:31:26.498] [901700000000002|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000002|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000002|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:31:26.498] [901700000000004|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000004|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000004|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:31:26.498] [901700000000001|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000001|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:26.498] [901700000000001|rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2021-11-01 10:31:56.513] [901700000000003|rrc] [warning] Suitable cell selection failed in [1] cells. [0] out of PLMN, [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:56.513] [901700000000003|rrc] [warning] Acceptable cell selection failed in [1] cells. [0] no SI, [0] reserved, [1] barred, ftai [0]
[2021-11-01 10:31:56.513] [901700000000003|rrc] [error] Cell selection failure, no suitable or acceptable cell found

avrodriguezgrad commented 2 years ago

I don't know if this is your problem, but AMF binds to: 11/01 10:27:29.334: [amf] INFO: ngap_server() [192.168.2.73]:38412 (../src/amf/ngap-sctp.c:53) and gNB looks for the AMF at: [2021-11-01 10:40:27.263] [sctp] [info] Trying to establish SCTP connection... (10.105.87.15:38412).

I think it is a k8s connectiviy issue, not related to our chart, so try to hardcode the AMF IP in the ueransim gnb config file and see if that solves your problem.

sam-sre commented 2 years ago

@avrodriguezgrad Thanks for the follow up..

I also tried hardcoding the AMF's Pod IP before but it didnt work, this is the reason I was asking about the meaning of the ${AMF_ADDR} variable, if it is the AMF K8s service IP or the AMF Pod IP.

Here in my current setup, the AMF Pod IP = 192.168.18.7 and the AMF svc IP=10.102.227.5 .. I hard coded the AMF Pod IP in the gNodeB but still getting the same error:

Logs from gnodeb container:

k logs ueransim-0 -c gnodeb -n open5gs
Launching gnb: nr-gnb -c gnb.yaml
UERANSIM v3.2.2
[2021-11-01 16:27:41.972] [sctp] [info] Trying to establish SCTP connection... (192.168.18.7:38412)
[2021-11-01 16:27:42.944] [rrc] [debug] UE[1] new signal detected
[2021-11-01 16:27:42.947] [rrc] [debug] UE[2] new signal detected
[2021-11-01 16:27:42.947] [rrc] [debug] UE[3] new signal detected
[2021-11-01 16:27:42.949] [rrc] [debug] UE[4] new signal detected
[2021-11-01 16:32:46.207] [sctp] [error] Connecting to 192.168.18.7:38412 failed. SCTP could not connect: Connection timed out

connectivity between UERANSIM Pod and AMF pod is fine and ping is working: Ping from gNodeB container to AMF Pod:

root@ueransim-0:/# ping 192.168.18.7
PING 192.168.18.7 (192.168.18.7) 56(84) bytes of data.
64 bytes from 192.168.18.7: icmp_seq=1 ttl=62 time=1.95 ms
64 bytes from 192.168.18.7: icmp_seq=2 ttl=62 time=1.45 ms
64 bytes from 192.168.18.7: icmp_seq=3 ttl=62 time=0.941 ms

I am using Calico and I did some network testing in my K8s cluster,,, It seems network is working normaly.

avrodriguezgrad commented 2 years ago

Can you tell us exactly which steps you took, at a bash level, to deploy your environment? I will recreate them to see if i have the same problem or try to find out what's going on.

Also, can you tell us which version of the chart are you deploying?

cgiraldo commented 2 years ago

It is OK that the binding IP in amf log and the amf connection IP in ueransim differ. ueransim receives the amf hostname through .Values.amf. If you provide the amf k8s service, ueransim will connect to the open5gs-amf service ClusterIP.

Here is a working deployment with kind (https://kind.sigs.k8s.io/). I modifed .Values.sd in ueransim to be the same as open5gs.

kind create cluster

helm repo add openverso https://gradiant.github.io/openverso-charts/

helm repo update

helm install open5gs openverso/open5gs

# Wait for all open5gs pods up and running

# Register UEs in webui.
kubectl port-forward deployment/open5gs-webui 3000:3000
# access http://localhost:3000 user: admin, password:1423
#Enter this information:
# IMSIs: 901700000000001 901700000000002 901700000000003 901700000000004
# KEY: 465B5CE8B199B49FAA5F0A2EE238A6BC
# OPC: E8ED289DEBA952E4283B54E88E6183CA
# Slice Configurations: SST=1 sd=ffffff

helm install ueransim openverso/ueransim --set sd=0xffffff

The open5gs-amf service should look like this:

kubectl describe svc open5gs-amf 
Name:              open5gs-amf
Namespace:         default
Labels:            app.kubernetes.io/component=amf
                   app.kubernetes.io/instance=open5gs
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=open5gs
                   helm.sh/chart=open5gs-0.3.7
Annotations:       meta.helm.sh/release-name: open5gs
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/component=amf,app.kubernetes.io/instance=open5gs,app.kubernetes.io/name=open5gs
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.84.122
IPs:               10.96.84.122
Port:              sbi  7777/TCP
TargetPort:        7777/TCP
Endpoints:         10.244.0.20:7777
Port:              ngap  38412/SCTP
TargetPort:        38412/SCTP
Endpoints:         10.244.0.20:38412
Session Affinity:  None
Events:            <none>

The Endpoints field should point to amf pod IP.

The amf logs:

Open5GS daemon v2.3.3

11/02 10:56:14.057: [app] INFO: Configuration: '/opt/open5gs/etc/open5gs/amf.yaml' (../lib/app/ogs-init.c:129)
11/02 10:56:14.061: [sbi] INFO: nghttp2_server() [10.244.0.20]:7777 (../lib/sbi/nghttp2-server.c:144)
11/02 10:56:14.061: [amf] INFO: ngap_server() [10.244.0.20]:38412 (../src/amf/ngap-sctp.c:53)
11/02 10:56:14.061: [sctp] INFO: AMF initialize...done (../src/amf/app.c:33)
11/02 10:56:14.062: [amf] INFO: [7f3a8580-3bcb-41ec-83d3-f5721325ed03] NF registred [Heartbeat:10s] (../src/amf/nf-sm.c:198)
11/02 10:56:15.640: [amf] INFO: [799cc26e-3bcb-41ec-bcb0-8d1e193ffa9d] (NRF-notify) NF registered (../src/amf/nnrf-handler.c:182)
11/02 10:56:15.640: [amf] INFO: [799cc26e-3bcb-41ec-bcb0-8d1e193ffa9d] (NRF-notify) NF Profile updated (../src/amf/nnrf-handler.c:201)
11/02 10:56:32.291: [amf] INFO: [8a175da2-3bcb-41ec-8919-3d72ac797611] (NRF-notify) NF registered (../src/amf/nnrf-handler.c:182)
11/02 10:56:32.291: [amf] INFO: [8a175da2-3bcb-41ec-8919-3d72ac797611] (NRF-notify) NF Profile updated (../src/amf/nnrf-handler.c:201)
11/02 11:04:00.940: [amf] INFO: gNB-N2 accepted[10.244.0.23]:47061 in ng-path module (../src/amf/ngap-sctp.c:105)
11/02 11:04:00.940: [amf] INFO: gNB-N2 accepted[10.244.0.23] in master_sm module (../src/amf/amf-sm.c:607)
11/02 11:04:00.940: [amf] INFO: [GNB] max_num_of_ostreams : 30 (../src/amf/context.c:854)
11/02 11:04:00.940: [amf] INFO: [Added] Number of gNBs is now 1 (../src/amf/context.c:869)
11/02 11:04:00.971: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:361)
11/02 11:04:00.971: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2005)
11/02 11:04:00.972: [amf] INFO:     RAN_UE_NGAP_ID[1] AMF_UE_NGAP_ID[1] TAC[1] CellID[0x100] (../src/amf/ngap-handler.c:497)
11/02 11:04:00.972: [amf] INFO: [suci-0-901-70-0000-0-0-0000000004] Unknown UE by SUCI (../src/amf/context.c:1385)
11/02 11:04:00.972: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1187)
11/02 11:04:00.972: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:130)
11/02 11:04:00.972: [gmm] INFO: [suci-0-901-70-0000-0-0-0000000004]    SUCI (../src/amf/gmm-handler.c:156)
11/02 11:04:00.972: [app] WARNING: Try to discover [AUSF] (../lib/sbi/path.c:110)
11/02 11:04:00.972: [amf] INFO: [71048eac-3bcb-41ec-a8ac-c3f42ef2414a] (NF-discover) NF registered (../src/amf/nnrf-handler.c:342)
11/02 11:04:00.972: [amf] INFO: [71048eac-3bcb-41ec-a8ac-c3f42ef2414a] (NF-discover) NF Profile updated (../src/amf/nnrf-handler.c:402)
11/02 11:04:00.974: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:361)
11/02 11:04:00.974: [amf] INFO: [Added] Number of gNB-UEs is now 2 (../src/amf/context.c:2005)
11/02 11:04:00.974: [amf] INFO:     RAN_UE_NGAP_ID[2] AMF_UE_NGAP_ID[2] TAC[1] CellID[0x100] (../src/amf/ngap-handler.c:497)
11/02 11:04:00.974: [amf] INFO: [suci-0-901-70-0000-0-0-0000000003] Unknown UE by SUCI (../src/amf/context.c:1385)
11/02 11:04:00.974: [amf] INFO: [Added] Number of AMF-UEs is now 2 (../src/amf/context.c:1187)
11/02 11:04:00.974: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:130)
11/02 11:04:00.974: [gmm] INFO: [suci-0-901-70-0000-0-0-0000000003]    SUCI (../src/amf/gmm-handler.c:156)
11/02 11:04:00.975: [gmm] WARNING: Authentication failure(Synch failure) (../src/amf/gmm-sm.c:523)
11/02 11:04:00.975: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:361)
11/02 11:04:00.975: [amf] INFO: [Added] Number of gNB-UEs is now 3 (../src/amf/context.c:2005)
11/02 11:04:00.975: [amf] INFO:     RAN_UE_NGAP_ID[3] AMF_UE_NGAP_ID[3] TAC[1] CellID[0x100] (../src/amf/ngap-handler.c:497)
11/02 11:04:00.975: [amf] INFO: [suci-0-901-70-0000-0-0-0000000002] Unknown UE by SUCI (../src/amf/context.c:1385)
11/02 11:04:00.975: [amf] INFO: [Added] Number of AMF-UEs is now 3 (../src/amf/context.c:1187)
11/02 11:04:00.975: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:130)
11/02 11:04:00.975: [gmm] INFO: [suci-0-901-70-0000-0-0-0000000002]    SUCI (../src/amf/gmm-handler.c:156)
11/02 11:04:00.977: [gmm] WARNING: Authentication failure(Synch failure) (../src/amf/gmm-sm.c:523)
11/02 11:04:00.979: [gmm] WARNING: Authentication failure(Synch failure) (../src/amf/gmm-sm.c:523)
11/02 11:04:00.979: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:361)
11/02 11:04:00.979: [amf] INFO: [Added] Number of gNB-UEs is now 4 (../src/amf/context.c:2005)
11/02 11:04:00.979: [amf] INFO:     RAN_UE_NGAP_ID[4] AMF_UE_NGAP_ID[4] TAC[1] CellID[0x100] (../src/amf/ngap-handler.c:497)
11/02 11:04:00.979: [amf] INFO: [suci-0-901-70-0000-0-0-0000000001] Unknown UE by SUCI (../src/amf/context.c:1385)
11/02 11:04:00.979: [amf] INFO: [Added] Number of AMF-UEs is now 4 (../src/amf/context.c:1187)
11/02 11:04:00.979: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:130)
11/02 11:04:00.979: [gmm] INFO: [suci-0-901-70-0000-0-0-0000000001]    SUCI (../src/amf/gmm-handler.c:156)
11/02 11:04:00.982: [gmm] WARNING: Authentication failure(Synch failure) (../src/amf/gmm-sm.c:523)
11/02 11:04:00.989: [gmm] INFO: [imsi-901700000000004] Registration complete (../src/amf/gmm-sm.c:1015)
11/02 11:04:00.989: [amf] INFO: [imsi-901700000000004] Configuration update command (../src/amf/nas-path.c:389)
11/02 11:04:00.989: [gmm] INFO:     UTC [2021-11-02T11:04:00] Timezone[0]/DST[0] (../src/amf/gmm-build.c:502)
11/02 11:04:00.989: [gmm] INFO:     LOCAL [2021-11-02T11:04:00] Timezone[0]/DST[0] (../src/amf/gmm-build.c:507)
11/02 11:04:00.989: [amf] INFO: [Added] Number of AMF-Sessions is now 1 (../src/amf/context.c:2017)
11/02 11:04:00.989: [gmm] INFO: UE SUPI[imsi-901700000000004] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1041)
11/02 11:04:00.989: [app] WARNING: Try to discover [NSSF] (../lib/sbi/path.c:110)
11/02 11:04:00.989: [amf] INFO: [7e800002-3bcb-41ec-b9b6-773f1489741f] (NF-discover) NF registered (../src/amf/nnrf-handler.c:342)
11/02 11:04:00.989: [amf] INFO: [7e800002-3bcb-41ec-b9b6-773f1489741f] (NF-discover) NF Profile updated (../src/amf/nnrf-handler.c:402)
11/02 11:04:00.990: [amf] INFO: [7c6a7748-3bcb-41ec-a3f8-01e96184335a] (NF-discover) NF registered (../src/amf/nnrf-handler.c:342)
11/02 11:04:00.990: [amf] INFO: [7c6a7748-3bcb-41ec-a3f8-01e96184335a] (NF-discover) NF Profile updated (../src/amf/nnrf-handler.c:402)
11/02 11:04:00.995: [gmm] INFO: [imsi-901700000000003] Registration complete (../src/amf/gmm-sm.c:1015)
11/02 11:04:00.995: [amf] INFO: [imsi-901700000000003] Configuration update command (../src/amf/nas-path.c:389)
11/02 11:04:00.995: [gmm] INFO:     UTC [2021-11-02T11:04:00] Timezone[0]/DST[0] (../src/amf/gmm-build.c:502)
11/02 11:04:00.995: [gmm] INFO:     LOCAL [2021-11-02T11:04:00] Timezone[0]/DST[0] (../src/amf/gmm-build.c:507)
11/02 11:04:00.995: [amf] INFO: [Added] Number of AMF-Sessions is now 2 (../src/amf/context.c:2017)
11/02 11:04:00.995: [gmm] INFO: UE SUPI[imsi-901700000000003] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1041)
11/02 11:04:00.995: [gmm] INFO: [imsi-901700000000002] Registration complete (../src/amf/gmm-sm.c:1015)
11/02 11:04:00.995: [amf] INFO: [imsi-901700000000002] Configuration update command (../src/amf/nas-path.c:389)
11/02 11:04:00.995: [gmm] INFO:     UTC [2021-11-02T11:04:00] Timezone[0]/DST[0] (../src/amf/gmm-build.c:502)
11/02 11:04:00.995: [gmm] INFO:     LOCAL [2021-11-02T11:04:00] Timezone[0]/DST[0] (../src/amf/gmm-build.c:507)
11/02 11:04:00.995: [amf] INFO: [Added] Number of AMF-Sessions is now 3 (../src/amf/context.c:2017)
11/02 11:04:00.995: [gmm] INFO: UE SUPI[imsi-901700000000002] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1041)
11/02 11:04:01.203: [gmm] INFO: [imsi-901700000000001] Registration complete (../src/amf/gmm-sm.c:1015)
11/02 11:04:01.203: [amf] INFO: [imsi-901700000000001] Configuration update command (../src/amf/nas-path.c:389)
11/02 11:04:01.203: [gmm] INFO:     UTC [2021-11-02T11:04:01] Timezone[0]/DST[0] (../src/amf/gmm-build.c:502)
11/02 11:04:01.203: [gmm] INFO:     LOCAL [2021-11-02T11:04:01] Timezone[0]/DST[0] (../src/amf/gmm-build.c:507)
11/02 11:04:01.203: [amf] INFO: [Added] Number of AMF-Sessions is now 4 (../src/amf/context.c:2017)
11/02 11:04:01.203: [gmm] INFO: UE SUPI[imsi-901700000000001] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1041)

The ueransim gnb logs:

Launching gnb: nr-gnb -c gnb.yaml
UERANSIM v3.2.2
[2021-11-02 11:04:00.939] [sctp] [info] Trying to establish SCTP connection... (10.96.84.122:38412)
[2021-11-02 11:04:00.940] [sctp] [info] SCTP connection established (10.96.84.122:38412)
[2021-11-02 11:04:00.940] [sctp] [debug] SCTP association setup ascId[27]
[2021-11-02 11:04:00.941] [ngap] [debug] Sending NG Setup Request
[2021-11-02 11:04:00.941] [ngap] [debug] NG Setup Response received
[2021-11-02 11:04:00.941] [ngap] [info] NG Setup procedure is successful
[2021-11-02 11:04:00.970] [rrc] [debug] UE[1] new signal detected
[2021-11-02 11:04:00.971] [rrc] [debug] UE[2] new signal detected
[2021-11-02 11:04:00.971] [rrc] [debug] UE[3] new signal detected
[2021-11-02 11:04:00.971] [rrc] [debug] UE[4] new signal detected
[2021-11-02 11:04:00.971] [rrc] [info] RRC Setup for UE[3]
[2021-11-02 11:04:00.971] [rrc] [info] RRC Setup for UE[1]
[2021-11-02 11:04:00.971] [ngap] [debug] Initial NAS message received from UE[3]
[2021-11-02 11:04:00.971] [ngap] [debug] Initial NAS message received from UE[1]
[2021-11-02 11:04:00.974] [rrc] [info] RRC Setup for UE[4]
[2021-11-02 11:04:00.975] [ngap] [debug] Initial NAS message received from UE[4]
[2021-11-02 11:04:00.978] [rrc] [info] RRC Setup for UE[2]
[2021-11-02 11:04:00.978] [ngap] [debug] Initial NAS message received from UE[2]
[2021-11-02 11:04:00.987] [ngap] [debug] Initial Context Setup Request received
[2021-11-02 11:04:00.989] [ngap] [debug] Initial Context Setup Request received
[2021-11-02 11:04:00.989] [ngap] [debug] Initial Context Setup Request received
[2021-11-02 11:04:00.996] [ngap] [info] PDU session resource(s) setup for UE[3] count[1]
[2021-11-02 11:04:01.000] [ngap] [info] PDU session resource(s) setup for UE[1] count[1]
[2021-11-02 11:04:01.001] [ngap] [info] PDU session resource(s) setup for UE[4] count[1]
[2021-11-02 11:04:01.001] [ngap] [debug] Initial Context Setup Request received
[2021-11-02 11:04:01.214] [ngap] [info] PDU session resource(s) setup for UE[2] count[1]

You can check the ues tunnels to the open5gs core with:

kubectl exec -ti ueransim-0 -c ues -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 5e:c6:d9:c2:45:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.0.23/24 brd 10.244.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5cc6:d9ff:fec2:4547/64 scope link 
       valid_lft forever preferred_lft forever
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.2/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
    inet6 fe80::6825:d1a3:93a:b54b/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
5: uesimtun1: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.3/32 scope global uesimtun1
       valid_lft forever preferred_lft forever
    inet6 fe80::508b:8b60:22c8:b5cb/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
6: uesimtun2: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.4/32 scope global uesimtun2
       valid_lft forever preferred_lft forever
    inet6 fe80::4a43:aa11:4ce1:4e2b/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
7: uesimtun3: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.5/32 scope global uesimtun3
       valid_lft forever preferred_lft forever
    inet6 fe80::857:d42a:74e2:beca/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

sam-sre commented 2 years ago

@avrodriguezgrad Thanks for the follow up.. I am using an automated environment inspired by Jeff Geerling's work here

I am running an Ubuntu VM, inside this VM the Ansible playbooks are building K8s cluster (via kubeadm) with 1Master & 1Worker inside 2 vagrant boxes. The playbooks also deploy Helm, Calico, NFS Dynamic provisioner, MetalLB, Nginx controller, metrics server and monitoring stack (Promethues & Grafana) among other stuff.

I am using the vagrant box image generic/ubuntu2004 but I also used other images to make sure the Linux image is not the problem.

This is my testing/development environemnt where I tested many different projects with no issues so far.

Have you tested the Open5gs charts within environemnts other than KinD ? e.g K3s or Kubeadm

BR

sam-sre commented 2 years ago

@cgiraldo Thanks for the follow up..

I tried it with KinD and it works !

Have you tested the Open5gs charts within environemnts other than KinD ? e.g K3s or Kubeadm

BR

cgiraldo commented 2 years ago

Yes. We have it working with a kubeadm cluster with 1 master and 3 workers in ubuntu 20.04 in Dell servers. Our current CNI is flannel. Maybe Calico is having problems with SCTP? Maybe related with this https://github.com/projectcalico/calico/issues/2570 ?

sam-sre commented 2 years ago

@cgiraldo Calico announced SCTP support since 2018, announcement here.

In 2020, we tested a commercial 5G Core fully containerized and Calico was the CNI in use, so I dont think this is the case here. I am trying now Cilium CNI and I am getting the same results.

Flannel is not an option in my case since we want to perform tests related to eBPF/VPP later on and those Dataplane options are available in Calico and Cilium but not in Flannel.

Can you please share the kernel version used when you deployed it via kubeadm and Ubuntu20.04

cgiraldo commented 2 years ago

Linux 5.4.0-88-generic #99-Ubuntu SMP

cgiraldo commented 2 years ago

We have tested kind + cilium (https://docs.cilium.io/en/v1.9/gettingstarted/kind/).

We DON'T have SCTP connectivity between ueransim gnodeb and open5gs amf.

maybe related with SCTP support in cilium? (https://github.com/cilium/cilium/pull/6912)

We will try with kind + calico.

sam-sre commented 2 years ago

@cgiraldo Thanks for following up.

In our team we tried also Ciliumwith a commercial 5G Core (in Nov 2020) and it worked just fine .. This one was a big 5G deployment with heavy performance testing and we didnt encounter any CNI related problems.

Yes. We have it working with a kubeadm cluster with 1 master and 3 workers in ubuntu 20.04 in Dell servers. Our current CNI is flannel. Maybe Calico is having problems with SCTP? Maybe related with this projectcalico/calico#2570 ?

Can you try Calicowith your Kubeadmdeployment? Did you had to install any special modules/packages here? or you just used Ubuntu20.04 as it is?

cgiraldo commented 2 years ago

We have tested kind + calico and it works.

We have connectivity gnodeb <-> amf and ueransim ues register succesfully in the core.

Regarding out kubeamd deployment, it is a pre-prod environment, so we can't currently change the CNI.

We plan to include Multus to split control plane network and data plane network and try some CNIs for Q1 2022 (kube-ovn, ovn-kubernetes, cilium, etc.).

sam-sre commented 2 years ago

I solved it, now SCTP connection is established. All together I did the following:

I updated kernel version for all K8s nodes.
The NFS provisioner backing up the PVC for MongoDB was not stable, which had an effect on the UEs info added via the WebUI,, now it is fixed.
The sd=0xffffff was not read correctly If it is added via the values.yml file (I downloaded the values.yml file to edit it locally), but it is being read correctly if I use the --set sd=0xffffff option,, not sure why..

Many Thanks for your support..

BR

Gradiant / 5g-charts

[sctp] [error]: could not connect: Connection timed out #27