Expose upf and amf services to external gnodeb

[desimetallica]

Hello, thanks for the great work! I'm using this charts and I would like to use external gnodeb so I want to connect it with the core. I suppose that I need to expose N3 and N2 interfaces between gnodeb and the k8s cluster. I have no loadbalancer at the moment. Do I need to expose the service like node-port instead ClusterIP and probably those one related to UPF and AMF like open-5gs-upf-pfcp,open-5gs-upf-gtpu and open5gs-amf-ngap, open5gs-amf-sbi? It will work? Thanks :)

[cgiraldo]

nodePort range is 30000-32768, so it could be treaky if your gnodeb expect a standard port (i.e., 38412 for N2, 2152 UDP for N3).

I would go with a LoadBalancer. We have test metallb in a baremetal kubernetes without problems.

AFAIK aws loadbalancers does not support SCTP so it is a problem for N2.

It will be enough to configure open5gs-amf-ngap and open5gs-upf-gtpu as LoadBalancers.

Then you have to set config.upf.gtpu.advertise in open5gs-upf values to point to the external IP or hostname of your N3 interface in order to inform the gnodeb how to connect the dataplane.

[desimetallica]

Hi cgiraldo, thank you for your support on this topic, I've configured metalb. Starting from you 5gSA-values.yaml present into your tutorial I've correctly configured the amf.services.ngap.type and also upf.services.gtpu.type to LoadBalancer. Nevertheless I'm facing an issue during the setup of gnb on network and I'm getting Setup Failure. Cause: radioNetwork - slice-not-supported on srsRAN side while on amf log I'm reading Cannot find S_NSSAI. Check 'amf.plmn_support.s_nssai' configuration.

There are some more configuration on core side that I need to perform on the values file?

[desimetallica]

It was a misconfiguration into my values file and with srsRan on sd value. I've commented out sd value from the default configuration. For what i've seen on srsRAn is not possible to configure the sd value...

[cgiraldo]

Hi, Glad to hear you finally fix it!

[desimetallica]

Hello @cgiraldo , I'm moving forward to this task... I'm working with a setup of double x310 one used as gnb and one used as ue. At the moment I'm stuck trying to find the gtpu traffic between gnb and upf.gtpu service. I've modified the values.yaml file to get IP from metalLB and advertise the upf service as u suggested:

upf:
   config:
     gtpu:
       dev: ""
       advertise: "192.168.12.3"
   services:
     gtpu:
       type: LoadBalancer
       loadBalancerIP: "192.168.12.3"

Looking at the logs of amf i can see:

03/20 08:37:01.381: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:361)
03/20 08:37:01.381: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2225)
03/20 08:37:01.381: [amf] INFO: Unknown UE by 5G-S_TMSI[AMF_ID:0x20000,M_TMSI:0x57ab8cab] (../src/amf/ngap-handler.c:431)
03/20 08:37:01.381: [amf] INFO:     RAN_UE_NGAP_ID[4] AMF_UE_NGAP_ID[4] TAC[1] CellID[0x66c001] (../src/amf/ngap-handler.c:497)
03/20 08:37:01.381: [amf] INFO: [suci-0-999-70-0000-0-0-0000000001] known UE by SUCI (../src/amf/context.c:1534)
03/20 08:37:01.381: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:135)
03/20 08:37:01.381: [gmm] INFO: [suci-0-999-70-0000-0-0-0000000001]    SUCI (../src/amf/gmm-handler.c:149)
03/20 08:37:01.403: [amf] INFO: [imsi-999700000000001:1] Release SM context [204] (../src/amf/amf-sm.c:467)
03/20 08:37:01.403: [amf] INFO: [Removed] Number of AMF-Sessions is now 0 (../src/amf/context.c:2246)
03/20 08:37:01.795: [gmm] INFO: [imsi-999700000000001] Registration complete (../src/amf/gmm-sm.c:1124)
03/20 08:37:01.795: [amf] INFO: [imsi-999700000000001] Configuration update command (../src/amf/nas-path.c:430)
03/20 08:37:01.795: [gmm] INFO:     UTC [2023-03-20T08:37:01] Timezone[0]/DST[0] (../src/amf/gmm-build.c:531)
03/20 08:37:01.795: [gmm] INFO:     LOCAL [2023-03-20T08:37:01] Timezone[0]/DST[0] (../src/amf/gmm-build.c:536)
03/20 08:37:01.795: [amf] INFO: [Added] Number of AMF-Sessions is now 1 (../src/amf/context.c:2239)
03/20 08:37:01.795: [gmm] INFO: UE SUPI[imsi-999700000000001] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1062)
03/20 08:37:02.027: [gmm] ERROR: [imsi-999700000000001] No GUTI allocated (../src/amf/gmm-sm.c:346)

On UE I can see the tun_srsue interface created but can't ping the gateway ofc. Unfortunately I'm looking on gnb PC the traffic to the upf service with the following commands I can't see any traffic:

sudo tcpdump -i eno2 dst port 2152 or sudo tcpdump -i eno2 dst port 2123

I'm probably missing something... but I can't figure out what in particular...? Thanks again for the great work here :)

[desimetallica]

But should I see some traffic from gnb on port 2152 or 2123 right?

[desimetallica]

Should I specify some others parameters? like:

services.gtpu.ports.gtpu
services.gtpu.loadBalancerIP
services.gtpu.type

Also in SMF I've seen these parameters. At the moment I've configured just config.upf.gtpu.advertise and dev (is that device?)

[cgiraldo]

dev is the network interface that the service will bind. In your case you are not using multus, so you have to remove the value or set it to the default interface of your container that should be "eth0".

[desimetallica]

Checking inside the pod of UPF, I can see correct process:

udp               UNCONN             0                  0                               10.244.2.232:2152                                0.0.0.0:*                 users:(("open5gs-upfd",pid=1,fd=6))             

Honestly I don't understand why the gnb do not produce any packet to the dest port 2152 to any ip. It should be the ones assigned on advertise parameter, in my case "192.168.12.3" but tcpdump don't capture anything.

[desimetallica]

I've searched inside UPF pod the upf.yaml:

root@open5gs-upf-796d44cbf-fnrxw:~/etc/open5gs/..2023_03_20_10_47_53.528259051# cat upf.yaml 

logger:
  level: info
parameter: {}

upf:
  pfcp:
    - dev: "eth0"
      port: 8805
  gtpu:
    - dev: eth0
      port: 2152
  subnet:
    -
      addr: 10.45.0.1/16
      dev: ogstun
      dnn: internet

smf:
  pfcp:
    - name: open5gs-smf-pfcp
      port: 8805

Don't we need here a line under gtpu with my: advertise: 192.168.12.3

[cgiraldo]

Yes, you need here the line with the adverstise. You have provided the advertise in a wrong "path" .

Check out the values of upf: https://github.com/Gradiant/openverso-charts/blob/15c34efa990f677e5aa24c24d42d45e549af7bed/charts/open5gs-upf/values.yaml#L78

Your values should be:

upf:
   config:
     upf:
       gtpu:
         dev: ""
         advertise: "192.168.12.3"
   services:
     gtpu:
       type: LoadBalancer
       loadBalancerIP: "192.168.12.3"

[desimetallica]

Yes agree with you. But I don't know why if i wrote like that helm gives me an error:

Error: INSTALLATION FAILED: template: open5gs/charts/upf/templates/configmap.yaml:11:3: executing "open5gs/charts/upf/templates/configmap.yaml" at <tpl (.Files.Get "resources/config/upf.yaml") .>: error calling tpl: error during tpl function execution for "{{ $open5gsName := .Release.Name }}

logger:
  level: {{ .Values.config.logLevel }}
parameter: {}

upf:
  pfcp:
    - dev: "eth0"
      port: {{ .Values.containerPorts.pfcp }}
  gtpu:
    - dev: {{ default "eth0" .Values.config.upf.gtpu.dev }}
      port: {{ .Values.containerPorts.gtpu }}
      {{- if .Values.config.upf.gtpu.advertise }}
      advertise: "{{ tpl .Values.config.upf.gtpu.advertise }}"
      {{- end }}
  subnet:
    {{- range .Values.config.subnetList }}
    - {{- omit . "createDev" "enableNAT" | toYaml | nindent 6 }}
    {{- end }}

smf:
  pfcp:
    - name: {{ default (printf "%s-smf-pfcp" $open5gsName) .Values.config.smf.pfcp.hostname }}
      port: {{ default 8805 .Values.config.smf.pfcp.port }}
": template: open5gs/charts/upf/templates/configmap.yaml:15:21: executing "open5gs/charts/upf/templates/configmap.yaml" at <tpl>: wrong number of args for tpl: want 2 got 1

To give you a complete view of the values.yaml:

values.yaml

``` mongodb: persistence: enabled: false hss: enabled: false mme: enabled: false pcrf: enabled: false smf: config: pcrf: enabled: false sgwc: enabled: false sgwu: enabled: false amf: config: guamiList: - plmn_id: mcc: "999" mnc: "70" amf_id: region: 2 set: 1 taiList: - plmn_id: mcc: "999" mnc: "70" tac: [1, 2, 3] plmnList: - plmn_id: mcc: "999" mnc: "70" s_nssai: - sst: 1 #sd: "0x111111" services: ngap: type: LoadBalancer loadBalancerIP: "192.168.12.2" upf: config: upf: gtpu: dev: "eth0" advertise: "192.168.12.3" services: gtpu: type: LoadBalancer loadBalancerIP: "192.168.12.3" extraEnvVars: - name: http_proxy value: http://192.168.90.17:3128 - name: https_proxy value: http://192.168.90.17:3128 - name: no_proxy value: localhost,127.0.0.1 nssf: # enabled: true config: nsiList: - nrf: hostname: "" # if empty default is autogenerated open5gs svc fullname port: 7777 sst: 1 #sd: "0x111111" webui: ingress: enabled: false services: http: ## @param service.type Kubernetes Service type ## type: LoadBalancer loadBalancerIP: "192.168.12.1" populate: enabled: true initCommands: - open5gs-dbctl add_ue_with_apn 999700000000001 465B5CE8B199B49FAA5F0A2EE238A6BC E8ED289DEBA952E4283B54E88E6183CA internet - open5gs-dbctl add_ue_with_apn 999700000000002 465B5CE8B199B49FAA5F0A2EE238A6BC E8ED289DEBA952E4283B54E88E6183CA internet # - open5gs-dbctl add_ue_with_slice 999700000000002 465B5CE8B199B49FAA5F0A2EE238A6BC E8ED289DEBA952E4283B54E88E6183CA internet 1 111111 # - open5gs-dbctl add_ue_with_slice 999700000000002 465B5CE8B199B49FAA5F0A2EE238A6BC E8ED289DEBA952E4283B54E88E6183CA internet 1 111111 ```

[cgiraldo]

Ok. You find a bug in our chart.

https://github.com/Gradiant/openverso-charts/blob/15c34efa990f677e5aa24c24d42d45e549af7bed/charts/open5gs-upf/resources/config/upf.yaml#L15

We miss a point in the tpl function. We should fix it tomorrow. In the meantime, if you want yo fix it locally:

advertise: "{{ tpl .Values.config.upf.gtpu.advertise . }}"

[desimetallica]

Super! :) Thanks!

[cgiraldo]

https://github.com/Gradiant/openverso-charts/releases/tag/open5gs-2.0.10 fix the problem. You can also update the repo (helm repo update) and upgrade the install.

[desimetallica]

Thanks for the super fast fix, I can confirm to see the correct IP of gtp tunnels (192.168.12.3) and also the gtp-TEID now is provided correctly in the ngap packets (PDUSessionResourceSetupResponse) But I'm still trying to understand why I got no connection between UE and CN. Still no traffic between gnb and UPF.

I suppose is something related to S_NSSAI, since with srsRAN something weird happens with SST and SD not sure the config in the core is ok with the UE config:

03/21 09:22:38.420: [gmm] INFO: UE SUPI[imsi-999700000000001] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1062)
03/21 09:22:38.651: [gmm] ERROR: [imsi-999700000000001] No GUTI allocated (../src/amf/gmm-sm.c:346)

[desimetallica]

Update from my side. Although I got this "No GUTI allocated error" the connection is ok. What I forgot to do is to add a table on UE side an entry ‘/etc/iproute2/rt_tables’ and the proper route with ip route add 10.45.0.0/24 dev tun_srsue table TheTableNameAdded