edgecomllc / eupf

5G User Plane Function (UPF) based on eBPF
Apache License 2.0
95 stars 14 forks source link

Check all deployment configurations for relevance #547

Closed PapaySail closed 3 weeks ago

PapaySail commented 1 month ago

Need to deploy from scratch as per instructions. See if there are any bugs and what kind of bugs. Need to strictly follow these instructions https://github.com/edgecomllc/eupf/blob/main/docs%2Fdocs-ru_ru%2Fdeployments%2FREADME.md

Docker-compose deployments:

K8s deployments

PapaySail commented 1 month ago

open5gs-compose

configure firewall `bash fw.sh` flushes rules - it can be harmful
+ Hardcoded `eth0` interface
- and it's not needed for successfull iperf test!
# flush rules iptables -F iptables -X iptables -t raw -F iptables -t raw -X iptables -t nat -F iptables -t nat -X iptables -t mangle -F iptables -t mangle -X Hardcoded `eth0` interface ```ruby iptables -t nat -A POSTROUTING -s 172.20.0.0/16 -d 0.0.0.0/0 -o eth0 -j MASQUERADE iptables -t nat -A POSTROUTING -s 10.46.0.0/16 -o eth0 -j MASQUERADE ```

for i in $(seq 5201 5208); do (iperf3 -s -p $i &) ; done

Iperf needed to be installed

``` sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ for i in $(seq 5201 5208); do (iperf3 -s -p $i &) ; done sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 Command 'iperf3' not found, but can be installed with: sudo apt install iperf3 ```

empty env GNB_IP, UE2 can't start Because of project name is `open5gs-compose`, env in docker.yaml should be `GNB_HOSTNAME: open5gs-compose-gnb-1` ``` sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ sudo docker-compose -f docker-compose.ue2.yaml logs ue2-03 open5gs-compose-ue2-03-1 | GNB_IP: open5gs-compose-ue2-03-1 | Launching ue: nr-ue -c ue.yaml open5gs-compose-ue2-03-1 | UERANSIM v3.2.6 open5gs-compose-ue2-03-1 | terminate called after throwing an instance of 'LibError' open5gs-compose-ue2-03-1 | what(): Bad Inet address: null Resource temporarily unavailable open5gs-compose-ue2-03-1 | /entrypoint.sh: line 41: 14 Aborted (core dumped) nr-ue -c ue.yaml $@ ```
run iperf tests unsuccessfull because of: - hardcoded IPERF_HOST: 10.128.0.14 - TCP checksum error, see #170 TODO to run iperf test: Needed to set to `IPERF_HOST: 172.20.0.1` `sudo ethtool -K br-b8765a6637ca tx off` -- for interface --gateway=172.20.0.1
When fixed, iperf tested successfully

```ruby - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-30.04 sec 240 MBytes 67.0 Mbits/sec receiver [ 5] 29.00-30.00 sec 11.5 MBytes 96.5 Mbits/sec [ 5] 30.00-30.04 sec 324 KBytes 68.6 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-30.04 sec 237 MBytes 66.2 Mbits/sec receiver [ 5] 30.00-30.04 sec 450 KBytes 84.9 Mbits/sec [ 5] 30.00-30.04 sec 288 KBytes 59.1 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ ID] Interval Transfer Bitrate [ 5] 0.00-30.04 sec 246 MBytes 68.6 Mbits/sec receiver [ 5] 0.00-30.04 sec 252 MBytes 70.3 Mbits/sec receiver [ 5] 30.00-30.05 sec 462 KBytes 70.0 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-30.05 sec 242 MBytes 67.5 Mbits/sec receiver [ 5] 30.00-30.05 sec 710 KBytes 118 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-30.05 sec 256 MBytes 71.4 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5203 ----------------------------------------------------------- ----------------------------------------------------------- Server listening on 5206 ----------------------------------------------------------- ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ----------------------------------------------------------- Server listening on 5205 ----------------------------------------------------------- ----------------------------------------------------------- Server listening on 5204 ----------------------------------------------------------- ----------------------------------------------------------- Server listening on 5202 ----------------------------------------------------------- ```

PapaySail commented 1 month ago

Free5GC & ULCL

make commands

  1. make eupf to install eUPF deploy as upfb -- unsuccessfull:
     Events:
      Type     Reason           Age   From               Message
      ----     ------           ----  ----               -------
      Normal   Scheduled        61s   default-scheduler  Successfully assigned free5gculcl/edgecomllc-eupf-6488f49c4b-sn7p7 to edgecom-test-node01
      Warning  SysctlForbidden  61s   kubelet            forbidden sysctl: "net.ipv6.conf.all.forwarding" not allowlisted
    ✔ Details
sergo@edgecom-test-node01:~$ sudo kubelet --allowed-unsafe-sysctls 'net.ipv4.ip_forward,net.ipv6.conf.all.forwarding'
Flag --allowed-unsafe-sysctls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
I0507 20:27:57.482421   37319 server.go:399] "Kubelet version" kubeletVersion="v1.24.3"
I0507 20:27:57.482516   37319 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""

Solution from manual Reconfiguring the kubelet on a running node

sergo@edgecom-test-node01:~$ sudo cp /etc/kubernetes/kubelet-config.yaml new-kubelet-config.yaml
$ sudo nano new-kubelet-config.yaml
sergo@edgecom-test-node01:~$ sudo kubectl -n kube-system create configmap my-node-config --from-file=new-kubelet-config.yaml --append-hash -o yaml
apiVersion: v1
data:
  new-kubelet-config.yaml: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    nodeStatusUpdateFrequency: "10s"
    failSwapOn: True
    authentication:
      anonymous:
        enabled: false
      webhook:
        enabled: True
      x509:
        clientCAFile: /etc/kubernetes/ssl/ca.crt
    authorization:
      mode: AlwaysAllow
    staticPodPath: /etc/kubernetes/manifests
    cgroupDriver: systemd
    containerLogMaxFiles: 5
    containerLogMaxSize: 10Mi
    maxPods: 110
    podPidsLimit: -1
    address: 10.20.253.172
    readOnlyPort: 0
    healthzPort: 10248
    healthzBindAddress: 127.0.0.1
    kubeletCgroups: /systemd/system.slice
    clusterDomain: cluster.local
    protectKernelDefaults: true
    rotateCertificates: true
    clusterDNS:
    - 169.254.25.10
    kubeReserved:
      cpu: 200m
      memory: 512Mi
    resolvConf: "/run/systemd/resolve/resolv.conf"
    allowedUnsafeSysctls:
    - net.ipv4.ip_forward
    - net.ipv6.conf.all.forwarding

    eventRecordQPS: 5
    shutdownGracePeriod: 60s
    shutdownGracePeriodCriticalPods: 20s
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-08T16:06:30Z"
  name: my-node-config-662mcffb5k
  namespace: kube-system
  resourceVersion: "72162413"
  uid: 0ace9431-2b35-494c-a7db-0139df994ef8
sergo@edgecom-test-node01:~$
sergo@edgecom-test-node01:~$ sudo kubectl edit node edgecom-test-node01
node/edgecom-test-node01 edited

add the following YAML under spec:

  configSource:
    configMap:
        name: my-node-config-662mcffb5k
        namespace: kube-system
        kubeletConfigKey: kubelet
sergo@edgecom-test-node01:~$ export NODE_NAME=edgecom-test-node01
sergo@edgecom-test-node01:~$ sudo kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
node/edgecom-test-node01 patched

❌ all above does not work ✔ Ok. Working method:

sudo nano /etc/kubernetes/kubelet-config.yaml 
sudo systemctl restart kubelet


  1. make upf to install Free5gc UPFs CrashLoopBackOff 2024-05-08T17:00:32.904269017Z [ERRO][UPF][Main] UPF Cli Run Error: open Gtp5g: version mismatch: gtp5g version(0.7.3) should be 0.8.1 <= verion < 0.9.0 , please update it

    ✔ Updated gtp5g kernel module

    ```ruby sudo su apt-get update; apt-get install git build-essential -y; \ cd /tmp; \ git clone --depth 1 https://github.com/free5gc/gtp5g.git; \ cd gtp5g/; \ make && make install sergo@edgecom-test-node01:/tmp/gtp5g$ sudo modprobe -r gtp5g sergo@edgecom-test-node01:/tmp/gtp5g$ sudo lsmod | grep ^gtp sergo@edgecom-test-node01:/tmp/gtp5g$ sudo modprobe gtp5g sergo@edgecom-test-node01:/tmp/gtp5g$ sudo lsmod | grep ^gtp gtp5g 126976 0 ```


  1. make ueransim to install gNodeB and UE simulators. UE connects and disconnects
    Details

[2024-05-08 17:11:34.717] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]                                                                                                         [2024-05-08 17:11:36.008] [nas] [debug] PDU Session Establishment Accept received                                                                                                                               [2024-05-08 17:11:36.008] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-08 17:11:36.041] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.1.0.1] is up.                                                                             [2024-05-08 17:11:39.012] [rrc] [debug] Signal lost for cell[2], total [0] cells in coverage                                                                                                                    [2024-05-08 17:11:39.012] [nas] [error] Radio link failure detected
[2024-05-08 17:11:39.012] [nas] [info] UE switches to state [CM-IDLE]                                                                                                                                           [2024-05-08 17:11:39.012] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:39.012] [nas] [info] UE switches to state [MM-REGISTERED/PLMN-SEARCH]
[2024-05-08 17:11:39.012] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:40.139] [rrc] [debug] New signal detected for cell[3], total [1] cells in coverage                                                                                                            [2024-05-08 17:11:41.096] [nas] [error] PLMN selection failure, no cells in coverage
[2024-05-08 17:11:42.482] [rrc] [warning] Acceptable cell selection failed in [1] cells. [1] no SI, [0] reserved, [0] barred, ftai [0]                                                                          [2024-05-08 17:11:42.482] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2024-05-08 17:11:43.298] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:44.399] [nas] [info] UE switches to state [MM-REGISTERED/NO-CELL-AVAILABLE]                                                                                                                   [2024-05-08 17:11:49.539] [nas] [info] Selected plmn[208/93]
[2024-05-08 17:11:49.539] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]                                                                                                                     [2024-05-08 17:11:49.539] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:49.539] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-08 17:11:54.346] [rrc] [debug] Signal lost for cell[3], total [0] cells in coverage                                                                                                                    [2024-05-08 17:11:54.346] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:54.346] [nas] [info] UE switches to state [MM-REGISTERED/PLMN-SEARCH]
[2024-05-08 17:11:54.346] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:55.347] [rrc] [debug] New signal detected for cell[4], total [1] cells in coverage
[2024-05-08 17:11:56.510] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:58.712] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:59.540] [nas] [info] UE switches to state [MM-REGISTERED/NO-CELL-AVAILABLE]
[2024-05-08 17:11:59.541] [nas] [info] Selected plmn[208/93]                                                                                                                                                    [2024-05-08 17:11:59.541] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]                                                                                                                     [2024-05-08 17:11:59.541] [nas] [info] UE switches to state [MM-REGISTERED/PS]
[2024-05-08 17:11:59.541] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-08 17:12:08.216] [nas] [debug] Uplink data status changed PSI[1] pending[true]
[2024-05-08 17:12:08.216] [nas] [debug] Service request required due to [IDLE-UPLINK-DATA-PENDING]
[2024-05-08 17:12:08.216] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_data]
[2024-05-08 17:12:08.216] [nas] [debug] Sending Service Request due to [IDLE-UPLINK-DATA-PENDING]
[2024-05-08 17:12:08.216] [nas] [info] UE switches to state [MM-SERVICE-REQUEST-INITIATED]
[2024-05-08 17:12:08.216] [rrc] [debug] Sending RRC Setup Request
[2024-05-08 17:12:08.553] [rrc] [debug] Signal lost for cell[4], total [0] cells in coverage
[2024-05-08 17:12:08.553] [nas] [error] Active cell change in [CM-IDLE] state while MM specific procedure is ongoing
[2024-05-08 17:12:08.553] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-08 17:12:08.553] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-08 17:12:08.553] [nas] [error] PLMN selection failure, no cells in coverage
[2024-05-08 17:12:09.742] [rrc] [debug] New signal detected for cell[5], total [1] cells in coverage
[2024-05-08 17:12:10.823] [nas] [error] PLMN selection failure, no cells in coverage

💡 The reason of "signal lost" is a fake default gateways in GnB's deployment config:

        annotations:
          k8s.v1.cni.cncf.io/networks: >-
            [ { "name": "n2network-ueransim", "interface": "n2", "ips": [
            "10.100.50.250/29" ], "gateway": [ "10.100.50.254" ] }, { "name":
            "n3network-ueransim", "interface": "n3", "ips": [ "10.100.50.236/28"
            ], "gateway": [ "10.100.50.238" ] } ]

Solution is to remove , "gateway": [ "10.100.50.nnn" ] -- TODO: find a way how to make corrections to a helm deployment.

📝 Known bug: #465

Every peer should be configured to be monitored (see gtp_peer config option)

✔ fixed in #551

OK. See the difference in the first hop of traceroute from UE:

root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build# traceroute -i uesimtun0 www.google.com -w 1 -m3
traceroute to www.google.com (74.125.131.147), 3 hops max, 60 byte packets
 1  10.233.110.134 (10.233.110.134)  2.038 ms  1.972 ms  1.958 ms
 2  188-120-253-172.kube-prometheus-stack-kube-scheduler.kube-system.svc.cluster.local (188.120.253.172)  1.949 ms  1.941 ms  1.933 ms
 3  10.0.0.1 (10.0.0.1)  1.919 ms  1.910 ms  1.887 ms
root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build# traceroute -i uesimtun0 1.1.1.1 -w 1 -m3
traceroute to 1.1.1.1 (1.1.1.1), 3 hops max, 60 byte packets
 1  10.233.110.158 (10.233.110.158)  1.277 ms  1.263 ms  1.256 ms
 2  188-120-253-172.kubernetes.default.svc.cluster.local (188.120.253.172)  1.248 ms  1.239 ms  1.217 ms
 3  10.0.0.1 (10.0.0.1)  1.208 ms  1.197 ms  1.188 ms
root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build#
PapaySail commented 1 month ago

Open5GS & Calico BGP - successfull

name namespace restarts owners node qos age status
eupf-6c4768c9b4-7k45t open5gs 0 ReplicaSet (eupf-6c4768c9b4) edgecom-test-node01 Guaranteed 1715443181000 Running
eupf2-54d595fdd4-66n4x open5gs 0 ReplicaSet (eupf2-54d595fdd4) edgecom-test-node01 Guaranteed 1715443982000 Running
gnb-ueransim-gnb-585fdf8c5b-cmg4f open5gs 0 ReplicaSet (gnb-ueransim-gnb-585fdf8c5b) edgecom-test-node01 BestEffort 1715444893000 Running
open5gs-amf-6647bbf488-276sl open5gs 0 ReplicaSet (open5gs-amf-6647bbf488) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-ausf-6b6c79777b-l94fb open5gs 0 ReplicaSet (open5gs-ausf-6b6c79777b) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-bsf-584bbc4f74-pj9dl open5gs 0 ReplicaSet (open5gs-bsf-584bbc4f74) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-mongodb-6b7cfd5c8d-sj9wg open5gs 0 ReplicaSet (open5gs-mongodb-6b7cfd5c8d) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-nrf-c7dcc4c7d-qg6ll open5gs 0 ReplicaSet (open5gs-nrf-c7dcc4c7d) edgecom-test-node01 BestEffort 1715444456000 Running
open5gs-nssf-ff6c4d95b-7wz9x open5gs 0 ReplicaSet (open5gs-nssf-ff6c4d95b) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-pcf-5c4846dd6b-47xbn open5gs 2 ReplicaSet (open5gs-pcf-5c4846dd6b) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-populate-57cbd7d569-vr2pt open5gs 0 ReplicaSet (open5gs-populate-57cbd7d569) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-smf-7df749f7-2tps5 open5gs 0 ReplicaSet (open5gs-smf-7df749f7) edgecom-test-node01 BestEffort 1715444595000 Running
open5gs-smf2-6bf4d967c-sjwqg open5gs 0 ReplicaSet (open5gs-smf2-6bf4d967c) edgecom-test-node01 BestEffort 1715444801000 Running
open5gs-udm-5f8cfcd-bdmft open5gs 0 ReplicaSet (open5gs-udm-5f8cfcd) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-udr-788f54787-xxjs9 open5gs 2 ReplicaSet (open5gs-udr-788f54787) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-webui-699c9bf9d8-lwtqw open5gs 0 ReplicaSet (open5gs-webui-699c9bf9d8) edgecom-test-node01 BestEffort 1715444455000 Running
ueransim1-ueransim-ues-567d89c5dd-hc8p2 open5gs 0 ReplicaSet (ueransim1-ueransim-ues-567d89c5dd) edgecom-test-node01 BestEffort 1715444957000 Running
ueransim2-ueransim-ues-79669647f6-wzc4f open5gs 0 ReplicaSet (ueransim2-ueransim-ues-79669647f6) edgecom-test-node01 BestEffort 1715445106000 Running

PapaySail commented 1 month ago

Open5GS & Load Balanced eUPF - successfull

Details

```ruby PS C:\Users\SergO\Documents\GitHub\eupf\docs\deployments\open5gs-with-scaling-eupf> kubectl get po -n open5gs -l "app.kubernetes.io/name=eupf" -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES eupf-0 1/1 Running 0 3m40s 10.233.110.136 edgecom-test-node01 eupf-1 1/1 Running 0 3m20s 10.233.110.132 edgecom-test-node01 ``` SMF log: ```ruby 05/13 16:35:22.646: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010) 05/13 16:35:22.646: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051) 05/13 16:35:22.647: [sbi] WARNING: Try to discover [nudm-sdm] (../lib/sbi/path.c:348) 05/13 16:35:22.648: [sbi] INFO: [1bafd4d6-1146-41ef-bd74-d3c6b28ef87a] (NF-discover) NF registered (../lib/sbi/nnrf-handler.c:829) 05/13 16:35:22.648: [sbi] INFO: [1bafd4d6-1146-41ef-bd74-d3c6b28ef87a] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:8505/13 16:35:22.653: [sbi] WARNING: Try to discover [npcf-smpolicycontrol] (../lib/sbi/path.c:348) 05/13 16:35:22.654: [sbi] INFO: [2b90c5d6-1146-41ef-9d4f-e5658e726d87] (NF-discover) NF registered (../lib/sbi/nnrf-handler.c:829) 05/13 16:35:22.654: [sbi] INFO: [2b90c5d6-1146-41ef-9d4f-e5658e726d87] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:85 05/13 16:35:22.661: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.2] IPv6[] (../src/smf/npcf-handler.c:528) 05/13 16:35:22.663: [gtp] INFO: gtp_connect() [10.233.110.136]:2152 (../lib/gtp/path.c:60) 05/13 16:38:04.317: [smf] INFO: [Added] Number of SMF-UEs is now 2 (../src/smf/context.c:1010) 05/13 16:38:04.317: [smf] INFO: [Added] Number of SMF-Sessions is now 2 (../src/smf/context.c:3051) 05/13 16:38:04.325: [smf] INFO: UE SUPI[imsi-999700000000002] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:528) 05/13 16:38:04.327: [gtp] INFO: gtp_connect() [10.233.110.132]:2152 (../lib/gtp/path.c:60) ``` UE1 console: ```ruby bash-5.1# ping -I uesimtun0 10.233.110.136 PING 10.233.110.136 (10.233.110.136): 56 data bytes 64 bytes from 10.233.110.136: seq=0 ttl=64 time=0.815 ms 64 bytes from 10.233.110.136: seq=1 ttl=64 time=0.858 ms 64 bytes from 10.233.110.136: seq=2 ttl=64 time=1.915 ms 64 bytes from 10.233.110.136: seq=3 ttl=64 time=1.381 ms 64 bytes from 10.233.110.136: seq=4 ttl=64 time=1.500 ms ^C --- 10.233.110.136 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 0.815/1.293/1.915 ms bash-5.1# ping -I uesimtun0 10.233.110.132 PING 10.233.110.132 (10.233.110.132): 56 data bytes ^C --- 10.233.110.132 ping statistics --- 15 packets transmitted, 0 packets received, 100% packet loss bash-5.1# ip a s dev uesimtun0 4: uesimtun0: mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet 10.11.0.2/32 scope global uesimtun0 ``` UE2 console: ```ruby bash-5.1# ping -I uesimtun0 10.233.110.136 -c 5 PING 10.233.110.136 (10.233.110.136): 56 data bytes --- 10.233.110.136 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss bash-5.1# ping -I uesimtun0 10.233.110.132 -c 5 PING 10.233.110.132 (10.233.110.132): 56 data bytes 64 bytes from 10.233.110.132: seq=0 ttl=64 time=0.939 ms 64 bytes from 10.233.110.132: seq=1 ttl=64 time=1.139 ms 64 bytes from 10.233.110.132: seq=2 ttl=64 time=1.100 ms 64 bytes from 10.233.110.132: seq=3 ttl=64 time=6.400 ms 64 bytes from 10.233.110.132: seq=4 ttl=64 time=0.976 ms --- 10.233.110.132 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 0.939/2.110/6.400 ms bash-5.1# ip a s dev uesimtun0 4: uesimtun0: mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet 10.11.0.3/32 scope global uesimtun0 ```

PapaySail commented 1 month ago

Open5GS & srsRAN & Calico BGP Check steps

  1. exec shell in UE pod -- wrong command. Correct will be: kubectl -n srs-open5gs exec -ti statefulset/srsran-srsran5gue-zmq-gnb -- /bin/bash
  2. run ICMP test -- unsuccessfull
    Details

PS C:\Users\SergO\Documents\GitHub\eupf\docs\deployments\srsran-gnb> kubectl -n srs-open5gs exec -ti statefulset/srsran-srsran5gue-zmq-gnb -- /bin/bash
Defaulted container "gnb" out of: gnb, ue
root@srsran-srsran5gue-zmq-gnb-0:/# ping -I tun_srsue 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.11.0.2 tun_srsue: 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049ms

root@srsran-srsran5gue-zmq-gnb-0:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if409: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1430 qdisc noqueue state UP group default
    link/ether 66:fc:26:53:2b:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.233.110.137/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::64fc:26ff:fe53:2b14/64 scope link
       valid_lft forever preferred_lft forever
4: tun_srsue: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.2/24 scope global tun_srsue
       valid_lft forever preferred_lft forever
root@srsran-srsran5gue-zmq-gnb-0:/#
root@srsran-srsran5gue-zmq-gnb-0:/# ping -I tun_srsue 10.233.64.11
PING 10.233.64.11 (10.233.64.11) from 10.11.0.2 tun_srsue: 56(84) bytes of data.
^C
--- 10.233.64.11 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9224ms

root@srsran-srsran5gue-zmq-gnb-0:/#

SRSran GnB log:

+ /opt/srsRAN_Project/target/bin/gnb -c gnb.yml
Warning: Scheduling priority of thread "gnb_ctrl" not changed. Cause: Not enough privileges.
Warning: Scheduling priority of thread "du_cell#0" not changed. Cause: Not enough privileges.
Warning: Scheduling priority of thread "phy_worker" not changed. Cause: Not enough privileges.
2024-05-14T18:20:30.022340 [GNB     ] [W] Could not check scaling governor. filename=/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor error="No such file or directory"
2024-05-14T18:20:30.022387 [GNB     ] [W] DRM KMS polling is enabled, which may hinder performance. You can disable it using the "srsran_performance" script
2024-05-14T18:20:30.025034 [IO-EPOLL] [E] epoll_ctl failed for fd=0
2024-05-14T18:20:30.025035 [GNB     ] [E] Couldn't register stdin handler
2024-05-14T18:20:30.213949 [zmq:tx:0:0] [I] Binding to address tcp://127.0.0.1:2000.
2024-05-14T18:20:30.289730 [zmq:rx:0:0] [I] Connecting to address tcp://127.0.0.1:2001.
2024-05-14T18:20:35.823437 [MAC     ] [W] [    66.8] UL rnti=0x4601 lcid=61 CE: Discarding MAC CE. Cause: C-RNTI is not associated with any existing UE
2024-05-14T18:22:13.317015 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:14.341467 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:15.351123 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:18.415462 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001

OK. Solution is to change image to eupf:main 🆒 fact: after restart of upf, smf processes this events and sends command to create session as it was. So, UE gets connectivity immediately.

Details: PFCP restoration

eUPF log after restart: ```ruby [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) [GIN-debug] GET /metrics --> github.com/edgecomllc/eupf/cmd/api/rest.(*ApiHandler).InitMetricsRoute.func1.1 (4 handlers) 2024/05/31 17:41:21 INF running on :8080 2024/05/31 17:41:21 INF running on :9090 2024/05/31 17:41:22 INF Got Association Setup Request from: 10.233.110.143. 2024/05/31 17:41:22 INF Association Setup Request: Node ID: 10.233.110.143 Recovery Time: 2024-05-30 08:13:36 +0000 UTC 2024/05/31 17:41:22 INF Saving new association: &{ID:10.233.110.143 Addr:10.233.110.143 NextSessionID:1 NextSequenceID:1 Sessions:map[] HeartbeatChannel:0xc0000ae4e0 FailedHeartbeats:0 HeartbeatsActive:false Mutex:{state:0 sema:0}} 2024/05/31 17:41:22 INF Got Session Establishment Request from: 10.233.110.143. 2024/05/31 17:41:22 INF Session Establishment Request: CreatePDR ID: 1 FAR ID: 1 QER ID: 1 URR ID: 1 Source Interface: 1 UE IPv4 Address: 10.11.0.3 CreatePDR ID: 2 Outer Header Removal: 0 FAR ID: 2 QER ID: 1 Source Interface: 0 TEID: 5034 Ipv4: 10.233.64.11 Ipv6: UE IPv4 Address: 10.11.0.3 CreatePDR ID: 3 Outer Header Removal: 0 FAR ID: 1 QER ID: 1 Source Interface: 3 TEID: 8912 Ipv4: 10.233.64.11 Ipv6: CreatePDR ID: 4 Outer Header Removal: 0 FAR ID: 3 Source Interface: 0 TEID: 5034 Ipv4: 10.233.64.11 Ipv6: SDF Filter: permit out 58 from ff02::2/128 to assigned CreateFAR ID: 1 Apply Action: [2 0] Forwarding Parameters: Network Instance: internet Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.233.110.176 IPv6Address: PortNumber:0 CTag:0 STag:0} CreateFAR ID: 2 Apply Action: [2 0] Forwarding Parameters: Network Instance: internet CreateFAR ID: 3 Apply Action: [2 0] Forwarding Parameters: Network Instance: internet Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.233.110.143 IPv6Address: PortNumber:0 CTag:0 STag:0} CreateQER ID: 1 Gate Status DL: 0 Gate Status UL: 0 Max Bitrate DL: 999999 Max Bitrate UL: 999999 QFI: 1 CreateURR ID: 1 Measurement Method: 2 Volume Threshold: &{Flags:1 TotalVolume:104857600 UplinkVolume:0 DownlinkVolume:0} CreateBAR ID: 1 2024/05/31 17:41:22 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:2960058634 LocalIP:188803338 TransportLevelMarking:0} 2024/05/31 17:41:22 INF WARN: No OuterHeaderCreation 2024/05/31 17:41:22 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0} 2024/05/31 17:41:22 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:2406410506 LocalIP:188803338 TransportLevelMarking:0} 2024/05/31 17:41:22 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:999999000 MaxBitrateDL:999999000 StartUL:0 StartDL:0} 2024/05/31 17:41:22 Matched groups: ["permit out 58 from ff02::2/128 to assigned" "58" "ff02::2" "128" "" "assigned" "" ""] 2024/05/31 17:41:22 INF Session Establishment Request from 10.233.110.143 accepted. ``` SMF log: ```ruby 05/31 16:28:02.911: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010) 05/31 16:28:02.911: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051) 05/31 16:28:02.919: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:528) 05/31 17:17:05.321: [sbi] INFO: [da4a3af0-15b8-41ef-8e58-3d6de1144d1a] NF expired (../lib/sbi/nf-sm.c:318) 05/31 17:17:05.326: [sbi] INFO: [e9c93eb8-15b8-41ef-aacb-15e80733941f] NF expired (../lib/sbi/nf-sm.c:318) 05/31 17:17:05.374: [sbi] INFO: [34d464ce-1e5a-41ef-9bcf-bd90bf5b1752] NF expired (../lib/sbi/nf-sm.c:318) 05/31 17:41:22.721: [pfcp] ERROR: Remote PFCP restarted [3926161136<3926166081] in Heartbeat RSP (../lib/pfcp/handler.c:76) 05/31 17:41:22.721: [smf] INFO: PFCP de-associated [10.233.64.11]:8805 (../src/smf/pfcp-sm.c:201) 05/31 17:41:22.722: [pfcp] WARNING: F-TEID allocation/release not supported with peer [10.233.64.11]:8805 (../lib/pfcp/handler.c:182) 05/31 17:41:22.722: [smf] INFO: PFCP associated [10.233.64.11]:8805 (../src/smf/pfcp-sm.c:186) 05/31 17:41:22.722: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/pfcp-sm.c:442) 05/31 17:41:22.722: [smf] ERROR: PFCP restoration (../src/smf/pfcp-sm.c:197) ```

Fixed in #552

PapaySail commented 1 month ago

Free5GC & Calico BGP

  1. install eupf make upf :: problem: hardcoded "master": "eth0" in values/eupf.yaml
Pod can't create. Status: ContainerCreating

Pod description: ```ruby Warning FailedCreatePodSandBox 74s (x16 over 103s) kubelet (combined from similar events): Failed to create pod sandbox: rpc err or: code = Unknown desc = failed to setup network for sandbox "7b793c609a914881a32db2f7e3350992cfd1162b17790a985e4e62b6f841f864": plugin type="m ultus" name="multus-cni-network" failed (add): [free5gc24/eupf-57ccfb74bd-m4js5:n3network-free5gc-free5gc-upf]: error adding container to networ k "n3network-free5gc-free5gc-upf": failed to lookup master "eth0": Link not found ```

  1. exec shell in UE pod :: error in doc : correct command is: kubectl -n free5gc exec -ti deployment/ueransim1-ue -- /bin/bash

✔ fixed in #551

PapaySail commented 1 month ago

✔ Docker-compose Free5GC

docker-compose up -d

```ruby sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose up -d [+] Running 21/21 _ Network free5gc-compose_privnet_n6 Created 0.1s _ Network free5gc-compose_default Created 0.1s _ Network free5gc-compose_privnet Created 0.1s _ Container prometheus Started 1.3s _ Container upf Started 1.6s _ Container grafana Started 1.7s _ Container eupf Started 2.6s _ Container nat Started 2.4s _ Container mongodb Started 0.9s _ Container ui Started 1.3s _ Container webui Started 2.1s _ Container nrf Started 2.4s _ Container nssf Started 3.9s _ Container udm Started 3.9s _ Container ausf Started 3.9s _ Container amf Started 3.4s _ Container pcf Started 4.1s _ Container smf Started 4.0s _ Container udr Started 3.9s _ Container ueransim Started 4.2s _ Container n3iwf Started 4.5s sergo@edgecom:~/edgecom24/free5gc-compose$ sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash root@1d4b478b1c91:/ueransim# ./nr-ue -c config/uecfg.yaml UERANSIM v3.2.6 [2024-05-16 17:37:49.842] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH] [2024-05-16 17:37:49.843] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage [2024-05-16 17:37:49.846] [nas] [info] Selected plmn[208/93] [2024-05-16 17:37:52.343] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE] [2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/PS] [2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE] [2024-05-16 17:37:52.343] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE] [2024-05-16 17:37:52.343] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 17:37:52.343] [nas] [debug] Sending Initial Registration [2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-REGISTER-INITIATED] [2024-05-16 17:37:52.343] [rrc] [debug] Sending RRC Setup Request [2024-05-16 17:37:52.346] [rrc] [info] RRC connection established [2024-05-16 17:37:52.346] [rrc] [info] UE switches to state [RRC-CONNECTED] [2024-05-16 17:37:52.346] [nas] [info] UE switches to state [CM-CONNECTED] [2024-05-16 17:37:52.378] [nas] [debug] Authentication Request received [2024-05-16 17:37:52.389] [nas] [debug] Security Mode Command received [2024-05-16 17:37:52.389] [nas] [debug] Selected integrity[2] ciphering[0] [2024-05-16 17:37:52.440] [nas] [debug] Registration accept received [2024-05-16 17:37:52.440] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE] [2024-05-16 17:37:52.440] [nas] [debug] Sending Registration Complete [2024-05-16 17:37:52.440] [nas] [info] Initial Registration is successful [2024-05-16 17:37:52.440] [nas] [debug] Sending PDU Session Establishment Request [2024-05-16 17:37:52.463] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 17:37:52.709] [nas] [debug] PDU Session Establishment Accept received [2024-05-16 17:37:52.709] [nas] [info] PDU Session establishment is successful PSI[1] [2024-05-16 17:37:52.725] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.60.0.1] is up. ### at another console sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash root@1d4b478b1c91:/ueransim# ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: ip_vti0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 3: uesimtun0: mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet 10.60.0.1/32 scope global uesimtun0 valid_lft forever preferred_lft forever 689: eth0@if690: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:0a:64:c8:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.100.200.16/24 brd 10.100.200.255 scope global eth0 valid_lft forever preferred_lft forever root@1d4b478b1c91:/ueransim# root@1d4b478b1c91:/ueransim# ping -I uesimtun0 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 10.60.0.1 uesimtun0: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.19 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.58 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.68 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=55 time=1.66 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=55 time=1.59 ms ^C --- 1.1.1.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 1.584/1.738/2.190/0.228 ms root@1d4b478b1c91:/ueransim# ``` eupf logs: `sudo docker-compose logs edgecom-upf |less` ```ruby eupf | 2024/05/16 17:37:52 Handling PFCP message from 10.100.200.15:8805 eupf | 2024/05/16 17:37:52 Got Session Modification Request from: 10.100.200.15. eupf | 2024/05/16 17:37:52 Finding association for 10.100.200.15 eupf | 2024/05/16 17:37:52 Finding session 2 eupf | 2024/05/16 17:37:52 eupf | Session Modification Request: eupf | UpdatePDR ID: 2 eupf | FAR ID: 2 eupf | Source Interface: 1 eupf | UE IPv4 Address: 10.60.0.1 eupf | UpdateFAR ID: 2 eupf | Apply Action: [2] eupf | Update forwarding Parameters: eupf | Network Instance: internet eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address: PortNumber:0 CTag:0 STag:0} eupf | 2024/05/16 17:37:52 Updating FAR info: 2, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMark ing:0} GlobalId:1} eupf | 2024/05/16 17:37:52 EBPF: Update FAR: internalId=1, farInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 Transport LevelMarking:0} eupf | 2024/05/16 17:37:52 Both F-TEID IE and UE IP Address IE are missing eupf | 2024/05/16 17:37:57 Sent Heartbeat Request to: 10.100.200.15 eupf | 2024/05/16 17:37:57 Received 16 bytes from 10.100.200.15:8805 ```

Free5GC with UpLink Classifier config throught three eUPFs

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d

```ruby sergo@edgecom:~/edgecom24/free5gc-compose$ git checkout ulcl-n9upf-experimetns Branch 'ulcl-n9upf-experimetns' set up to track remote branch 'ulcl-n9upf-experimetns' from 'origin'. Switched to a new branch 'ulcl-n9upf-experimetns' docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose ps NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS amf free5gc/amf:v3.3.0 "./amf -c ./config/a_" free5gc-amf 2 minutes ago Up 2 minutes 8000/tcp ausf free5gc/ausf:v3.3.0 "./ausf -c ./config/_" free5gc-ausf 2 minutes ago Up 2 minutes 8000/tcp grafana grafana/grafana "/run.sh" grafana 2 minutes ago Up 2 minutes 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp i-eupf ghcr.io/edgecomllc/eupf:0.5.0 "sh /app/bin/entrypo_" edgecom-i-upf 2 minutes ago Up 2 minutes 0.0.0.0:8082->8080/tcp, :::8082->8080/tcp, 0.0.0.0:9092->9090/tcp, :::9092->9090/tcp mongodb mongo "docker-entrypoint.s_" db 2 minutes ago Up 2 minutes 27017/tcp n3iwf free5gc/n3iwf:v3.3.0 "sh -c './n3iwf-ipse_" free5gc-n3iwf 2 minutes ago Up 2 minutes nat free5gc-compose-edgecom-nat "sh /app/bin/busy-po_" edgecom-nat 2 minutes ago Up 2 minutes nrf free5gc/nrf:v3.3.0 "./nrf -c ./config/n_" free5gc-nrf 2 minutes ago Up 2 minutes 8000/tcp nssf free5gc/nssf:v3.3.0 "./nssf -c ./config/_" free5gc-nssf 2 minutes ago Up 2 minutes 8000/tcp pcf free5gc/pcf:v3.3.0 "./pcf -c ./config/p_" free5gc-pcf 2 minutes ago Up 2 minutes 8000/tcp prometheus prom/prometheus "/bin/prometheus --c_" prometheus 2 minutes ago Up 2 minutes 0.0.0.0:9091->9090/tcp, :::9091->9090/tcp psa-eupf ghcr.io/edgecomllc/eupf:0.5.0 "sh /app/bin/entrypo_" edgecom-psa-upf 2 minutes ago Up 2 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp psa-eupf-2 ghcr.io/edgecomllc/eupf:0.5.0 "sh /app/bin/entrypo_" edgecom-psa-upf-2 2 minutes ago Up 2 minutes smf free5gc/smf:v3.3.0 "./smf -c ./config/s_" free5gc-smf 2 minutes ago Up 2 minutes 8000/tcp udm free5gc/udm:v3.3.0 "./udm -c ./config/u_" free5gc-udm 2 minutes ago Up 2 minutes 8000/tcp udr free5gc/udr:v3.3.0 "./udr -c ./config/u_" free5gc-udr 2 minutes ago Up 2 minutes 8000/tcp ueransim free5gc/ueransim:v3.3.0 "./nr-gnb -c ./confi_" ueransim 2 minutes ago Up 2 minutes webui free5gc/webui:v3.3.0 "./webui -c ./config_" free5gc-webui 2 minutes ago Up 2 minutes 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp sergo@edgecom:~/edgecom24/free5gc-compose$ sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash root@aaad991d992b:/ueransim# ./nr-ue -c config/uecfg.yaml & [1] 24 root@aaad991d992b:/ueransim# UERANSIM v3.2.6 [2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH] [2024-05-16 18:01:03.356] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage [2024-05-16 18:01:03.356] [nas] [info] Selected plmn[208/93] [2024-05-16 18:01:03.356] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE] [2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PS] [2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE] [2024-05-16 18:01:03.356] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE] [2024-05-16 18:01:03.357] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 18:01:03.357] [nas] [debug] Sending Initial Registration [2024-05-16 18:01:03.357] [nas] [info] UE switches to state [MM-REGISTER-INITIATED] [2024-05-16 18:01:03.357] [rrc] [debug] Sending RRC Setup Request [2024-05-16 18:01:03.357] [rrc] [info] RRC connection established [2024-05-16 18:01:03.357] [rrc] [info] UE switches to state [RRC-CONNECTED] [2024-05-16 18:01:03.357] [nas] [info] UE switches to state [CM-CONNECTED] [2024-05-16 18:01:03.387] [nas] [debug] Authentication Request received [2024-05-16 18:01:03.398] [nas] [debug] Security Mode Command received [2024-05-16 18:01:03.398] [nas] [debug] Selected integrity[2] ciphering[0] [2024-05-16 18:01:03.451] [nas] [debug] Registration accept received [2024-05-16 18:01:03.451] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE] [2024-05-16 18:01:03.451] [nas] [debug] Sending Registration Complete [2024-05-16 18:01:03.451] [nas] [info] Initial Registration is successful [2024-05-16 18:01:03.451] [nas] [debug] Sending PDU Session Establishment Request [2024-05-16 18:01:03.451] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 18:01:03.720] [nas] [debug] PDU Session Establishment Accept received [2024-05-16 18:01:03.720] [nas] [info] PDU Session establishment is successful PSI[1] [2024-05-16 18:01:03.735] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.62.0.1] is up. root@aaad991d992b:/ueransim# ping -I uesimtun0 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 10.62.0.1 uesimtun0: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.01 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.61 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.66 ms ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.608/1.759/2.007/0.176 ms root@aaad991d992b:/ueransim# ping -I uesimtun0 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 10.62.0.1 uesimtun0: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=16.9 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=17.0 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=17.0 ms ^C --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 16.873/16.951/17.002/0.056 ms ``` ### Logs `sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-i-upf |less -R` look for `8.8.8.8` ```ruby i-eupf | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17. i-eupf | i-eupf | i-eupf | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17 i-eupf | 2024/05/16 18:01:03 INF Finding session 2 i-eupf | 2024/05/16 18:01:03 INF i-eupf | Session Modification Request: i-eupf | CreatePDR ID: 3 i-eupf | Outer Header Removal: 0 i-eupf | FAR ID: 3 i-eupf | QER ID: 1 i-eupf | Source Interface: 0 i-eupf | TEID: 1 i-eupf | Ipv4: 10.100.200.14 i-eupf | Ipv6: i-eupf | UE IPv4 Address: 10.62.0.1 i-eupf | SDF Filter: permit out ip from 10.62.0.1 to 8.8.8.8/32 i-eupf | CreatePDR ID: 4 i-eupf | Outer Header Removal: 0 i-eupf | FAR ID: 4 i-eupf | QER ID: 1 i-eupf | Source Interface: 1 i-eupf | TEID: 4 i-eupf | Ipv4: 10.100.200.14 i-eupf | Ipv6: i-eupf | CreateFAR ID: 3 i-eupf | Apply Action: [2] i-eupf | Forwarding Parameters: i-eupf | Network Instance: internet i-eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address: PortNumber:0 CTag:0 STag:0} i-eupf | CreateFAR ID: 4 i-eupf | Apply Action: [2] i-eupf | Forwarding Parameters: i-eupf | Network Instance: internet i-eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.19 IPv6Address: PortNumber:0 CTag:0 STag:0} i-eupf | i-eupf | i-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=2, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 4, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=3, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1 SdfFilter:{Protocol:1 SrcAddress:{Type:1 Ip:10.62.0.1 Mask:ffffffff} SrcPortRange:{LowerBound:0 UpperBound:65535} DstAddress:{Type:1 Ip:8.8.8.8 Mask:ffffffff} DstPortRange:{LowerBound:0 UpperBound:65535}}} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=4, pdrInfo={OuterHeaderRemoval:0 FarId:3 QerId:1 SdfFilter:} i-eupf | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805 ``` `sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf |less -R` ```ruby psa-eupf | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17. psa-eupf | 2024/05/16 18:01:03 INF psa-eupf | Session Establishment Request: psa-eupf | CreatePDR ID: 1 psa-eupf | Outer Header Removal: 0 psa-eupf | FAR ID: 1 psa-eupf | QER ID: 2 psa-eupf | QER ID: 1 psa-eupf | Source Interface: 0 psa-eupf | TEID: 1 psa-eupf | Ipv4: 10.100.200.15 psa-eupf | Ipv6: psa-eupf | UE IPv4 Address: 10.62.0.1 psa-eupf | CreatePDR ID: 2 psa-eupf | FAR ID: 2 psa-eupf | QER ID: 2 psa-eupf | QER ID: 1 psa-eupf | Source Interface: 1 psa-eupf | UE IPv4 Address: 10.62.0.1 psa-eupf | CreateFAR ID: 1 psa-eupf | Apply Action: [2] psa-eupf | Forwarding Parameters: psa-eupf | Network Instance: internet psa-eupf | CreateFAR ID: 2 psa-eupf | Apply Action: [2] psa-eupf | Forwarding Parameters: psa-eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.100.200.14 IPv6Address: PortNumber:0 CTag:0 STag:0} psa-eupf | CreateQER ID: 2 psa-eupf | Gate Status DL: 0 psa-eupf | Gate Status UL: 0 psa-eupf | QFI: 1 psa-eupf | CreateQER ID: 1 psa-eupf | Gate Status DL: 0 psa-eupf | Gate Status UL: 0 psa-eupf | Max Bitrate DL: 1000000 psa-eupf | Max Bitrate UL: 2000000 psa-eupf | QFI: 0 psa-eupf | psa-eupf | psa-eupf | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation psa-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF Saving QER info to session: 2, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:} psa-eupf | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted. psa-eupf | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805 ``` `sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf-2 |less -R` ```ruby psa-eupf-2 | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17. psa-eupf-2 | 2024/05/16 18:01:03 INF psa-eupf-2 | Session Establishment Request: psa-eupf-2 | CreatePDR ID: 1 psa-eupf-2 | Outer Header Removal: 0 psa-eupf-2 | FAR ID: 1 psa-eupf-2 | QER ID: 1 psa-eupf-2 | Source Interface: 0 psa-eupf-2 | TEID: 1 psa-eupf-2 | Ipv4: 10.100.200.16 psa-eupf-2 | Ipv6: psa-eupf-2 | UE IPv4 Address: 10.62.0.1 psa-eupf-2 | CreateFAR ID: 1 psa-eupf-2 | Apply Action: [2] psa-eupf-2 | Forwarding Parameters: psa-eupf-2 | Network Instance: internet psa-eupf-2 | CreateQER ID: 1 psa-eupf-2 | Gate Status DL: 0 psa-eupf-2 | Gate Status UL: 0 psa-eupf-2 | Max Bitrate DL: 1000000 psa-eupf-2 | Max Bitrate UL: 2000000 psa-eupf-2 | QFI: 0 psa-eupf-2 | psa-eupf-2 | psa-eupf-2 | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation psa-eupf-2 | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:} psa-eupf-2 | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted. psa-eupf-2 | 2024/05/16 18:01:03 INF Received 137 bytes from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17. psa-eupf-2 | psa-eupf-2 | psa-eupf-2 | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17 psa-eupf-2 | 2024/05/16 18:01:03 INF Finding session 2 psa-eupf-2 | 2024/05/16 18:01:03 INF psa-eupf-2 | Session Modification Request: psa-eupf-2 | CreatePDR ID: 2 psa-eupf-2 | FAR ID: 2 psa-eupf-2 | QER ID: 1 psa-eupf-2 | Source Interface: 1 psa-eupf-2 | UE IPv4 Address: 10.62.0.1 psa-eupf-2 | CreateFAR ID: 2 psa-eupf-2 | Apply Action: [2] psa-eupf-2 | Forwarding Parameters: psa-eupf-2 | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:4 IPv4Address:10.100.200.14 IPv6Address: PortNumber:0 CTag:0 STag:0} psa-eupf-2 | psa-eupf-2 | psa-eupf-2 | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:} psa-eupf-2 | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:04 INF Handling PFCP message from 10.100.200.17:8805 ```

PapaySail commented 1 month ago

✔ Docker-compose Free5GC

docker-compose up -d

```ruby sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose up -d [+] Running 21/21 _ Network free5gc-compose_privnet_n6 Created 0.1s _ Network free5gc-compose_default Created 0.1s _ Network free5gc-compose_privnet Created 0.1s _ Container prometheus Started 1.3s _ Container upf Started 1.6s _ Container grafana Started 1.7s _ Container eupf Started 2.6s _ Container nat Started 2.4s _ Container mongodb Started 0.9s _ Container ui Started 1.3s _ Container webui Started 2.1s _ Container nrf Started 2.4s _ Container nssf Started 3.9s _ Container udm Started 3.9s _ Container ausf Started 3.9s _ Container amf Started 3.4s _ Container pcf Started 4.1s _ Container smf Started 4.0s _ Container udr Started 3.9s _ Container ueransim Started 4.2s _ Container n3iwf Started 4.5s sergo@edgecom:~/edgecom24/free5gc-compose$ sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash root@1d4b478b1c91:/ueransim# ./nr-ue -c config/uecfg.yaml UERANSIM v3.2.6 [2024-05-16 17:37:49.842] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH] [2024-05-16 17:37:49.843] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage [2024-05-16 17:37:49.846] [nas] [info] Selected plmn[208/93] [2024-05-16 17:37:52.343] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE] [2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/PS] [2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE] [2024-05-16 17:37:52.343] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE] [2024-05-16 17:37:52.343] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 17:37:52.343] [nas] [debug] Sending Initial Registration [2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-REGISTER-INITIATED] [2024-05-16 17:37:52.343] [rrc] [debug] Sending RRC Setup Request [2024-05-16 17:37:52.346] [rrc] [info] RRC connection established [2024-05-16 17:37:52.346] [rrc] [info] UE switches to state [RRC-CONNECTED] [2024-05-16 17:37:52.346] [nas] [info] UE switches to state [CM-CONNECTED] [2024-05-16 17:37:52.378] [nas] [debug] Authentication Request received [2024-05-16 17:37:52.389] [nas] [debug] Security Mode Command received [2024-05-16 17:37:52.389] [nas] [debug] Selected integrity[2] ciphering[0] [2024-05-16 17:37:52.440] [nas] [debug] Registration accept received [2024-05-16 17:37:52.440] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE] [2024-05-16 17:37:52.440] [nas] [debug] Sending Registration Complete [2024-05-16 17:37:52.440] [nas] [info] Initial Registration is successful [2024-05-16 17:37:52.440] [nas] [debug] Sending PDU Session Establishment Request [2024-05-16 17:37:52.463] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 17:37:52.709] [nas] [debug] PDU Session Establishment Accept received [2024-05-16 17:37:52.709] [nas] [info] PDU Session establishment is successful PSI[1] [2024-05-16 17:37:52.725] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.60.0.1] is up. ### at another console sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash root@1d4b478b1c91:/ueransim# ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: ip_vti0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 3: uesimtun0: mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet 10.60.0.1/32 scope global uesimtun0 valid_lft forever preferred_lft forever 689: eth0@if690: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:0a:64:c8:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.100.200.16/24 brd 10.100.200.255 scope global eth0 valid_lft forever preferred_lft forever root@1d4b478b1c91:/ueransim# root@1d4b478b1c91:/ueransim# ping -I uesimtun0 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 10.60.0.1 uesimtun0: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.19 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.58 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.68 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=55 time=1.66 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=55 time=1.59 ms ^C --- 1.1.1.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 1.584/1.738/2.190/0.228 ms root@1d4b478b1c91:/ueransim# ``` eupf logs: `sudo docker-compose logs edgecom-upf |less` ```ruby eupf | 2024/05/16 17:37:52 Handling PFCP message from 10.100.200.15:8805 eupf | 2024/05/16 17:37:52 Got Session Modification Request from: 10.100.200.15. eupf | 2024/05/16 17:37:52 Finding association for 10.100.200.15 eupf | 2024/05/16 17:37:52 Finding session 2 eupf | 2024/05/16 17:37:52 eupf | Session Modification Request: eupf | UpdatePDR ID: 2 eupf | FAR ID: 2 eupf | Source Interface: 1 eupf | UE IPv4 Address: 10.60.0.1 eupf | UpdateFAR ID: 2 eupf | Apply Action: [2] eupf | Update forwarding Parameters: eupf | Network Instance: internet eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address: PortNumber:0 CTag:0 STag:0} eupf | 2024/05/16 17:37:52 Updating FAR info: 2, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMark ing:0} GlobalId:1} eupf | 2024/05/16 17:37:52 EBPF: Update FAR: internalId=1, farInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 Transport LevelMarking:0} eupf | 2024/05/16 17:37:52 Both F-TEID IE and UE IP Address IE are missing eupf | 2024/05/16 17:37:57 Sent Heartbeat Request to: 10.100.200.15 eupf | 2024/05/16 17:37:57 Received 16 bytes from 10.100.200.15:8805 ```

Free5GC with UpLink Classifier config throught three eUPFs

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d

```ruby sergo@edgecom:~/edgecom24/free5gc-compose$ git checkout ulcl-n9upf-experimetns Branch 'ulcl-n9upf-experimetns' set up to track remote branch 'ulcl-n9upf-experimetns' from 'origin'. Switched to a new branch 'ulcl-n9upf-experimetns' docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose ps NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS amf free5gc/amf:v3.3.0 "./amf -c ./config/a_" free5gc-amf 2 minutes ago Up 2 minutes 8000/tcp ausf free5gc/ausf:v3.3.0 "./ausf -c ./config/_" free5gc-ausf 2 minutes ago Up 2 minutes 8000/tcp grafana grafana/grafana "/run.sh" grafana 2 minutes ago Up 2 minutes 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp i-eupf ghcr.io/edgecomllc/eupf:0.5.0 "sh /app/bin/entrypo_" edgecom-i-upf 2 minutes ago Up 2 minutes 0.0.0.0:8082->8080/tcp, :::8082->8080/tcp, 0.0.0.0:9092->9090/tcp, :::9092->9090/tcp mongodb mongo "docker-entrypoint.s_" db 2 minutes ago Up 2 minutes 27017/tcp n3iwf free5gc/n3iwf:v3.3.0 "sh -c './n3iwf-ipse_" free5gc-n3iwf 2 minutes ago Up 2 minutes nat free5gc-compose-edgecom-nat "sh /app/bin/busy-po_" edgecom-nat 2 minutes ago Up 2 minutes nrf free5gc/nrf:v3.3.0 "./nrf -c ./config/n_" free5gc-nrf 2 minutes ago Up 2 minutes 8000/tcp nssf free5gc/nssf:v3.3.0 "./nssf -c ./config/_" free5gc-nssf 2 minutes ago Up 2 minutes 8000/tcp pcf free5gc/pcf:v3.3.0 "./pcf -c ./config/p_" free5gc-pcf 2 minutes ago Up 2 minutes 8000/tcp prometheus prom/prometheus "/bin/prometheus --c_" prometheus 2 minutes ago Up 2 minutes 0.0.0.0:9091->9090/tcp, :::9091->9090/tcp psa-eupf ghcr.io/edgecomllc/eupf:0.5.0 "sh /app/bin/entrypo_" edgecom-psa-upf 2 minutes ago Up 2 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp psa-eupf-2 ghcr.io/edgecomllc/eupf:0.5.0 "sh /app/bin/entrypo_" edgecom-psa-upf-2 2 minutes ago Up 2 minutes smf free5gc/smf:v3.3.0 "./smf -c ./config/s_" free5gc-smf 2 minutes ago Up 2 minutes 8000/tcp udm free5gc/udm:v3.3.0 "./udm -c ./config/u_" free5gc-udm 2 minutes ago Up 2 minutes 8000/tcp udr free5gc/udr:v3.3.0 "./udr -c ./config/u_" free5gc-udr 2 minutes ago Up 2 minutes 8000/tcp ueransim free5gc/ueransim:v3.3.0 "./nr-gnb -c ./confi_" ueransim 2 minutes ago Up 2 minutes webui free5gc/webui:v3.3.0 "./webui -c ./config_" free5gc-webui 2 minutes ago Up 2 minutes 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp sergo@edgecom:~/edgecom24/free5gc-compose$ sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash root@aaad991d992b:/ueransim# ./nr-ue -c config/uecfg.yaml & [1] 24 root@aaad991d992b:/ueransim# UERANSIM v3.2.6 [2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH] [2024-05-16 18:01:03.356] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage [2024-05-16 18:01:03.356] [nas] [info] Selected plmn[208/93] [2024-05-16 18:01:03.356] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE] [2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PS] [2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE] [2024-05-16 18:01:03.356] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE] [2024-05-16 18:01:03.357] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 18:01:03.357] [nas] [debug] Sending Initial Registration [2024-05-16 18:01:03.357] [nas] [info] UE switches to state [MM-REGISTER-INITIATED] [2024-05-16 18:01:03.357] [rrc] [debug] Sending RRC Setup Request [2024-05-16 18:01:03.357] [rrc] [info] RRC connection established [2024-05-16 18:01:03.357] [rrc] [info] UE switches to state [RRC-CONNECTED] [2024-05-16 18:01:03.357] [nas] [info] UE switches to state [CM-CONNECTED] [2024-05-16 18:01:03.387] [nas] [debug] Authentication Request received [2024-05-16 18:01:03.398] [nas] [debug] Security Mode Command received [2024-05-16 18:01:03.398] [nas] [debug] Selected integrity[2] ciphering[0] [2024-05-16 18:01:03.451] [nas] [debug] Registration accept received [2024-05-16 18:01:03.451] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE] [2024-05-16 18:01:03.451] [nas] [debug] Sending Registration Complete [2024-05-16 18:01:03.451] [nas] [info] Initial Registration is successful [2024-05-16 18:01:03.451] [nas] [debug] Sending PDU Session Establishment Request [2024-05-16 18:01:03.451] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig] [2024-05-16 18:01:03.720] [nas] [debug] PDU Session Establishment Accept received [2024-05-16 18:01:03.720] [nas] [info] PDU Session establishment is successful PSI[1] [2024-05-16 18:01:03.735] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.62.0.1] is up. root@aaad991d992b:/ueransim# ping -I uesimtun0 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 10.62.0.1 uesimtun0: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.01 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.61 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.66 ms ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.608/1.759/2.007/0.176 ms root@aaad991d992b:/ueransim# ping -I uesimtun0 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 10.62.0.1 uesimtun0: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=16.9 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=17.0 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=17.0 ms ^C --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 16.873/16.951/17.002/0.056 ms ``` ### Logs `sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-i-upf |less -R` look for `8.8.8.8` ```ruby i-eupf | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17. i-eupf | i-eupf | i-eupf | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17 i-eupf | 2024/05/16 18:01:03 INF Finding session 2 i-eupf | 2024/05/16 18:01:03 INF i-eupf | Session Modification Request: i-eupf | CreatePDR ID: 3 i-eupf | Outer Header Removal: 0 i-eupf | FAR ID: 3 i-eupf | QER ID: 1 i-eupf | Source Interface: 0 i-eupf | TEID: 1 i-eupf | Ipv4: 10.100.200.14 i-eupf | Ipv6: i-eupf | UE IPv4 Address: 10.62.0.1 i-eupf | SDF Filter: permit out ip from 10.62.0.1 to 8.8.8.8/32 i-eupf | CreatePDR ID: 4 i-eupf | Outer Header Removal: 0 i-eupf | FAR ID: 4 i-eupf | QER ID: 1 i-eupf | Source Interface: 1 i-eupf | TEID: 4 i-eupf | Ipv4: 10.100.200.14 i-eupf | Ipv6: i-eupf | CreateFAR ID: 3 i-eupf | Apply Action: [2] i-eupf | Forwarding Parameters: i-eupf | Network Instance: internet i-eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address: PortNumber:0 CTag:0 STag:0} i-eupf | CreateFAR ID: 4 i-eupf | Apply Action: [2] i-eupf | Forwarding Parameters: i-eupf | Network Instance: internet i-eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.19 IPv6Address: PortNumber:0 CTag:0 STag:0} i-eupf | i-eupf | i-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=2, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 4, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=3, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1 SdfFilter:{Protocol:1 SrcAddress:{Type:1 Ip:10.62.0.1 Mask:ffffffff} SrcPortRange:{LowerBound:0 UpperBound:65535} DstAddress:{Type:1 Ip:8.8.8.8 Mask:ffffffff} DstPortRange:{LowerBound:0 UpperBound:65535}}} i-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=4, pdrInfo={OuterHeaderRemoval:0 FarId:3 QerId:1 SdfFilter:} i-eupf | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805 ``` `sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf |less -R` ```ruby psa-eupf | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17. psa-eupf | 2024/05/16 18:01:03 INF psa-eupf | Session Establishment Request: psa-eupf | CreatePDR ID: 1 psa-eupf | Outer Header Removal: 0 psa-eupf | FAR ID: 1 psa-eupf | QER ID: 2 psa-eupf | QER ID: 1 psa-eupf | Source Interface: 0 psa-eupf | TEID: 1 psa-eupf | Ipv4: 10.100.200.15 psa-eupf | Ipv6: psa-eupf | UE IPv4 Address: 10.62.0.1 psa-eupf | CreatePDR ID: 2 psa-eupf | FAR ID: 2 psa-eupf | QER ID: 2 psa-eupf | QER ID: 1 psa-eupf | Source Interface: 1 psa-eupf | UE IPv4 Address: 10.62.0.1 psa-eupf | CreateFAR ID: 1 psa-eupf | Apply Action: [2] psa-eupf | Forwarding Parameters: psa-eupf | Network Instance: internet psa-eupf | CreateFAR ID: 2 psa-eupf | Apply Action: [2] psa-eupf | Forwarding Parameters: psa-eupf | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.100.200.14 IPv6Address: PortNumber:0 CTag:0 STag:0} psa-eupf | CreateQER ID: 2 psa-eupf | Gate Status DL: 0 psa-eupf | Gate Status UL: 0 psa-eupf | QFI: 1 psa-eupf | CreateQER ID: 1 psa-eupf | Gate Status DL: 0 psa-eupf | Gate Status UL: 0 psa-eupf | Max Bitrate DL: 1000000 psa-eupf | Max Bitrate UL: 2000000 psa-eupf | QFI: 0 psa-eupf | psa-eupf | psa-eupf | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation psa-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0} psa-eupf | 2024/05/16 18:01:03 INF Saving QER info to session: 2, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:} psa-eupf | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:} psa-eupf | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted. psa-eupf | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805 ``` `sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf-2 |less -R` ```ruby psa-eupf-2 | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17. psa-eupf-2 | 2024/05/16 18:01:03 INF psa-eupf-2 | Session Establishment Request: psa-eupf-2 | CreatePDR ID: 1 psa-eupf-2 | Outer Header Removal: 0 psa-eupf-2 | FAR ID: 1 psa-eupf-2 | QER ID: 1 psa-eupf-2 | Source Interface: 0 psa-eupf-2 | TEID: 1 psa-eupf-2 | Ipv4: 10.100.200.16 psa-eupf-2 | Ipv6: psa-eupf-2 | UE IPv4 Address: 10.62.0.1 psa-eupf-2 | CreateFAR ID: 1 psa-eupf-2 | Apply Action: [2] psa-eupf-2 | Forwarding Parameters: psa-eupf-2 | Network Instance: internet psa-eupf-2 | CreateQER ID: 1 psa-eupf-2 | Gate Status DL: 0 psa-eupf-2 | Gate Status UL: 0 psa-eupf-2 | Max Bitrate DL: 1000000 psa-eupf-2 | Max Bitrate UL: 2000000 psa-eupf-2 | QFI: 0 psa-eupf-2 | psa-eupf-2 | psa-eupf-2 | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation psa-eupf-2 | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:} psa-eupf-2 | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted. psa-eupf-2 | 2024/05/16 18:01:03 INF Received 137 bytes from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17. psa-eupf-2 | psa-eupf-2 | psa-eupf-2 | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17 psa-eupf-2 | 2024/05/16 18:01:03 INF Finding session 2 psa-eupf-2 | 2024/05/16 18:01:03 INF psa-eupf-2 | Session Modification Request: psa-eupf-2 | CreatePDR ID: 2 psa-eupf-2 | FAR ID: 2 psa-eupf-2 | QER ID: 1 psa-eupf-2 | Source Interface: 1 psa-eupf-2 | UE IPv4 Address: 10.62.0.1 psa-eupf-2 | CreateFAR ID: 2 psa-eupf-2 | Apply Action: [2] psa-eupf-2 | Forwarding Parameters: psa-eupf-2 | Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:4 IPv4Address:10.100.200.14 IPv6Address: PortNumber:0 CTag:0 STag:0} psa-eupf-2 | psa-eupf-2 | psa-eupf-2 | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0} psa-eupf-2 | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:} psa-eupf-2 | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805 psa-eupf-2 | 2024/05/16 18:01:04 INF Handling PFCP message from 10.100.200.17:8805 ```

PapaySail commented 1 month ago

OAI Kubernetes

The deployment still not ready. 🤷‍♂️ Nothing to check.

  1. install eupf ⚠️ not finished, work in progress make upf

  2. configure calico BGP settings. Here, we configure Calico BGP peer, create Calico IP Pool (for NAT) and configure Felix for save external routes (recevied by BGP from eUPF BIRD) ⚠️ not finished, work in progress make calico

PapaySail commented 1 month ago

✔ Docker-compose OAI 5G SA mode with L2 nFAPI simulator

  1. Start containers in the following order:

    service "oai-spgwu" has neither an image nor a build context specified: invalid compose project

    ```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ cp -ar ~/edgecom24/eupf/docs/deployments/oai-nfapi-sim-compose/* ./ sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ls docker-compose.override.yml docker-compose.yaml eupf nat README.md sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose up -d mysql oai-nrf oai-amf oai-smf edgecom-nat edgecom-upf service "oai-spgwu" has neither an image nor a build context specified: invalid compose project ```

    ✔ service renamed to oai-upf :: update to docker-compose.override.yaml needed

    sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose up -d mysql oai-nrf oai-amf oai-smf edgecom-nat edgecom-upf
    service "prometheus" refers to undefined volume prom_data: invalid compose project

    ✔ update to docker-compose.override.yaml needed:

    volumes:
      prom_data:

    ✔ no such service: oai-nrf

    SMF exited: [smf_app] [error] Could not read network interface configuration

    • due to new config file
    • removed image: oaisoftwarealliance/oai-smf:v1.5.1
    • added
      networks:
        public_net:
          aliases:
            - oai-upf
docker pull oaisoftwarealliance/oai-gnb:develop

```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker pull oaisoftwarealliance/oai-gnb:develop develop: Pulling from oaisoftwarealliance/oai-gnb edaedc954fb5: Already exists 90e8c9a558e0: Already exists 544dfcb95356: Pull complete 5aaeb55fa076: Pull complete 943517f19fa8: Pull complete c5ce95fed1cd: Pull complete 650a9bbfdc1e: Pull complete 6c40c55a2a3e: Pull complete b116ff4d25cb: Pull complete e46081b32958: Pull complete 955f4243c199: Pull complete 4f4fb700ef54: Pull complete a07e06ffa612: Pull complete Digest: sha256:58dac7ea7322641c664ae58f1d6e4bbb05a5843d12c257913038fd085338840c Status: Downloaded newer image for oaisoftwarealliance/oai-gnb:develop docker.io/oaisoftwarealliance/oai-gnb:develop sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose up -d oai-gnb ```

❌ UE not connected

oaitun_ue1 interface without ip address

```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun 769: oaitun_ue1: mtu 1500 qdisc noop state DOWN group default qlen 500 ```

Troubleshooting

SMF logs regarding UPF: `Did not find a UPF config for UPF-associated association 192.168.71.134. Use empty UPF profile` ```ruby l2sim-oai-smf | [2024-05-22 21:09:36.870] [smf_app] [debug] Reverse Resolving Try #0 l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Resolve IP Addr 192.168.71.134, FQDN eupf.l2sim-oai-public-net l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Did not find a UPF config for UPF-associated association 192.168.71.134. Use empty UPF profile with one assumed N3 and N6 interface. l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Could not find other edges for UPF, just add UPF as a node l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Successfully added UPF node: 192.168.71.134, (2252843200) l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Successfully added UPF graph edge for 192.168.71.134: N3() l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Successfully added UPF graph edge for 192.168.71.134: N6() l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] UPF graph l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] * 192.168.71.134 --> N3(), N6(), l2sim-oai-smf | l2sim-oai-smf | l2sim-oai-smf | [2024-05-22 21:09:36.871] [smf_app] [debug] Added PFCP assocation with UPF config: l2sim-oai-smf | + 192.168.71.134 l2sim-oai-smf | + host...................................: 192.168.71.134 l2sim-oai-smf | + port...................................: 8805 ``` ⚠ The original deployment itself is faulty: ```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose ps -a NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS l2sim-mysql mysql:8.0 "docker-entrypoint.s_" mysql 40 minutes ago Up 40 minutes (healthy) 3306/tcp, 33060/tcp l2sim-oai-amf oaisoftwarealliance/oai-amf:v2.0.0 "/openair-amf/bin/oa_" oai-amf 40 minutes ago Up 40 minutes (healthy) 80/tcp, 9090/tcp, 38412/sctp l2sim-oai-ext-dn oaisoftwarealliance/trf-gen-cn5g:focal "/bin/bash -c ' ipta_" oai-ext-dn 40 minutes ago Up 40 minutes (healthy) l2sim-oai-gnb oaisoftwarealliance/oai-gnb:develop "/tini -v -- /opt/oa_" oai-gnb 39 minutes ago Up 39 minutes (healthy) l2sim-oai-nr-ue0 oaisoftwarealliance/oai-nr-ue:develop "/tini -v -- /opt/oa_" oai-nr-ue0 About a minute ago Up About a minute (healthy) l2sim-oai-smf oaisoftwarealliance/oai-smf:v2.0.0 "/openair-smf/bin/oa_" oai-smf 40 minutes ago Up 40 minutes (healthy) 80/tcp, 8080/tcp, 8805/udp l2sim-oai-upf oaisoftwarealliance/oai-upf:v2.0.0 "/openair-upf/bin/oa_" oai-upf 40 minutes ago Up 40 minutes (healthy) 2152/udp, 8805/udp l2sim-proxy oai-lte-multi-ue-proxy:latest "/oai-lte-multi-ue-p_" proxy About a minute ago Up About a minute (healthy) sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun 825: oaitun_ue1: mtu 1500 qdisc noop state DOWN group default qlen 500 sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose logs oai-amf |less sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose logs -t oai-nr-ue0 |less sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun 825: oaitun_ue1: mtu 1500 qdisc noop state DOWN group default qlen 500 ``` Let's get back in history an start from commit [chore(ci): updating 5G rfsimulator scenarios to release v2.0](https://gitlab.eurecom.fr/oai/openairinterface5g/-/commit/f1d080d31515d3cc9a31a41fd548753ccba4f09f) It is parent to commit with changes to docker-compose and config files. ```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ git checkout f1d080d31515d3cc9a31a41fd548753ccba4f09f Note: switching to 'f1d080d31515d3cc9a31a41fd548753ccba4f09f'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at f1d080d315 chore(ci): updating 5G rfsimulator scenarios to release v2.0 sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ``` Trying to start it as is = unsuccessful, no IP addres assigned to `oaitun_ue1` Next, will try to use image tag 2023.w43 for oai-nr-ue and oai-gnb. OK! Yes, this is working images. ```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun 846: oaitun_ue1: mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500 inet 12.1.1.2/24 brd 12.1.1.255 scope global oaitun_ue1 sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose ps -a NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS l2sim-mysql mysql:8.0 "docker-entrypoint.s_" mysql 54 minutes ago Up 54 minutes (healthy) 3306/tcp, 33060/tcp l2sim-oai-amf oaisoftwarealliance/oai-amf:v1.5.0 "python3 /openair-am_" oai-amf 54 minutes ago Up 54 minutes (healthy) 80/tcp, 9090/tcp, 38412/sctp l2sim-oai-ext-dn oaisoftwarealliance/trf-gen-cn5g:focal "/bin/bash -c ' ipta_" oai-ext-dn 54 minutes ago Up 54 minutes (healthy) l2sim-oai-gnb oaisoftwarealliance/oai-gnb:2023.w43 "/opt/oai-gnb/bin/en_" oai-gnb 2 minutes ago Up 2 minutes (healthy) l2sim-oai-nr-ue0 oaisoftwarealliance/oai-nr-ue:2023.w43 "/opt/oai-nr-ue/bin/_" oai-nr-ue0 About a minute ago Up About a minute (healthy) l2sim-oai-nrf oaisoftwarealliance/oai-nrf:v1.5.0 "python3 /openair-nr_" oai-nrf 54 minutes ago Up 54 minutes (healthy) 80/tcp, 9090/tcp l2sim-oai-smf oaisoftwarealliance/oai-smf:v1.5.0 "python3 /openair-sm_" oai-smf 54 minutes ago Up 54 minutes (healthy) 80/tcp, 8080/tcp, 8805/udp l2sim-oai-spgwu oaisoftwarealliance/oai-spgwu-tiny:v1.5.0 "python3 /openair-sp_" oai-spgwu 54 minutes ago Up 54 minutes (healthy) 2152/udp, 8805/udp l2sim-proxy oai-lte-multi-ue-proxy:latest "/oai-lte-multi-ue-p_" proxy About a minute ago Up About a minute (healthy) sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=79.7 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=112 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=151 ms --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 79.722/114.461/151.339/29.276 ms ```

OK, with eUPF and oaisoftwarealliance/oai-gnb:2023.w43, oaisoftwarealliance/oai-nr-ue:2023.w43. Just no icmp replies.

Details & troubleshooting

```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep tun 865: oaitun_ue1: mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500 inet 12.1.1.2/24 brd 12.1.1.255 scope global oaitun_ue1 sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data. --- 1.1.1.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2040ms sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 172.17.0.1 PING 172.17.0.1 (172.17.0.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data. ^C --- 172.17.0.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2054ms sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 10.100.250.1 PING 10.100.250.1 (10.100.250.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data. --- 10.100.250.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2037ms sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose ps -a NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS eupf ghcr.io/edgecomllc/eupf:7086b71cb1dbc254db84c30e14fa4a68d3beb132 "sh /app/bin/entrypo_" edgecom-upf 5 minutes ago Up 5 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp, 0.0.0.0:8880->8080/tcp, :::8880->8080/tcp l2sim-mysql mysql:8.0 "docker-entrypoint.s_" mysql 5 minutes ago Up 5 minutes (healthy) 3306/tcp, 33060/tcp l2sim-oai-amf oaisoftwarealliance/oai-amf:v1.5.0 "python3 /openair-am_" oai-amf 5 minutes ago Up 5 minutes (healthy) 80/tcp, 9090/tcp, 38412/sctp l2sim-oai-gnb oaisoftwarealliance/oai-gnb:2023.w43 "/opt/oai-gnb/bin/en_" oai-gnb 5 minutes ago Up 5 minutes (healthy) l2sim-oai-nr-ue0 oaisoftwarealliance/oai-nr-ue:2023.w43 "/opt/oai-nr-ue/bin/_" oai-nr-ue0 4 minutes ago Up 4 minutes (healthy) l2sim-oai-nrf oaisoftwarealliance/oai-nrf:v1.5.0 "python3 /openair-nr_" oai-nrf 5 minutes ago Up 5 minutes (healthy) 80/tcp, 9090/tcp l2sim-oai-smf oaisoftwarealliance/oai-smf:v1.5.1 "python3 /openair-sm_" oai-smf 5 minutes ago Up 5 minutes (healthy) 80/tcp, 8080/tcp, 8805/udp l2sim-proxy oai-lte-multi-ue-proxy:latest "/oai-lte-multi-ue-p_" proxy 4 minutes ago Up 4 minutes (healthy) nat 5g_l2sim_tdd-edgecom-nat "sh /app/bin/busy-po_" edgecom-nat 5 minutes ago Up 5 minutes sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ``` Troubleshooting: ```ruby sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 1 172.17.0.1 PING 172.17.0.1 (172.17.0.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data. --- 172.17.0.1 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 1 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data. --- 1.1.1.1 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo tcpdump -i br-oainfapi-n6 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on br-oainfapi-n6, link-type EN10MB (Ethernet), snapshot length 262144 bytes 14:30:04.784634 IP 12.1.1.2 > 172.17.0.1: ICMP echo request, id 16, seq 1, length 64 14:30:04.784716 IP 10.100.250.2 > 172.17.0.1: ICMP echo request, id 16, seq 1, length 64 14:30:04.784745 IP 172.17.0.1 > 10.100.250.2: ICMP echo reply, id 16, seq 1, length 64 14:30:04.784754 IP 172.17.0.1 > 12.1.1.2: ICMP echo reply, id 16, seq 1, length 64 14:30:09.931117 ARP, Request who-has 10.100.250.2 tell 10.100.250.1, length 28 14:30:09.931144 ARP, Request who-has 10.100.250.1 tell 10.100.250.2, length 28 14:30:09.931182 ARP, Reply 10.100.250.1 is-at 02:42:10:2b:4a:d7 (oui Unknown), length 28 14:30:09.931168 ARP, Reply 10.100.250.2 is-at 02:42:0a:64:fa:02 (oui Unknown), length 28 14:30:30.008260 IP 12.1.1.2 > one.one.one.one: ICMP echo request, id 17, seq 1, length 64 14:30:30.008351 IP 10.100.250.2 > one.one.one.one: ICMP echo request, id 17, seq 1, length 64 14:30:30.009622 IP one.one.one.one > 10.100.250.2: ICMP echo reply, id 17, seq 1, length 64 14:30:30.009640 IP one.one.one.one > 12.1.1.2: ICMP echo reply, id 17, seq 1, length 64 ``` ``` sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose exec edgecom-nat sh /app # ip r default via 10.100.250.1 dev eth0 10.100.250.0/24 dev eth0 scope link src 10.100.250.2 /app # ip route add 12.1.1.0/24 via `nslookup upfn6.free5gc.org | awk '/^Address: / { print $2 }'` /app # ip r default via 10.100.250.1 dev eth0 10.100.250.0/24 dev eth0 scope link src 10.100.250.2 12.1.1.0/24 via 10.100.250.3 dev eth0 /app # exit sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 1 1.1.1.1 PING 1.1.1.1 (1.1.1.1) from 12.1.1.4 oaitun_ue1: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=82.7 ms --- 1.1.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 82.736/82.736/82.736/0.000 ms ```

Solution will be to remove dev eth1 from nat's script command ip route add. ✔ fixed in #552