search

edgecomllc / eupf

5G User Plane Function (UPF) based on eBPF
Apache License 2.0
103 stars 20 forks source link

Test eUPF with OMEC gNBSim #474

Open pirog-spb opened 11 months ago

pirog-spb commented 11 months ago

I'm as a eUPF developer want to be sure that eUPF can process a lot of UEs from a lot of gNBs in order to provide solution for commercial use.

Acceptance tests:

PapaySail commented 11 months ago

Explore: the official SD-Core documentation

Helm Chart: gerrit repository https://gerrit.opencord.org/plugins/gitiles/sdcore-helm-charts/+/refs/heads/master/5g-ran-sim/ RAN SIM Helm Chart: 5g-ran-sim repository https://charts.aetherproject.org/

The gNBSim tool simulates gNodeB and UE The configuration has following major fields:

  • gnbs: List of gNB’s to be simulated. Each item in the list holds configuration specific to a gNB. Support of Multiple gNBs: Two gnbs are configured by default.
  • profiles: List of test/simulation profiles.

Docker image omecproject/5gc-gnbsim Newest is: main-PR_92-a52e745 https://hub.docker.com/layers/omecproject/5gc-gnbsim/main-PR_92-a52e745/images/sha256-50256348d3aeb8f95e5b7bfb004ceb45109a51198bb0690da77301334323ff21?context=explore

PapaySail commented 11 months ago

Explore: Lookin at Aether OnRamp

main-gnbsim.yml: Configures the RAN emulator (gNBsim) to run in one or more servers, independent of the Core. Details documented here.

$ cd vars
$ cp main-gnbsim.yml main.yml

It links to https://github.com/opennetworkinglab/aether-gnbsim/blob/master/Makefile with ansible-playbook installing docker container:

gnbsim-docker-install:
ansible-playbook -i $(HOSTS_INI_FILE) $(GNBSIM_ROOT_DIR)/docker.yml --tags install \
--extra-vars "ROOT_DIR=$(ROOT_DIR)" --extra-vars $(EXTRA_VARS)

So, this deployment is not for Kubernetes. ❌

PapaySail commented 11 months ago

Explore: Looking at https://gerrit.opencord.org/plugins/gitiles/sdcore-helm-charts/+/refs/heads/master/5g-ran-sim/

git clone "https://gerrit.opencord.org/sdcore-helm-charts"
cd sdcore-helm-charts/sdcore-helm-charts/5g-ran-sim
helm dep update #Update Helm dependencies
helm install -n sdcore-5g --create-namespace

🆗 It creates statefulset gnbsim with service gnbsim-headles Next is to set values and implement it with Open5gs

PapaySail commented 11 months ago

Open5GS parameters to use is:

    plmnList:
      - plmn_id:
          mcc: "999"
          mnc: "70"
        s_nssai:
          - sst: 1
            sd: "0x111111"

Enter gnbsim pod using kubectl exec command and run following commands, 

$ ./gnbsim

Note: By default, the gNB Sim reads the configuration from /gnbsim/config/gnb.conf file.

PapaySail commented 11 months ago

Ok, two gNBs connected successfully at port :9478 & :9488.

AMF log:

```ruby 11/20 18:16:47.836: [amf] INFO: gNB-N2 accepted[10.233.64.42]:9487 in ng-path module (../src/amf/ngap-sctp.c:113) 11/20 18:16:47.836: [amf] INFO: gNB-N2 accepted[10.233.64.42] in master_sm module (../src/amf/amf-sm.c:733) 11/20 18:16:47.836: [amf] INFO: [Added] Number of gNBs is now 1 (../src/amf/context.c:1175) 11/20 18:16:47.836: [amf] INFO: gNB-N2[10.233.64.42] max_num_of_ostreams : 30 (../src/amf/amf-sm.c:772) 11/20 18:16:47.841: [amf] INFO: gNB-N2 accepted[10.233.64.42]:9488 in ng-path module (../src/amf/ngap-sctp.c:113) 11/20 18:16:47.841: [amf] INFO: gNB-N2 accepted[10.233.64.42] in master_sm module (../src/amf/amf-sm.c:733) 11/20 18:16:47.841: [amf] INFO: [Added] Number of gNBs is now 2 (../src/amf/context.c:1175) 11/20 18:16:47.841: [amf] INFO: gNB-N2[10.233.64.42] max_num_of_ostreams : 30 (../src/amf/amf-sm.c:772) 11/20 18:16:47.849: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:372) 11/20 18:16:47.849: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2481) 11/20 18:16:47.849: [amf] INFO: RAN_UE_NGAP_ID[1694498816] AMF_UE_NGAP_ID[1] TAC[1] CellID[0x1] (../src/amf/ngap-handler.c:533) 11/20 18:16:47.850: [amf] INFO: [suci-0-999-70-0-0-0-0000000001] Unknown UE by SUCI (../src/amf/context.c:1778) 11/20 18:16:47.850: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1560) 11/20 18:16:47.850: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:985) 11/20 18:16:47.850: [gmm] INFO: [suci-0-999-70-0-0-0-0000000001] SUCI (../src/amf/gmm-handler.c:152) 11/20 18:16:47.850: [gmm] ERROR: Cannot find Served TAI[PLMN_ID:02f839,TAC:1] (../src/amf/gmm-handler.c:302) 11/20 18:16:47.850: [gmm] ERROR: gmm_handle_registration_request() failed [12] (../src/amf/gmm-sm.c:1008) 11/20 18:16:47.850: [amf] WARNING: [suci-0-999-70-0-0-0-0000000001] Registration reject [12] (../src/amf/nas-path.c:219) 11/20 18:16:47.853: [amf] INFO: UE Context Release [Action:3] (../src/amf/ngap-handler.c:1632) 11/20 18:16:47.853: [amf] INFO: RAN_UE_NGAP_ID[1694498816] AMF_UE_NGAP_ID[1] (../src/amf/ngap-handler.c:1633) 11/20 18:16:47.853: [amf] INFO: SUCI[suci-0-999-70-0-0-0-0000000001] (../src/amf/ngap-handler.c:1636) 11/20 18:16:47.853: [amf] INFO: [Removed] Number of gNB-UEs is now 0 (../src/amf/context.c:2488) 11/20 18:16:47.853: [amf] INFO: [Removed] Number of AMF-UEs is now 0 (../src/amf/context.c:1653) 11/20 18:16:52.857: [amf] INFO: gNB-N2[10.233.64.42] connection refused!!! (../src/amf/amf-sm.c:785) 11/20 18:16:52.857: [amf] INFO: [Removed] Number of gNBs is now 1 (../src/amf/context.c:1202) 11/20 18:16:52.858: [amf] INFO: gNB-N2[10.233.64.42] connection refused!!! (../src/amf/amf-sm.c:785) 11/20 18:16:52.859: [amf] INFO: [Removed] Number of gNBs is now 0 (../src/amf/context.c:1202) ```

There is error in log:

 ERROR: Cannot find Served TAI[PLMN_ID:02f839,TAC:1]

but I can't see the source of mistake. Look at /gnbsim/config/gnb.conf:

      name: gnb1
      supportedTaList:
      - broadcastPlmnList:
        - plmnId:
            mcc: 999
            mnc: 70
          taiSliceSupportList:
          - sd: "111111"
            sst: 1
        tac: "000001"
.........
  profiles:
  - dataPktCount: 5
    defaultAs: 172.17.0.1
    enable: true
    execInParallel: false
    gnbName: gnb1
    key: 465B5CE8B199B49FAA5F0A2EE238A6BC
    opc: E8ED289DEBA952E4283B54E88E6183CA
    plmnId:
      mcc: 999
      mnc: 70
    profileName: profile2
    profileType: pdusessest
    startImsi: 999700000000001
    stepTrigger: false
    ueCount: 1

Converted 02f839 to plmn 208.93 🤷‍♂️ It's hard to find misconfiguration in file /gnbsim/config/gnb.conf

PapaySail commented 11 months ago

log level:trace OK {999 70}

2023-11-20T21:42:31Z [TRAC][GNBSIM][GNodeB][gnb1] GNodeB Initialized &{ 9487 10.233.64.42 2152 gnb1 {0xc00045b700  0xc000127800 } [{000001 [{{999 70} [{1 111111}]}]}] 0xc000135620 0xc00051a680 0xc00052f980 0xc00052f9e0 <nil> 0xc000229170 0xc000534450 0xc000127b60 0xc000536930}
PapaySail commented 11 months ago

With plmn 208.93 open5gs environment
@pirog-spb -- deployed at our cluster at dev open5gs20893/gnbsim-0

./gnbsim proceeded more, but crashes with [FATA][GNBSIM][App] Invalid Event ID: 0x2000054 [amf] ERROR: ngap_send_to_ran_ue: Expectation `rv == OGS_OK' failed.

Details

./gnbsim output: ```ruby 2023-11-22T10:26:03Z [INFO][GNBSIM][SimUe][imsi-208930000000001] Start new procedure PDU-SESSION-ESTABLISHMENT-PROCEDURE 2023-11-22T10:26:03Z [INFO][GNBSIM][SimUe][imsi-208930000000001] Initiating UE Requested PDU Session Establishment Procedure 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][ControlPlaneTransport] Wrote 22 bytes 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][GnbCpUe][3238002688] Handling event: UL-INFO-TRANSFER-EVENT 2023-11-22T10:26:03Z [INFO][GNBSIM][RealUe][imsi-208930000000001] Handling: PDU-SESSION-ESTABLISHMENT-REQUEST-EVENT 2023-11-22T10:26:03Z [INFO][NAS][Message] Encode ExtendedProtocolConfigurationOptions in EncodePDUSessionEstablishmentRequest 2023-11-22T10:26:03Z [INFO][GNBSIM][SimUe][imsi-208930000000001] Handling event: PDU-SESSION-ESTABLISHMENT-REQUEST-EVENT 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][ControlPlaneTransport] Wrote 56 bytes 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][GnbCpUe][3238002688] Handling event: UL-INFO-TRANSFER-EVENT 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][ControlPlaneTransport] Wrote 82 bytes 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][ControlPlaneTransport] Read 69 bytes from 10.233.43.233 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][GnbUeDao] Fetching GnbCpUe for RANUENGAPID: 3238002688 2023-11-22T10:26:03Z [INFO][GNBSIM][GNodeB][GnbCpUe][3238002688] Handling event: DOWNLINK-NAS-TRANSPORT-EVENT 2023-11-22T10:26:03Z [INFO][GNBSIM][SimUe][imsi-208930000000001] Handling event: DL-INFO-TRANSFER-EVENT 2023-11-22T10:26:03Z [INFO][GNBSIM][RealUe][imsi-208930000000001] Handling: DL-INFO-TRANSFER-EVENT 2023-11-22T10:26:03Z [INFO][GNBSIM][RealUe][imsi-208930000000001] Calculate NAS MAC (algorithm: 2, DLCount: 0x2) 2023-11-22T10:26:03Z [INFO][GNBSIM][RealUe][imsi-208930000000001] NAS integrity key: 0247eca8605d6fc0aba644ffeeaaf4b4 cmac value: 0x893adc32 2023-11-22T10:26:03Z [INFO][GNBSIM][RealUe][imsi-208930000000001] Received Message Type: 84 2023-11-22T10:26:03Z [FATA][GNBSIM][App] Invalid Event ID: 0x2000054 ``` ```ruby bash-5.1# cat summary.log bash-5.1# ls gnbsim gnbsim.log summary.log bash-5.1# ls -la total 21572 drwxr-xr-x 1 root root 4096 Nov 22 10:26 . drwxr-xr-x 1 root root 4096 Nov 22 10:26 .. -rwxr-xr-x 1 root root 22058466 Dec 2 2022 gnbsim -rw-r--r-- 1 root root 13612 Nov 22 10:26 gnbsim.log -rw-r--r-- 1 root root 0 Nov 22 10:26 summary.log ``` Root source is froom AMF [amf] ERROR: ngap_send_to_ran_ue: Expectation `rv == OGS_OK' failed.:: ```ruby 11/22 11:25:49.668: [amf] INFO: [Added] Number of AMF-Sessions is now 1 (../src/amf/context.c:2502) 11/22 11:25:49.668: [gmm] INFO: UE SUPI[imsi-208930000000001] DNN[internet] S_NSSAI[SST:1 SD:0x111111] (../src/amf/gmm-handler.c:1228) 11/22 11:25:49.674: [amf] INFO: gNB-N2[10.233.78.78] connection refused!!! (../src/amf/amf-sm.c:785) 11/22 11:25:49.674: [amf] INFO: [Removed] Number of gNBs is now 1 (../src/amf/context.c:1202) 11/22 11:25:49.675: [amf] INFO: gNB-N2[10.233.78.78] connection refused!!! (../src/amf/amf-sm.c:785) 11/22 11:25:49.675: [amf] INFO: [Removed] Number of gNBs is now 0 (../src/amf/context.c:1202) 11/22 11:25:49.680: [amf] ERROR: gNB has already been removed (../src/amf/ngap-path.c:56) 11/22 11:25:49.680: [amf] ERROR: ngap_send_to_ran_ue: Expectation `rv == OGS_OK' failed. (../src/amf/ngap-path.c:95) 11/22 11:25:49.680: [amf] ERROR: amf_namf_comm_handle_n1_n2_message_transfer: Expectation `r == OGS_OK' failed. (../src/amf/namf-handler.c:210) ```

But eUPF have successfully created session:

2023/11/22 11:34:44 Handling PFCP message from 10.233.78.76:8805
2023/11/22 11:34:44 Got Session Establishment Request from: 10.233.78.76.
2023/11/22 11:34:44
Session Establishment Request:
  CreatePDR ID: 1
    FAR ID: 1
    QER ID: 1
    URR ID: 1
    Source Interface: 1
    UE IPv4 Address: 10.11.0.7
  CreatePDR ID: 2
    Outer Header Removal: 0
    FAR ID: 2
    QER ID: 1
    Source Interface: 0
    TEID: 22
    Ipv4: 10.233.0.179
    Ipv6: <nil>
    UE IPv4 Address: 10.11.0.7
........
2023/11/22 11:34:44 Saving FAR info to session: 1, {Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0}
2023/11/22 11:34:44 EBPF: Put FAR: internalId=1, qerInfo={Action:12 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0}
2023/11/22 11:34:44 WARN: No OuterHeaderCreation
2023/11/22 11:34:44 Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0}
2023/11/22 11:34:44 EBPF: Put FAR: internalId=2, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0}
2023/11/22 11:34:44 Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:6 RemoteIP:1280239882 LocalIP:188803338 TransportLevelMarking:0}
2023/11/22 11:34:44 EBPF: Put FAR: internalId=3, qerInfo={Action:2 OuterHeaderCreation:1 Teid:6 RemoteIP:1280239882 LocalIP:188803338 TransportLevelMarking:0}
2023/11/22 11:34:44 Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:999999000 MaxBitrateDL:999999000 StartUL:0 StartDL:0}
2023/11/22 11:34:44 EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:999999000 MaxBitrateDL:999999000 StartUL:0 StartDL:0}
2023/11/22 11:34:44 EBPF: Put PDR Downlink: ipv4=10.11.0.7, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:1}
2023/11/22 11:34:44 EBPF: Put PDR Uplink: teid=22, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1}
2023/11/22 11:34:44 EBPF: Put PDR Uplink: teid=23, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:1}
2023/11/22 11:34:44 Session Establishment Request from 10.233.78.76 accepted.
PapaySail commented 10 months ago

@pirog-spb Testbed deployed at our dev server. Open terminal at open5gs20893/gnbsim-0 and start ./gnbsim --cfg gnb.conf

There is profiles sections in gnb.conf profile2 pdusessest is enabled registration proceeded successfull, pdu session establish crashes gnbsim with [FATA][GNBSIM][App] Invalid Event ID: 0x2000054

  profiles: # profile information
    - profileType: register # profile type
      profileName: profile1 # uniqely identifies a profile within application
      enable: false # Set true to execute the profile, false otherwise.
......
    - profileType: pdusessest # profile type
      profileName: profile2 # uniqely identifies a profile within application
      enable: true # Set true to execute the profile, false otherwise.

results you can see by less summary.log

SMF log:

11/26 18:17:21.445: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:898)
11/26 18:17:21.445: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:2975)
11/26 18:17:21.458: [smf] ERROR: No handler for event OGS_EVENT_NAME_SBI_SERVER (../src/smf/gsm-sm.c:627)
11/26 18:17:21.458: [smf] INFO: UE SUPI[imsi-208930000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:495)
11/26 18:17:32.458: [sbi] ERROR: on_stream_close_callback() failed (5:STREAM_CLOSED) (../lib/sbi/nghttp2-server.c:846)
laf0rge commented 1 week ago

I as a eUPF developer want to be sure that eUPF can process a lot of UEs from a lot of gNBs in order to provide solution for commercial use.

I think if this is the goal, you shouldn't bother with a gNB + UE simulator and/or setting up a 5GC etc. - All you want to test is the UPF. Hence, all you need is something that

An experimental io_uring based GTP load generator is at https://gitea.osmocom.org/cellular-infrastructure/gtp-load-gen - but it lacks the PFCP part

There's also a client-side command-line tool that can generate sesssions via PFCP in the osmo-upf repository at https://gitea.osmocom.org/cellular-infrastructure/osmo-upf/src/branch/master/src/osmo-pfcp-tool

Right now, @pespin is working on a new, dedicated UPF benchmarking tool integrating all of those features, see https://gitea.osmocom.org/cellular-infrastructure/upf-benchmark