Select SMF based on sst, sd and dnn (S-NSSAI)

[pharezende]

Dear all,

I'm essentially trying to run this scenario (link) using Docker containers. I created a Virtual Machine (VM) running UERANSIM and another VM running this project as a standalone 5G core (sa-deploy.yaml). Specifically, I created new YAML files, one inside the upf folder named upf2 and another one called smf2 inside the smf folder. I configured all the parameters, including a new slice with st:1; sd: 000002 and a DNN named private.

I want the PDU session to be created by SMF1 passing through UPF1 and going through the internet DNN when the UE has an st:1 and sd:000001. In case a UE has an st:1 and sd:000002, SMF2 should be selected by the AMF and then establish a session passing through UPF2 and then to the private network (private DNN).

After configuring everything, I was able to deploy the containers, including smf2 and upf2. They are able to communicate with each other using PFCP. However, I'm having a problem where the AMF is selecting a default SMF and trying to create a PDU session with it, regardless of the slice and DNN considered. Therefore, it may occur that the chosen SMF does not have the DNN and then the SMF crashes during the PDU session establishment (in line with this). Then, only after this crash, the other SMF is selected and finally the PDU session can be created accordingly. Somehow, the AMF is not selecting the SMF based on these rules, considering only the latest deployed SMF (a similar problem happened here, but a fix was presented a while ago).

If anyone has successfully implemented a similar scenario without encountering any issues, I'd appreciate any insights or suggestions.

Example: The AMF selects an SMF that supports the private dnn and therefore the pdu is established for the first request from the UE. Subsequently, the same SMF receives a request for pdu session establishment to the internet pdn, and then it crashes.

log.zip Core

03/18 10:33:43.780: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1012)
03/18 10:33:43.780: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3108)
03/18 10:33:43.786: [smf] INFO: UE SUPI[imsi-999700000000002] DNN[private] IPv4[192.168.200.20] IPv6[] (../src/smf/npcf-handler.c:528)
03/18 10:33:43.786: [gtp] INFO: gtp_connect() [172.22.0.82]:2152 (../lib/gtp/path.c:60)
03/18 10:33:58.893: [smf] INFO: [Added] Number of SMF-UEs is now 2 (../src/smf/context.c:1012)
03/18 10:33:58.893: [smf] INFO: [Added] Number of SMF-Sessions is now 2 (../src/smf/context.c:3108)
03/18 10:33:58.895: [pfcp] ERROR: CHECK CONFIGURATION: Cannot find subnet [family:2, dnn:internet] (../lib/pfcp/context.c:1743)
03/18 10:33:58.895: [pfcp] ERROR: Please add FALLBACK subnet as below. (../lib/pfcp/context.c:1745)
03/18 10:33:58.895: [pfcp] ERROR:     subnet: (../lib/pfcp/context.c:1746)
03/18 10:33:58.895: [pfcp] ERROR:      - addr: 10.50.0.1/16 (../lib/pfcp/context.c:1748)
03/18 10:33:58.895: [smf] ERROR: ogs_pfcp_ue_ip_alloc() failed[77] (../src/smf/context.c:1548)
03/18 10:33:58.895: [smf] FATAL: smf_nudm_sdm_handle_get: Assertion `OGS_PFCP_CAUSE_REQUEST_ACCEPTED == smf_sess_set_ue_ip(sess)' failed. (../src/smf/nudm-handler.c:309)

RAN

^Croot@ueransim:~# ./UERANSIM/build/nr-ue -c UERANSIM/config/open5gs-ue.yaml
UERANSIM v3.2.6
[2023-03-18 10:33:59.198] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2023-03-18 10:33:59.198] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2023-03-18 10:33:59.198] [nas] [info] Selected plmn[999/70]
[2023-03-18 10:33:59.199] [rrc] [info] Selected cell plmn[999/70] tac[1] category[SUITABLE]
[2023-03-18 10:33:59.199] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2023-03-18 10:33:59.199] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2023-03-18 10:33:59.199] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2023-03-18 10:33:59.202] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-03-18 10:33:59.202] [nas] [debug] Sending Initial Registration
[2023-03-18 10:33:59.203] [rrc] [debug] Sending RRC Setup Request
[2023-03-18 10:33:59.203] [rrc] [info] RRC connection established
[2023-03-18 10:33:59.203] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2023-03-18 10:33:59.204] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2023-03-18 10:33:59.204] [nas] [info] UE switches to state [CM-CONNECTED]
[2023-03-18 10:33:59.208] [nas] [debug] Authentication Request received
[2023-03-18 10:33:59.212] [nas] [debug] Security Mode Command received
[2023-03-18 10:33:59.212] [nas] [debug] Selected integrity[2] ciphering[0]
[2023-03-18 10:33:59.221] [nas] [debug] Registration accept received
[2023-03-18 10:33:59.222] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2023-03-18 10:33:59.222] [nas] [debug] Sending Registration Complete
[2023-03-18 10:33:59.222] [nas] [info] Initial Registration is successful
[2023-03-18 10:33:59.222] [nas] [debug] Sending PDU Session Establishment Request
[2023-03-18 10:33:59.222] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-03-18 10:33:59.427] [nas] [debug] Configuration Update Command received
[2023-03-18 10:33:59.522] [nas] [error] SM forwarding failure for message type[193] with cause[PAYLOAD_NOT_FORWARDED]
[2023-03-18 10:33:59.522] [nas] [debug] Aborting SM procedure for PTI[1], PSI[1]
[2023-03-18 10:34:00.298] [nas] [debug] Sending PDU Session Establishment Request
[2023-03-18 10:34:00.298] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-03-18 10:34:17.215] [nas] [warning] Retransmitting PDU Session Establishment Request due to T3580 expiry

Best regards,

[herlesupreeth]

Can you post your config file and the pcap file while attempting to create PDU session on private DNN?

[pharezende]

Hello @herlesupreeth ,

Sure. You can find the code in this repo: https://github.com/pharezende/docker_open5gs/commits/master The core and ran pcaps here: traces.zip

In the evaluated scenario, the SMF2 (with the private dnn) is selected by the AMF. The first UE to be subscribed requests the private network, so everything worked smoothly. Afterward, the second UE (from the internet dnn) requested and then the SMF2 crashed.

Obs: I'm running docker-compose pointing to this file sa-deploy.yaml.

[herlesupreeth]

I had a look at the traces and it looks like a bug in open5gs. But before raising a issue there, I would suggest to remove the below part in smf.yaml and smf2.yaml under info section and give a try. I don't think it will fix the issue but worth a try since TAI remains the same between the slices and maybe there is some logical error in code (just my guess).

        tai:
          - plmn_id:
              mcc: MCC
              mnc: MNC
            tac: 1

If the issue still remains then I would suggest raising an issue in open5gs repo with a pcap taken for the following scenario

1. Bring down all 5GC components
2. Start the trace --> Reason to start the trace before bringing up 5GC is to check whether SMF(s) register with NRF with the slice information or not
3. Bring up all the components
4. Attach gNB and UEs as you did earlier
5. Stop the trace

Btw, I ran a quick test at my end to check whether SMF sends SMInfo containing slice information or not and it does so the issue you are facing more and more points to a potential bug in open5gs

[pharezende]

Hi @herlesupreeth. Didn't work. I will follow your suggestion and raise an issue on the open5GS repo. Regarding the wireshark trace, do you use a specific dissector to see the messages exchanged between the core functions? I can only see messages from NAS and NGAP, but not application level messages within the core. Only TCP messages are shown. My wireshark version is: 3.6.2.

[herlesupreeth]

For Core traffic, I use the below display filter pfcp || gtp || ngap || http2.data.data || http2.headers

Make sure you have set as mentioned here - https://github.com/open5gs/open5gs/discussions/751

[s5uishida]

Hi @pharezende and @herlesupreeth

I attached the pcap of my scenario and the SVG files created with 5G Trace Visualizer.

20230321-2.zip

According to these, in cp_0321_12_OS_002.svg, you may confirm the registration processing of information including smfInfo from SMF to SCP. Note that the number at the beginning of the sequence arrow in the figure corresponds to the pcap number.

I think that the mechanism of SMF selection by S-NSSAI works fine.

I'm sorry if I misunderstood.

[herlesupreeth]

Thanks @s5uishida for trying it out. Will try at my end and see if I face the issue

[pharezende]

Hi @herlesupreeth and @s5uishida,

@herlesupreeth I was able to display the traffic with your suggestion. Thanks! @s5uishida Thanks for your response. Everything looks in order in your scenario. Some of the differences are that you are using one dnn (internet) and my AMF is not receiving the smfInfo. There may be an issue in my configuration files or elsewhere.

[s5uishida]

Hi @pharezende

In my scenario, I focused only on the S-NSSAI differences to validate the SMF selection.

In addition, when I set dnn=private for SMF2 and UPF2, and tried, SMF2 was selected and connected to the data network under UPF2.

[pharezende]

Hi all,

@s5uishida Thanks for your feedback! @herlesupreeth I just cloned your repository on another VM and set up a simple scenario with a single UPF, SMF, and DNN. The PDU session was established and I was able to ping the internet from the UE, but the SMF information was not sent at all. Is this normal behavior? Please find the trace attached to this message.

I used this filter: json.member == "smfInfo"

experiment.zip

[s5uishida]

Hi @pharezende

I tried sa-deploy.yaml, adding settings for S-NSSAI(sst=1, sd=0x000001) in amf.yaml, smf.yaml and nssf.yaml. I attached the pcap and SVG files at that time.

20230325-4.zip

As seeing do_0325_02_OS_000.svg, you may see that the registration information including smfInfo is sent from SMF to SCP. Ping worked fine too.

[pharezende]

Hi @s5uishida ,

In your last trace ( inside 20230325-2.zip), the smfInfo is sent to the SCP, but it does not reach the AMF (172.22.0.10). However, if you check the trace within 20230321-2.zip, you can see that the smfInfo reaches the AMF (127.0.0.5). Were you able to implement my problematic scenario (i.e. with two smfs, upfs and dnns) using docker_open5gs?

[herlesupreeth]

@herlesupreeth I just cloned your repository on another VM and set up a simple scenario with a single UPF, SMF, and DNN. The PDU session was established and I was able to ping the internet from the UE, but the SMF information was not sent at all. Is this normal behavior? Please find the trace attached to this message.

No, its not normal. I tested it on my end and it did send SMInfo

Following a screenshot from the trace I took when I tried

[herlesupreeth]

@pharezende Btw, I tried at my end as well and SMF was crashing when trying out different slices with different DNNs

I pushed a new branch called slices_2upf_2smf which has the necessary files to reproduce this crash. @s5uishida It would be great if you could have a look and see if I am missing something or config is wrong. Many thanks

[s5uishida]

Hi @herlesupreeth

Probably, I think that it works by changing as follows.

--- sa-deploy.yaml.orig 2023-03-27 14:34:46.869911332 +0900
+++ sa-deploy.yaml      2023-03-27 14:55:58.810908273 +0900
@@ -137,6 +137,7 @@
     depends_on:
       - nrf
       - scp
+      - amf
     container_name: smf
     env_file:
       - .env
@@ -197,6 +198,7 @@
     depends_on:
       - nrf
       - scp
+      - amf
     container_name: smf2
     env_file:
       - .env
@@ -255,8 +257,6 @@
   amf:
     image: docker_open5gs
     depends_on:
-      - smf
-      - upf
       - nrf
       - scp
       - ausf

I attached the pcap and SVG files.

20230327-2.zip

[s5uishida]

Hi @pharezende

Have you tried the above sa-deploy.yaml changes?

[pharezende]

Hi @s5uishida,

Thanks for your comment. Not yet, I will try soon and let you know.

[pharezende]

Hi @s5uishida,

Sorry for the delayed response, I only had time to test it today. Using your proposal and the 'slices_2upf_2smf' branch, I was able to establish the PDU sessions with two devices - one with the 'internet' DNN and the other with the 'private' DNN. I will conduct more experiments in the next few days, but it seems like everything is working correctly now. Thank you very much! However, I didn't understand why changing the order in which services start (through depends_on option) can resolve the issue. Please let me know why this happens.

Thanks again!

[herlesupreeth]

Hi @s5uishida Thanks for the solution. And thanks @pharezende for verifying it. However, its quite strange changing the order of service bring up fixed it.

[s5uishida]

Hi @pharezende and @herlesupreeth

Thank you for confirming.

At #711 and #865 of *1 in the following SMF service registration sequence, POST /nnrf-nfm/v1/nf-status-notify is sent from SCP to AMF.

   AMF      SMF      SCP      NRF
    |        |        |        |
    |        |------->|        |
    |        |        |------->|
    |        |        |<-------|
*1) |<----------------|        |
    |---------------->|        |
    |        |<-------|        |
    |        |        |        |

Therefore, AMF must start before SMF.

However, I don't know why AMF has this NRF service.

[herlesupreeth]

@s5uishida Thanks a lot for investigating this and for the info.

@pharezende I plan to remove the slices_2upf_2smf branch soon as I don't want to maintain another branch. Let me know if you need it and I will wait until you take a copy of it on your fork and then delete it.

[s5uishida]

Hi @herlesupreeth

If the *1 sequence is not necessary, I think the original sa-deploy.yaml is correct in this case as well.

[pharezende]

Hi @herlesupreeth ,

Feel free to delete the slices_2upf_2smf branch. Thanks again @herlesupreeth and @s5uishida!

[s5uishida]

Hi @herlesupreeth

Sorry, but could you wait a moment before deleting the slices_2upf_2smf branch?

This Open5GS discussion references this branch.

It would be helpful if you could leave this branch open until this discussion is closed.

Of course, no maintenance is required :-)

Thank you very much in advance.

[herlesupreeth]

Sure @s5uishida . Thanks for raising this discussion over at open5gs

[s5uishida]

Hi @herlesupreeth

That discussion was closed. As a result, it seems that AMF should start before SMF.

Thank you very much for waiting for deleting the slices_2upf_2smf branch.

[herlesupreeth]

Saw the discussion. But sad to see that no proper explanation was there as to why the sequence of AMF and SMF bring up had an impact on multiple slices over multiple SMF/UPF. Thanks once again

[pharezende]

Hi again,

Currently, UEs from slices 1 and 2 have unrestricted access to both the Internet and Private Data Networks, despite going through different UPFs. However, I want to restrict UEs from slice 1 to only access the Internet, and drop any packets in the UPF if they attempt to access the Private Data Network.

While I understand that this can be achieved using ip-tables rules, I'm specifically looking for a way to program the UPFs to enforce this behavior. Based on my reading of the PFCP protocol (link), it seems it's possible to create forwarding rules to drop such packets using a combination of PDR/FAR information during the PFCP Session Establishment/Modification Request on the UPF. To achieve this, I may need to modify the smf.yaml/smf2.yaml files in the docker_open5gs directory. Do you have any examples of how to implement these rules?

Moreover, I'm wondering if there is another interface to define these forwarding rules without having to specify them in the YAML files.

Attached, you'll find a pcap that illustrates how the same UE can currently communicate with different Data Networks. (I'm using the slices_2upf_2smf branch).

Thanks again! trace-11-04.zip

[herlesupreeth]

Currently, UEs from slices 1 and 2 have unrestricted access to both the Internet and Private Data Networks, despite going through different UPFs. However, I want to restrict UEs from slice 1 to only access the Internet, and drop any packets in the UPF if they attempt to access the Private Data Network.

Are you referring to a MEC scenario? or like a local breakout scenario?

I'm specifically looking for a way to program the UPFs to enforce this behavior. Based on my reading of the PFCP protocol (link) To achieve this, I may need to modify the smf.yaml/smf2.yaml files in the docker_open5gs directory. Do you have any examples of how to implement these rules?

I don't think its possible via config change. But definitely possible via open5gs code changes. If you dont want to go down the route of source code change then IP tables is your best bet.

[s5uishida]

I don't think its possible via config change. But definitely possible via open5gs code changes. If you dont want to go down the route of source code change then IP tables is your best bet.

I think so, too.

@pharezende

By the way, you may try Uplink Classifier with free5GC. For your reference, my repository has example ULCL configurations using free5GC.

@herlesupreeth @pharezende

Sorry for digressing from docker_open5gs.

[pharezende]

Are you referring to a MEC scenario? or like a local breakout scenario?

Yes, something like a local breakout. So I assume it not possible to create any chains of UPFs connected through the N9 interface using config files, only via source code.

I don't think its possible via config change. But definitely possible via open5gs code changes. If you dont want to go down the route of source code change then IP tables is your best bet.

I see. Do you have an example on how to do this (or something similar) using the source code?

By the way, you may try Uplink Classifier with free5GC. For your reference, my repository has example ULCL configurations using free5GC.

Thanks! I will take a look on the tool and your experiments.🙂 It looks like there is also a docker version of free5GC.

[herlesupreeth]

I see. Do you have an example on how to do this (or something similar) using the source code?

Nope unfortunately. I don't have time to dig into the code. But would be cool addition to open5gs project. In terms of IPTables I can come up with a solution quickly if you could describe the setup you are trying to achieve (like the IP you want to allow in an UPF etc)

[pharezende]

I will go through the source code to understand more about it, but probably after taking a look at free5GC. I have actually implemented some rules using iptables, but the issue is that I was expecting to find a upf that looked more like an OpenFlow switch, fully programmable. I will let you know if I extend the source code with this feature. Thank you very much for your help!