search

herlesupreeth / docker_open5gs

Docker files to run open5gs + IMS + eNB + gNB + NR-UE in a docker
BSD 2-Clause "Simplified" License
326 stars 188 forks source link

Trying registration with NRF repetitively #9

Closed ljysdfz closed 3 years ago

ljysdfz commented 3 years ago

Hi, dude.

I've launched all the 5gs containers. But the terminal continues spitting following warnings.

amf          | 01/05 02:32:06.933: [amf] WARNING: [e398c962-4efd-41eb-a471-2f1403b477bc] Retry to registration with NRF (../src/amf/nf-sm.c:161)
smf          | 01/05 02:32:07.289: [smf] WARNING: [d90f754a-4efd-41eb-82e9-29bc882004a6] Retry to registration with NRF (../src/smf/nf-sm.c:161)
udm          | 01/05 02:32:07.495: [udm] WARNING: [de8c6410-4efd-41eb-b9f7-e9020de45231] Retry to registration with NRF (../src/udm/nf-sm.c:161)
ausf         | 01/05 02:32:08.196: [ausf] WARNING: [db634be6-4efd-41eb-adb2-a796ad4f4841] Retry to registration with NRF (../src/ausf/nf-sm.c:161)
udr          | 01/05 02:32:09.557: [udr] WARNING: [e6f164ac-4efd-41eb-ac5a-e1560a8cbc5d] Retry to registration with NRF (../src/udr/nf-sm.c:161)
amf          | 01/05 02:32:09.942: [amf] WARNING: [e398c962-4efd-41eb-a471-2f1403b477bc] Retry to registration with NRF (../src/amf/nf-sm.c:161)
smf          | 01/05 02:32:10.293: [smf] WARNING: [d90f754a-4efd-41eb-82e9-29bc882004a6] Retry to registration with NRF (../src/smf/nf-sm.c:161)
udm          | 01/05 02:32:10.500: [udm] WARNING: [de8c6410-4efd-41eb-b9f7-e9020de45231] Retry to registration with NRF (../src/udm/nf-sm.c:161)
fhoss        | [Thread-24] ERROR de.fhg.fokus.diameter.DiameterPeer.peer.StateMachine  - StateMachine: Peer scscf.ims.mnc003.mcc460.3gppnetwork.org:3870 not responding to connection attempt

I've merely modified the .env file as follows. The docker host is planned at 192.168.12.6. Other IPs have not been changed.

#Timezone been changed
#TZ=Asia/Shanghai

#MCC & MNC been changed
MCC=460
MNC=03

TEST_NETWORK=172.22.0.0/24
#Host IP been adapted
DOCKER_HOST_IP=192.168.12.6
#DOCKER_HOST_IP=172.22.0.6

(No more change below)
......
......

It'd be appreciated should any ideas be presented.

herlesupreeth commented 3 years ago

can you make sure you are not running multiple instances of the 5g core? i.e. send me the output of docker ps command

ljysdfz commented 3 years ago

@herlesupreeth 

fgs@open5gs:~$ sudo docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                                                                           NAMES
d05bf7d39ccb        docker_open5gs_mme     "/bin/sh -c '/mnt/mm…"   16 hours ago        Up 2 minutes        2123/udp, 3868/sctp, 3868/tcp, 3868/udp, 5868/sctp, 5868/udp, 36412/sctp, 5868/tcp              mme
0b450a465822        docker_open5gs_amf     "/bin/sh -c '/mnt/am…"   16 hours ago        Up 2 minutes        7777/tcp, 38412/sctp                                                                            amf
6afb1d85f3ee        docker_open5gs_hss     "/bin/sh -c '/mnt/hs…"   16 hours ago        Up 3 minutes        3868/sctp, 3868/tcp, 3868/udp, 0.0.0.0:3000->3000/tcp, 5868/sctp, 5868/tcp, 5868/udp            hss
f8e1bcbb389b        docker_pcscf           "/bin/sh -c '/mnt/pc…"   18 hours ago        Up 3 minutes        3871/tcp, 3871/udp, 5060/tcp, 5060/udp, 5100-5120/tcp, 5100-5120/udp, 6100/tcp, 6100-6120/udp   pcscf
a3f5fe52e8ff        docker_open5gs_sgwc    "/bin/sh -c '/mnt/sg…"   18 hours ago        Up 2 minutes        2123/udp, 8805/udp                                                                              sgwc
5af382b503b9        docker_open5gs_sgwu    "/bin/sh -c '/mnt/sg…"   18 hours ago        Up 2 minutes        2152/udp, 8805/udp                                                                              sgwu
c3865f7317ae        docker_scscf           "/bin/sh -c '/mnt/sc…"   18 hours ago        Up 3 minutes        3870/tcp, 3870/udp, 6060/tcp, 6060/udp                                                          scscf
2b4bcc992b26        docker_icscf           "/bin/sh -c '/mnt/ic…"   18 hours ago        Up 3 minutes        3869/tcp, 3869/udp, 4060/tcp, 4060/udp                                                          icscf
e43ca53e4d95        docker_open5gs_upf     "/bin/sh -c '/mnt/up…"   18 hours ago        Up 2 minutes        2152/udp, 8805/udp                                                                              upf
ce5607ddcc16        docker_fhoss           "/bin/sh -c /mnt/fho…"   18 hours ago        Up 3 minutes        3868/tcp, 3868/udp, 0.0.0.0:8080->8080/tcp                                                      fhoss
63b4cc25d718        docker_open5gs_udr     "/bin/sh -c '/mnt/ud…"   18 hours ago        Up 2 minutes        7777/tcp                                                                                        udr
90034446d0bd        docker_open5gs_udm     "/bin/sh -c '/mnt/ud…"   18 hours ago        Up 2 minutes        7777/tcp                                                                                        udm
39efe9f34f5d        docker_open5gs_ausf    "/bin/sh -c '/mnt/au…"   18 hours ago        Up 2 minutes        7777/tcp                                                                                        ausf
5648245e60d3        docker_open5gs_smf     "/bin/sh -c '/mnt/sm…"   18 hours ago        Up 2 minutes        2123/udp, 3868/sctp, 3868/tcp, 3868/udp, 5868/tcp, 5868/udp, 8805/udp, 5868/sctp, 7777/tcp      smf
b3717f2b2bde        docker_open5gs_pcrf    "/bin/sh -c '/mnt/pc…"   18 hours ago        Up 3 minutes        3868/sctp, 3868/tcp, 3868/udp, 5868/sctp, 5868/tcp, 5868/udp                                    pcrf
e3321e94c09c        docker_rtpengine       "/bin/sh -c /mnt/rtp…"   18 hours ago        Up 3 minutes        2223/udp, 49000-50000/udp                                                                       rtpengine
33431fa3bc04        docker_mysql           "/bin/sh -c /mysql_i…"   18 hours ago        Up 3 minutes        3306/tcp                                                                                        mysql
bdc9dd530e33        docker_dns             "/bin/sh -c '/mnt/dn…"   18 hours ago        Up 3 minutes        53/udp                                                                                          dns
7fbaf3260b45        docker_open5gs_nrf     "/bin/sh -c '/mnt/nr…"   18 hours ago        Up 2 minutes        7777/tcp                                                                                        nrf
60df588865aa        docker_open5gs_mongo   "/bin/sh -c /mnt/mon…"   18 hours ago        Up 3 minutes        27017/tcp, 27017/udp                                                                            mongo
herlesupreeth commented 3 years ago

please execute the below commands in the following order

docker-compose down
docker system prune -f
source .env
docker-compose up
ljysdfz commented 3 years ago

@herlesupreeth

Thank you, sir. After I have pruned the redundant images, it doesn't work as a result. It's quite weird.

ljysdfz commented 3 years ago

@herlesupreeth

Should I change the default test network 172.22.0.0/24, which is not actually configured as a VLAN by myself, into the exact virtual machine network 192.168.12.0/24 ?

ljysdfz commented 3 years ago

Given the following errors, I have to take the connection problem into account.

icscf        |  0(25) ERROR: db_mysql [km_my_con.c:139]: db_mysql_new_connection(): driver error: Can't connect to MySQL server on '172.22.0.17' (111)
icscf        |  0(25) ERROR: <core> [db.c:319]: db_do_init2(): could not add connection to the pool
icscf        |  0(25) ERROR: ims_icscf [db.c:114]: ims_icscf_db_init(): ERR:ims_icscf_db_init: cannot initialize database connection
icscf        |  0(25) ERROR: db_mysql [km_my_con.c:139]: db_mysql_new_connection(): driver error: Can't connect to MySQL server on '172.22.0.17' (111)
icscf        |  0(25) ERROR: <core> [db.c:319]: db_do_init2(): could not add connection to the pool
icscf        |  0(25) ERROR: ims_icscf [db.c:114]: ims_icscf_db_init(): ERR:ims_icscf_db_init: cannot initialize database connection
icscf        |  0(25) ERROR: db_mysql [km_my_con.c:139]: db_mysql_new_connection(): driver error: Can't connect to MySQL server on '172.22.0.17' (111)
icscf        |  0(25) ERROR: <core> [db.c:319]: db_do_init2(): could not add connection to the pool
icscf        |  0(25) ERROR: ims_icscf [db.c:114]: ims_icscf_db_init(): ERR:ims_icscf_db_init: cannot initialize database connection
icscf        |  0(25) ERROR: db_mysql [km_my_con.c:139]: db_mysql_new_connection(): driver error: Can't connect to MySQL server on '172.22.0.17' (111)
icscf        |  0(25) ERROR: <core> [db.c:319]: db_do_init2(): could not add connection to the pool
icscf        |  0(25) ERROR: ims_icscf [db.c:114]: ims_icscf_db_init(): ERR:ims_icscf_db_init: cannot initialize database connection
ljysdfz commented 3 years ago

image

Is this permission denied making sense ?

herlesupreeth commented 3 years ago

Is this permission denied making sense ?

You can ignore them

Should I change the default test network 172.22.0.0/24, which is not actually configured as a VLAN by myself

this is the docker internal network, it shouldnt be configured as VLAN on physical network

Strange, I am not facing any such errors in my setup and I am using the same repo and latest commit. Sorry cant help

ljysdfz commented 3 years ago

@herlesupreeth

Never mind, man. It's pretty kind of you for helping me exclude extensive disruptions. I'm keeping on it by making a brand new VM and building the docker from scratch. Hope I'll figure it out.

herlesupreeth commented 3 years ago

Let me know how it goes.

Btw, can you send me the version of docker-ce and docker-compose you are running?

ljysdfz commented 3 years ago

@herlesupreeth

Oh buddy you just reminded me of what might be a blunder I've done -- What I used before was docker.io rather than docker-ce!! Oh my goodness!! I don't know if this choice caused the defeat of my work, but that is exactly what I did indeed. And the version is named Docker version 19.03.6, build 369ce74a3c.

What I'm trying with for the moment is Docker version 20.10.2, build 2291f61, which is installed by following instructions posted in https://docs.docker.com/engine/install/ubuntu/. It should be the latest one.

As far as docker-compose goes, it's been always installed through directions of https://docs.docker.com/compose/install/. So the version number should be 1.27.4, which has been indicated in the curl command.

ljysdfz commented 3 years ago

@herlesupreeth

Hi, dude. I just came up with an inquiry. I wonder if I should change the IP of UPF into 192.168.12.XX in order for it to connect with INTERNET ?

herlesupreeth commented 3 years ago

No need to set that as the traffic to INTERNET would be masqueraded

ljysdfz commented 3 years ago

@herlesupreeth

Aha~ I found that the iptables command has been tucked in the upf/tun_if.py

ljysdfz commented 3 years ago

image

@herlesupreeth

Bro, these are two typical network interfaces in my host VM. Is it correct should I set the environment parameter DOCKER_HOST_IP as the IP of docker0 ?

herlesupreeth commented 3 years ago

Are you running the eNB in the same docker environment? If so then leave the DOCKER_HOST_IP=172.22.0.6 as it is. If not, then set the DOCKER_HOST_IP=192.168.12.6

ljysdfz commented 3 years ago

@herlesupreeth

Got it !! I dare not to modify anything for the time being. So all network functions should be expected to be running in the same docker environment which is supposed to be encompassed in the 172.22.0.0/24 network. I would leave the host IP to be 172.22.0.6.

Please forgive my ignorance, I feel doubtful about why the DOCKER_HOST_IP is identical to that of SGWU instead of UPF ? By intuition it's tending to bind the host IP to the outlet of the network.

herlesupreeth commented 3 years ago

Please forgive my ignorance, I feel doubtful about why the DOCKER_HOST_IP is identical to that of SGWU instead of UPF ? By intuition it's tending to bind the host IP to the outlet of the network.

I am assuming that you are using an eNB and not a gNB. If thats the case then the userplane traffic from eNB hits first SGWU and then UPF thats the reason DOCKER_HOST_IP should be set at SGWU and not UPF. But if its the gNB then it would have been UPF

ljysdfz commented 3 years ago

@herlesupreeth

Cool~ Let's go further~

I suppose that if I were to realize an SA architecture I should only run the network functions involved in the 5gc i.e. the following components, and finally the DOCKER_HOST_IP should be set as that of UPF. Is my thought making sense ?

image image

herlesupreeth commented 3 years ago

That is exactly right (assuming you are running the gNB also in the same docker environment)

ljysdfz commented 3 years ago

@herlesupreeth

Thanks man~ Yes I'm going to have all the components cuddle up in the same docker environment for the first build. CUPS may be a further plan.

I was walking through the docker-compose.yaml and trying to slim it down by removing unnecessary functions (EPC & IMS) according to their dependence. It figures to be able to run an SA which remains simply AMF/AUSF/UDM/UDR/SMF/UPF/PCRF/NRF + MongoDB. So I move on tailoring the docker-compose.yaml, only to find that every function in 5gc is available to exist independently of EPC other than AMF.

image

Now then, buddy, does it matter if I cross the hss off from the list (at the same time without building it at all)?

herlesupreeth commented 3 years ago

I just now pushed couple of changes and added a docker-compose file to bring up just the SA components, please take the latest changes

and use docker-compose file as follows to bring up only the SA components

docker-compose -f sa-deploy.yaml up

ljysdfz commented 3 years ago

@herlesupreeth

JESUS !! YOU MUST BE A GRACIOUS GIFT ADDRESSED FROM GOD !! On my way building it !!!

ljysdfz commented 3 years ago

@herlesupreeth

Bro, in spite of the atrocious fact that nothing change in terms of the failure to register with NRF, I've got to drive home and have a rest. This project is literally wearing me out. Hope to make progress tomorrow. Grateful for your selfless favors. g'nite.

ljysdfz commented 3 years ago

@herlesupreeth

Woe is me !!

image

herlesupreeth commented 3 years ago

I updated the UERANSIM docker files to use the tag v2.2.1, please take the latest commit and re-compile UERANSIM docker image and give it a try

ljysdfz commented 3 years ago

@herlesupreeth

Good job, dude!! For more information, It's noticeable that the latest commitment involves wide-range updates:

image

Should I build the project from scratch, i.e. the base image ? or Is it acceptable to re-build just the UERANSIM image for convenience ?

herlesupreeth commented 3 years ago

Should I build the project from scratch, i.e. the base image ? or Is it acceptable to re-build just the UERANSIM image for convenience ?

Oops, you are right. You need to re-compile the base image of open5gs and then re-build just the open5gs components as follows

docker-compose -f nsa-deploy.yaml build --no-cache

ljysdfz commented 3 years ago

@herlesupreeth

Aha~ Got a typo, man You probably mean sa-deploy.yaml for just 5gs (:

ljysdfz commented 3 years ago

@herlesupreeth

The ueransim.yaml is configured as following.

image

Perhaps this srsenb is supposed to be ueransim ?

herlesupreeth commented 3 years ago

You are right. Good eye :) Will fix it. It shouldnt be a problem unless you are running srsenb and ueransim at the same time

ljysdfz commented 3 years ago

@herlesupreeth

Come on, man. Nothing good eye, just obsession. lol. Yeah, UERANSIM has been consumed in the native build. It complies with 5gs, which is gorgeous, while updating too fast.

ljysdfz commented 3 years ago

@herlesupreeth

Hi, bro, appreciate the update to tag2.2.1. The container of UERANSIM is able to run. Still need your help, now that I've run into another hassle.

image

As you can see, the DNN (aka APN), which is defined in the WebUI as "internet", cant be identified by the 5gc. Personally, I used to configure the APN for specified NFs like UPF and SMF when open5gs was deployed as a native binary.

/etc/open5gs/upf.yaml image

/etc/open5gs/smf.yaml image image

As a result, its expected to add a parameter that stands for APN/DNN in .env file, simultaneously replenishing the contents of smf/smf_init.sh and upf/upf_init.sh in order to reflect the APN/DNN in respective containers.

herlesupreeth commented 3 years ago

Check your WebUI of open5gs whether you have added that IMSI or not. Also, send me a pcap of the above process so that I can debug more

ljysdfz commented 3 years ago

@herlesupreeth

Hey, guy~ I've been through another tough day in which I've tried every possible patch to the configuration of AMF & SMF, annnnnnd...failed~ The attempts I've made includes:

smf/smf.yaml image

amf/amf.yaml image

Here is the ueransim.pcap captured as you expected. Following steps are what I've done in order to generate the pcap.

The UERANSIM terminal shows:

2021-01-08 20:10:12.358 [ERRO] [NIMPL] [ue-001011234567895] Registration rejected with unhandled MMCause: DNN not supported or not subscribed in the slice

The Open5gs terminal shows:

amf | 01/08 20:10:02.302: [gmm] WARNING: Registration request (../src/amf/gmm-sm.c:493) amf | 01/08 20:10:02.303: [app] ERROR: Cannot discover [AUSF] (../lib/sbi/path.c:51) amf | 01/08 20:10:02.303: [amf] WARNING: [suci-0-001-01-0000-0-0-1234567895] Registration reject [90] (../src/amf/nas-path.c:89)

According to the prompts, it seems the communication between AMF & AUSF collapsed in a way that resulted in the failure of UE registration.

HERLE, YOU ARE THE FINAL STRAW I CAN GRASP !!

herlesupreeth commented 3 years ago

Send me the screenshot of open5gs WebGUi where you have added this UE with IMSI

ljysdfz commented 3 years ago

@herlesupreeth

Beg your pardon, forgot it~

image

image

image

image

The data has been taken from .env file in which there is an exemplar referred to as "UE1".

herlesupreeth commented 3 years ago

All the UE details looks good. Btw, can you try changing the IP type from IPv4v6 to only IPv4 and give it a try (this shouldnt have caused that).

Following is the screenshot of UERANSIM working with the latest commit in docker_open5gs repo in my setup image

So I would suggest to take in all the latest changes, re-compile the base image of open5gs and images of SA open5gs components and then re-compile UERANSIM and give it a try

ljysdfz commented 3 years ago

@herlesupreeth

Nice play!! On my way!!

ljysdfz commented 3 years ago

@herlesupreeth

Hi, dude. The compilation process is meant to get stuck at such a spot.

image

BTW, this scene used to happen every time the UERANSIM was compiled. But sometimes it would get through soon. Do you have any idea about it ? Should I do anything to tackle it ?

herlesupreeth commented 3 years ago

BTW, this scene used to happen every time the UERANSIM was compiled. But sometimes it would get through soon. Do you have any idea about it ? Should I do anything to tackle it ?

Those are Warning not Errors so I hope you can ignore them

ljysdfz commented 3 years ago

image

In light of what the open5gs suggests above, it should be reasonable to check the open5gs/lib/sbi/path.c.

image

As far as we can tell, the crux of the failure has ever been derived from a null NRF pointer.

I believe I have to deal with the historical NRF registration warning in the first place.

herlesupreeth commented 3 years ago

I am not sure what's the issue with your setup but just pasting here screenshot with my setup

image

ljysdfz commented 3 years ago

@herlesupreeth

Dear Herle !! Fiiiiiiiiiiiiiinally figured it !! You must cannot imagine how it happened !! I've set up an HTTP/HTTPS PROXY for the base image in order to bypass the Great Fire Wall put up at the national egress toward INTERNET. After I'd demolished the PROXY, Open5gs turned out to work in a way that NRF can be easily recognized. Thank you very much. Deeply grateful for all the favors you've extended to me.

Now that the containers have been successfully run, could you kindly give me instructions on how to develop and debug the Open5gs. I wanna establish an IDE for it.

herlesupreeth commented 3 years ago

Now that the containers have been successfully run, could you kindly give me instructions on how to develop and debug the Open5gs. I wanna establish an IDE for it.

I am no expert in open5gs, just go through the C code and you will figure it out by cross-referencing the specifications. I dont use IDE so cant comment on that

ljysdfz commented 3 years ago

@herlesupreeth

OK, thanks. Gotta close this issue for now.

ljysdfz commented 3 years ago

@herlesupreeth

Buddy, I realized the PCF failed to start. No log appended to the log/pcf.log file.

image

herlesupreeth commented 3 years ago

thats fine, as PCF is not there in open5gs tag v2.0.20, thats the base image built upon

ljysdfz commented 3 years ago

@herlesupreeth Hi, Herle. I've caught up on the project, only to bump into a legacy hiccup -- Retry to registration with NRF image

image

The NRF is alive and sound. I've closed down all the VPN or PROXY or something, but the NRF is still inaccessible. The current Open5gs is v.2.3.6 image

NRF logs: image

Could you please give me a bit of directions or guidance?