srsgnb fails to start

[dlake02]

Hello

I have the dockerized VoLTE and Open5Gs system. srsenb and oaignb run fine but when I try and start srsgnb I get this...

root@oai-gnb:/home/dlake/docker_open5gs# docker-compose -f srsgnb.yaml up && docker attach srsgnb WARNING: Found orphan containers (oaienb, oaignb) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. Recreating srsgnb ... done Attaching to srsgnb srsgnb | Configuring component: 'gnb' srsgnb | * Starting system message bus dbus [ OK ] srsgnb | avahi-daemon: unrecognized service srsgnb | Lower PHY in dual executor mode. srsgnb | srsgnb | --== srsRAN gNB (commit e043e1e40) ==-- srsgnb | srsgnb | Failed to bind SCTP socket to 172.22.0.37:0. Success srsgnb | srsGNB ERROR: Failed to create SCTP gateway. srsgnb | srsgnb | /mnt/srsran/srsran_init.sh: line 53: 37 Aborted (core dumped) gnb -c /etc/srsran/gnb.yml srsgnb exited with code 134

I've verified that the SCTP port on the AMF is available (38412 - oaignb connects to this just fine) so I think the issue is in the srsgnb.

I'd really welcome some help as I'm stumped...

Thanks!

David

[herlesupreeth]

I see the following in your log

Found orphan containers (oaienb, oaignb)

I would suggest to run docker system prune -f before attempting to start srsgnb

[dlake02]

I tried that and the same problem. root@oai-gnb:/home/dlake/docker_open5gs# docker-compose -f srsgnb.yaml up && docker attach srsgnb Starting srsgnb ... done Attaching to srsgnb srsgnb | Configuring component: 'gnb' srsgnb | Removing stale PID file /var/run/dbus/pid. srsgnb | Starting system message bus dbus [ OK ] srsgnb | avahi-daemon: unrecognized service srsgnb | Lower PHY in dual executor mode. srsgnb | srsgnb | --== srsRAN gNB (commit e043e1e40) ==-- srsgnb | srsgnb | Failed to bind SCTP socket to 172.22.0.37:0. Success srsgnb | srsGNB ERROR: Failed to create SCTP gateway. srsgnb | srsgnb | /mnt/srsran/srsran_init.sh: line 53: 39 Aborted (core dumped) gnb -c /etc/srsran/gnb.yml srsgnb exited with code 134

[herlesupreeth]

Please first run docker system prune -f and then can you please share the output of below command before running srsgnb ??

docker ps -a

can you also describe your setup? is srsgnb running in the same machine as core etc..?

[dlake02]

root@oai-gnb:/home/dlake/docker_open5gs# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES root@oai-gnb:/home/dlake/docker_open5gs#

root@oai-gnb:/home/dlake/docker_open5gs# docker compose -f srsgnb.yaml up -d && docker attach srsgnb WARN[0000] network default: network.external.name is deprecated. Please set network.name with external: true WARN[0000] Found orphan containers ([mme sgwu sgwc upf pcscf smf icscf scscf amf fhoss smsc osmomsc bsf pcf ausf udm udr nssf webui hss pcrf rtpengine osmohlr metrics mysql scp mongo nrf dns]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. [+] Running 1/1 ✔ Container srsgnb Started 0.3s

• Starting system message bus dbus [ OK ] avahi-daemon: unrecognized service Lower PHY in dual executor mode.

--== srsRAN gNB (commit e043e1e40) ==--

Failed to bind SCTP socket to 172.22.0.37:0. Success srsGNB ERROR: Failed to create SCTP gateway.

/mnt/srsran/srsran_init.sh: line 53: 37 Aborted (core dumped) gnb -c /etc/srsran/gnb.yml root@oai-gnb:/home/dlake/docker_open5gs#

I also loaded the latest gnb from srsRAN_Project and modified the yaml file to remove the 5gs and put the network into the same namespace and see the exact same issue.

I am running all the elements on the same machine. srsenb works perfectly as does oaignb.

[herlesupreeth]

I am not sure what modification you made, can you please paste the modified configuration file??

I just now ran srsRAN_Project gNB and its works just fine

WARN[0000] network default: network.external.name is deprecated. Please set network.name with external: true 
WARN[0000] Found orphan containers ([upf smf amf udr bsf nssf webui pcf udm ausf mongo scp nrf metrics]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 1/1
 ✔ Container srsgnb  Started                                                                                                                                                        0.4s 
 * Starting system message bus dbus                                                                                                                                               [ OK ] 
avahi-daemon: unrecognized service
Lower PHY in quad executor mode.

--== srsRAN gNB (commit e043e1e40) ==--

Connecting to AMF on 172.22.0.10:38412
Available radio types: uhd and zmq.
[INFO] [UHD] linux; GNU C++ version 11.3.0; Boost_107400; UHD_4.4.0.0-0ubuntu1~jammy1
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Making USRP object with args 'type=b200,num_recv_frames=64,num_send_frames=64'
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz... 
[INFO] [B200] Actually got clock rate 16.000000 MHz.
[INFO] [MULTI_USRP] Setting master clock rate selection to 'manual'.
[INFO] [B200] Asking for clock rate 23.040000 MHz... 
[INFO] [B200] Actually got clock rate 23.040000 MHz.
Cell pci=1, bw=20 MHz, dl_arfcn=632628 (n78), dl_freq=3489.42 MHz, dl_ssb_arfcn=632544, ul_freq=3489.42 MHz

==== gNodeB started ===
Type <t> to view trace

[dlake02]

My docker-compose.yaml for srsgnb:

version: '3.9' services:

gnb: container_name: srsran_gnb

Build info

image: srsran/gnb
build:
  context: ..
  dockerfile: docker/Dockerfile
  args:
    OS_VERSION: 22.04
# privileged mode is requred only for accessing usb devices
privileged: true
# Extra capabilities always required
cap_add:
  - SYS_NICE
  - CAP_SYS_PTRACE
volumes:
  # Access USB to use some SDRs
  - /dev/bus/usb/:/dev/bus/usb/
  # Sharing images between the host and the pod.
  # It's also possible to download the images inside the pod
  - /usr/share/uhd/images:/usr/share/uhd/images
# It creates a file/folder into /config_name inside the container
# Its content would be the value of the file used to create the config
configs:
  - gnb_config.yml
# Customize your desired network mode.
# current netowrk configuration creastes a private netwoek with both containers attached
# An alterantive would be `network: host"`. That would expose your host network into the container. It's the easiest to use if the 5gc is not in your PC
networks:
  default:
    ipv4_address: 172.22.0.37
# Command to run into the final container
# command: gnb log  --help
command: gnb -c /gnb_config.yml amf --addr 172.22.0.10 --bind_addr 172.22.0.37

configs: gnb_config.yml: file: ${GNB_CONFIG_PATH:-../configs/gnb_rf_b200_tdd_n78_20mhz.yml} # Path to your desired config file

networks: default: external: name: docker_open5gs_default

[herlesupreeth]

I see that you are using the docker compose from the srsRAN_Project repository. I would suggest to use run the below command from srsran folder in this repository (its uses the same srsRAN_Project repository based gNB)

docker build --no-cache --force-rm -t docker_srsran .

[dlake02]

Same problem:

Starting srsgnb ... done Lower PHY in dual executor mode.

--== srsRAN gNB (commit e043e1e40) ==--

Failed to bind SCTP socket to 172.22.0.37:0. Success srsGNB ERROR: Failed to create SCTP gateway.

/mnt/srsran/srsran_init.sh

Base system is:

root@oai-gnb:/home/dlake/docker_open5gs# uname -a Linux oai-gnb 5.4.0-155-lowlatency #172-Ubuntu SMP PREEMPT Fri Jul 7 16:52:35 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux root@oai-gnb:/home/dlake/docker_open5gs# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal

root@oai-gnb:/home/dlake/docker_open5gs# docker -v Docker version 24.0.5, build ced0996

[herlesupreeth]

Are you not running the 5GC before running gNB?? I find this error strange given that srsenb and oaignb is able to connect to Core Network properly on the same machine

[dlake02]

Yes - I don't understand it at-all because even if I change the SRS_GNB_IP address to the same that the SRS_ENB is running, srsgnb fails.

[dlake02]

I've found a little more detail in the gnb.log:

2023-08-07T09:23:37.231648 [GNB ] [W] Could not check scaling governor. filename=/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor error="No such file or directory" 2023-08-07T09:23:37.231669 [GNB ] [W] DRM KMS polling is enabled, which may hinder performance. You can disable it using the "srsran_performance" script 2023-08-07T09:23:37.234107 [SCTP-GW ] [E] Subscribing to SCTP events failed: Invalid argument 2023-08-07T09:23:37.234107 [SCTP-GW ] [E] Couldn't subscribe to SCTP events 2023-08-07T09:23:37.234136 [SCTP-GW ] [E] Failed to bind SCTP socket to 172.22.0.37:0. Success 2023-08-07T09:23:37.234136 [SCTP-GW ] [E] Couldn't bind to address 172.22.0.37:0

[herlesupreeth]

What is the host machine you are running on? Can you give more details?? CPU ?? and can you switch to using non-lowlatency kernel?

[dlake02]

I was wondering if the kernel was a problem - the point in the code that is complaining is the setsockoption and that is typically kernel related. I am just upgrading to 22.04 and reverting to generic kernel and will report back in the next 24 hours.

[dlake02]

Problem solved... I had VirtualBox still installed on the system. I've now changed to Ubuntu 22.04 with latest generic kernel and srsenb and srsgnb both working perfectly. I'm now configuring VoLTE.

Thank you for a wonderful project.

[pelopor23]

Hi @dlake02,

Could you please inform what version of generic kernel that you install, I'm facing the same problem, previously I'm using Ubuntu 20 and now I have switch to Ubuntu 22 and still facing the same problem.

This is the output at the terminal: WARN[0000] network default: network.external.name is deprecated. Please set network.name with external: true WARN[0000] Found orphan containers ([upf smf amf srsenb pcscf scscf icscf pyhss smsc mme sgwu sgwc osmomsc mysql hss pcrf osmohlr rtpengine dns udm udr pcf bsf ausf nssf webui metrics mongo scp nrf]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. [+] Running 1/1 ✔ Container srsgnb Started 0.4s

• Removing stale PID file /var/run/dbus/pid.
• Starting system message bus dbus [ OK ] avahi-daemon: unrecognized service Lower PHY in dual executor mode.

--== srsRAN gNB (commit 1afd7240f) ==--

Connecting to AMF on 172.22.0.10:38412 Failed to connect SCTP socket to 172.22.0.10:38412. error="Connection timed out" timeout=1384ms srsGNB ERROR: Failed to create SCTP gateway.

/mnt/srsran/srsran_init.sh: line 53: 40 Aborted (core dumped) gnb -c /etc/srsran/gnb.yml

I have already run 5G core before execute the gnb command.

And this is my kernel version: Linux opencell 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

@herlesupreeth could you please give some advise?

Thanks

[herlesupreeth]

@pelopor23

I would suggest to run docker system prune -f before attempting to start srsgnb

[pelopor23]

@herlesupreeth

I have already run docker system prune -f, but the problem still appears.

[herlesupreeth]

Increase the log level of gNB to debug and send me the logs

[pelopor23]

Could you please inform how to increase the log level?

[herlesupreeth]

In the srsran folder, goto gnb.yaml and change as follows

log:
  filename: /mnt/srsran/gnb.log                                   # Path of the log file.
  all_level: debug                                              # Logging level applied to all layers.

[pelopor23]

Here is the full log:

2023-08-18T01:20:33.911239 [GNB ] [I] Built in Release mode using commit 1afd7240f on branch main 2023-08-18T01:20:33.911243 [GNB ] [D] Required CPU features: sse4.1(ok) avx(ok) avx2(ok) fma(ok) pclmul(ok) 2023-08-18T01:20:33.911333 [GNB ] [W] CPU0 scaling governor is not set to performance, which may hinder performance. You can set it to performance using the "srsran_performance" script 2023-08-18T01:20:33.911386 [GNB ] [W] CPU1 scaling governor is not set to performance, which may hinder performance. You can set it to performance using the "srsran_performance" script 2023-08-18T01:20:33.911411 [GNB ] [W] CPU2 scaling governor is not set to performance, which may hinder performance. You can set it to performance using the "srsran_performance" script 2023-08-18T01:20:33.911432 [GNB ] [W] CPU3 scaling governor is not set to performance, which may hinder performance. You can set it to performance using the "srsran_performance" script 2023-08-18T01:20:33.911457 [GNB ] [W] DRM KMS polling is enabled, which may hinder performance. You can disable it using the "srsran_performance" script 2023-08-18T01:20:33.913078 [GNB ] [I] Connecting to AMF (172.22.0.10).. 2023-08-18T01:20:33.913138 [SCTP-GW ] [D] Setting RTO_INFO options on SCTP socket. Association 0, Initial RTO 120, Minimum RTO 120, Maximum RTO 500 2023-08-18T01:20:33.913142 [SCTP-GW ] [D] Setting SCTP_INITMSG options on SCTP socket. Max attempts 3, Max init attempts timeout 500 2023-08-18T01:20:33.913157 [SCTP-GW ] [D] Binding to 172.22.0.37 port 0 2023-08-18T01:20:33.913167 [SCTP-GW ] [D] Binding successful 2023-08-18T01:20:33.913199 [SCTP-GW ] [D] Setting RTO_INFO options on SCTP socket. Association 0, Initial RTO 120, Minimum RTO 120, Maximum RTO 500 2023-08-18T01:20:33.913202 [SCTP-GW ] [D] Setting SCTP_INITMSG options on SCTP socket. Max attempts 3, Max init attempts timeout 500 2023-08-18T01:20:33.913207 [SCTP-GW ] [D] Connecting to 172.22.0.10 port 38412 2023-08-18T01:20:35.296020 [SCTP-GW ] [D] Failed to connect to 172.22.0.10:38412 - Connection timed out 2023-08-18T01:20:35.296021 [SCTP-GW ] [D] Closing socket. 2023-08-18T01:20:35.296085 [SCTP-GW ] [E] Failed to connect SCTP socket to 172.22.0.10:38412. error="Connection timed out" timeout=1382ms

Please advise

[herlesupreeth]

Can you post your docker-compose file you are using to deploy the 5G Core??

[pelopor23]

Please check the following file:

sa-deploy.zip

[herlesupreeth]

Are you running gNB and 5GC in the same machine??

[pelopor23]

Yes, i'm running gnb and 5gc in the same machine

[herlesupreeth]

hmm..then I am really out of clue where the issue. From the logs you posted above, I dont see any issue with SCTP (not the kernel issue which the author of issue was facing). Dont know why it doesnt see the AMF

[pelopor23]

Hi,

I just want to inform that I have finally succeeded to start the gNB, after installing it on a different PC. I guess it's related to hardware issue.

[herlesupreeth]

Thank you for letting me know.