search

srsran / srsRAN_Project

Open source O-RAN 5G CU/DU solution from Software Radio Systems (SRS) https://docs.srsran.com/projects/project
https://www.srsran.com
GNU Affero General Public License v3.0
525 stars 179 forks source link

X310 gNB consistently crashing after 2-3 minutes #862

Open sebastiangrosvik opened 1 month ago

sebastiangrosvik commented 1 month ago

Issue Description

I have an issue with my X310-based SRS gNB consistently crashing after 2-3 minutes of operation. The crash occurs both with 40 MHz and 80 MHz bandwidth. The 40 MHz config runs fine with the B210.

Setup Details

Host specs: Ubuntu 22.04.3 LTS Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 4x 8GiB DIMM DDR4 Synchronous 2133 MHz (0.5 ns) UHD version: UHD_4.7.0.HEAD-0-ga5ed1872

Device specific configuration: ru_sdr: device_driver: uhd device_args: type=x300,addr=192.168.40.2,product=X310,master_clock_rate=184.32e6,dboard_clck_rate=92.16e6,send_frame_size=8000,recv_frame_size=8000 clock: internal sync: internal srate: 46.08 freq_offset: 0 clock_ppm: 0 lo_offset: 0 tx_gain: 31.5 rx_gain: 20 time_alignment_calibration: auto

Actual Behaviour

The gNB runs fine during and after startup. Our UEs connect. Then after some time the radio link disconnects all of the sudden, UEs are not able to reconnect. The console trace stops printing.

Two example console traces: |--------------------DL---------------------|-------------------------UL------------------------------ pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr 1 4602 | 15 1.0 27 256k 248 0 0% 0 | -3.7 -4.8 0 3.9M 600 0 0% 300k 56n 21 1 4602 | 15 1.0 27 223k 216 0 0% 0 | -3.7 -4.9 0 3.9M 600 0 0% 300k 97n 21 1 4602 | 15 1.0 27 211k 205 0 0% 0 | -3.7 -4.9 0 3.9M 600 0 0% 300k 89n 21 1 4602 | 15 1.0 27 214k 208 0 0% 58 | -3.7 -4.8 0 3.9M 600 0 0% 300k 89n 21 1 4602 | 15 1.0 24 64k 68 33 32% 0 | -31.4 -30.7 0 656k 102 498 83% 300k 284n 21 1 4602 | n/a n/a 0 0 0 0 0% 0 | -35.6 -37.6 0 0 0 600 100% 300k -512n 21 1 4602 | n/a n/a 0 0 0 0 0% 0 | -37.1 -34.6 0 0 0 600 100% 300k 773n 21 1 4602 | n/a n/a 0 0 0 0 0% 0 | -36.9 -34.9 0 0 0 600 100% 300k -569n 21 ^CStopping ..

      |--------------------DL---------------------|-------------------------UL------------------------------

pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr 1 4602 | 15 1.0 26 268M 1395 5 0% 30.3k | 36.1 ovl 27 243k 50 0 0% 0 -146n 29 1 4602 | 15 1.0 26 265M 1393 7 0% 30.5k | 35.8 ovl 27 243k 50 0 0% 0 -187n 29 1 4602 | 15 1.0 26 267M 1394 6 0% 26.1k | 36.0 ovl 27 243k 50 0 0% 0 -170n 29 1 4602 | 15 1.0 26 268M 1395 5 0% 17.2k | 35.9 ovl 27 243k 50 0 0% 0 -195n 29 1 4602 | 15 1.0 26 267M 1392 8 0% 26.3k | 35.7 ovl 27 243k 50 0 0% 0 -179n 29 1 4602 | 15 1.0 26 267M 1393 7 0% 26.3k | 35.9 ovl 27 243k 50 0 0% 0 -170n 29 1 4602 | 15 1.0 25 225M 1177 215 15% 4.46M | 11.4 -5.1 20 204k 42 16 27% 0 -602n 29 1 4602 | n/a n/a 19 9.9M 74 1318 94% 5.64M | -58.1 -14.1 0 0 0 100 100% 0 553n 29 1 4602 | n/a n/a 19 10M 71 1327 94% 5.59M | -57.1 -14.4 0 0 0 15 100% 0 1u 29 1 4602 | 2 1.0 10 5.1M 64 1317 95% 5.66M | -59.1 -12.9 0 0 0 10 100% 0 -537n 29 1 4602 | n/a n/a 0 504k 78 1309 94% 5.66M | -57.4 -13.5 0 0 0 15 100% 0 919n 29 ^CStopping ..

Additional Information

Ran the SRSran performance script. No change.

pgawlowicz commented 1 month ago

could you send your gnb logs?

sebastiangrosvik commented 1 month ago

gnb_x310.log

pgawlowicz commented 1 month ago

what about CPU utilization? could you first test with smaller BW like 20mhz?

sebastiangrosvik commented 1 month ago

Did a rough CPU comparison of the two SDRs at 40 MHz for 100 seconds using top: X310: CPU usage is stable at 139%, mem 7.3% B210: CPU usage is stable at 170%, mem 7.3%

pgawlowicz commented 1 month ago

does it work correctly with b210?

sebastiangrosvik commented 1 month ago

Yep, the b210 is stable. Never had this issue with it.

pgawlowicz commented 1 month ago

Could you pull the newest code base and try again?

sebastiangrosvik commented 1 month ago

Looks like the new version has solved the problem for us. Tested the newest code base with the X310 at 40 MHz bandwidth: Performance is stable, no crashing. At 80 MHz our channel quality is not stable, but at least we are not experiencing any crashing.

pgawlowicz commented 1 month ago

Probably, you will need to optimize the CPU affinities and number of threads. Please see expert_execution in gnb config: https://docs.srsran.com/projects/project/en/latest/user_manuals/source/config_ref.html

pgawlowicz commented 3 weeks ago

@sebastiangrosvik did you try tuning the CPU affinities and number of threads?