search

srsran / srsRAN_Project

Open source O-RAN 5G CU/DU solution from Software Radio Systems (SRS) https://docs.srsran.com/projects/project
https://www.srsran.com
GNU Affero General Public License v3.0
513 stars 173 forks source link

srsUE cannot connect to gNb #732

Open CrashxZ opened 3 months ago

CrashxZ commented 3 months ago

Issue Description

I followed this tutorial - https://docs.srsran.com/projects/project/en/latest/tutorials/source/srsUE/source/index.html

I am trying to use the over-the-air setup to establish connection between the srsUE and gNb, but the UE gets stuck after attaching UE, with the following message - "RF status: O=10, U=0, L=0". There are overflows and underflows on the gNb side too.

My gNb and UE start and here are the screenshots (will also attach log files) image image

Setup Details

PC1 - gNb and Open5GC Lenovo Legion 5 (AMD Ryzen 5 4600h + Nvidia GTX 1660 + 8GB RAM) Ubuntu - 22.04 UHD Version - UHD_4.1.0.5-3 Radio - B210

Here is my configuration file - gnb.yaml.txt

PC2 - srsUE Raspberry Pi4 8GB varient. Ubuntu - 22.04 UHD Version - UHD_4.1.0.5-3 Radio - B210

Here is my configuration file - ue.conf.txt

I have ensured that both setups have the cpu's set to performance and have also run the performance script.

Additional Information

I ran the usrp benchmarks scripts on both the pcs with the selected frequencies and had no issues there.

Here are my log files - amf.log.txt gnb.log.txt ue.log.txt

pgawlowicz commented 3 months ago

gnb log says:

2024-07-15T03:18:27.564308 [GNB     ] [W] CPU0 scaling governor is not set to performance, which may hinder performance. You can set it to performance using the "srsran_performance" script
pgawlowicz commented 3 months ago

also RPi5 might be too slow for srsUE, you could try lower BW maybe it helps.

CrashxZ commented 3 months ago

Sorry my bad, I reran the performance script and also this command - "sudo cpupower frequency-set --governor performance" , and ensured that the CPUs are set to performance mode. I also lowered the channel bandwidth on the gNb side to 5mhz (i don't know how to explicitly set it on the srsUE side). The behaviour is still the same, here are the new logs - ue.log.txt gnb.log.txt

I will also run the UE on the Nvidia Jetson TX2, since it might be more powerful, but is there any way to run the UE on the RPi 4 ?

CrashxZ commented 3 months ago

Installed srsran4g on the Jetson TX2 and tweaked the config a bit but still i cant establish connection. (also rerunning the performance script helped a bit, gNb overflows and underflows are less frequent) Note that the tx2 has ubuntu 18.04 and UHD 3.15. Here are the screenshots -

UE (TX2) - image

gNb - image

Here are the config files - gnb.yaml.txt mainak_ue.conf.txt

and here are the logs -

ue.log.txt gnb.log.txt

pgawlowicz commented 3 months ago

could you use srate: 30.72 in gnb config?

CrashxZ commented 3 months ago

Still the same - here are the logs. (also my Samsung S23 ultra cannot see the network when I do a network search, I understand it cant connect (without a programmed sim and a different config), but it should be able to see it right ?)

Logs - ue.log.txt gnb.log.txt

CrashxZ commented 3 months ago

I also tried running the UE from another x86 system (Core i7 11th gen and 16gb of RAM), just to check if its a performance issue. But it was the same response.

Then I thought it would be a Tx/Rx gain tuning issue, so i connected the 2 B210s using SMA cables, and tested it again both x86 machines. This time the out of sync message wasn't present, but instead it showed - "PBCH-MIB: CRC failed" in the UE logs.

here are the logs-

ue.log.txt gnb.log.txt

pgawlowicz commented 3 months ago

with 5MHz cell, could you use srate: 5.76 in gnb conifg? could you also increase log level to debug in the gnb config?

CrashxZ commented 3 months ago

@pgawlowicz , thanks a lot for taking the time to help me out - Here are the logs with the suggested changes. I still cant establish connection with the b210s connected via sma cables (on x86 machines)

ue.log.txt gnb.log.txt

pgawlowicz commented 3 months ago

please also change srate: 5.76 in ue config. Seems it is too slow to process signal samples with 23.04MHz

CrashxZ commented 3 months ago

That makes sense, I changed both srate to 5.76 , now it shows RRC connected, but then it is released right after -

image

The ip address is not assigned, there is nothing on the amf log too. Also should there be a network adapter created on the UE machine? where the ipaddress is assigned ? because i dont see any adapter that was created, when i use the command ifconfig.

here are the logs - gnb.log.txt ue.log.txt

I had to delete some lines from the ue log , it was 89 mb, hopefully, this will be or i'll provide a google drive link.

pgawlowicz commented 3 months ago

if ue is not getting an IP assigned, an interface will not be created. Could you try to apply this patch and send logs again? 0002-nr-rrc-fix-apply-t_offset-only-once-after-SIB1.patch.txt

CrashxZ commented 3 months ago

Ohh, I understand, here are the new logs with the patch -

gnb.log.txt ue.log.txt

pgawlowicz commented 3 months ago

Hi, are you sure the patch is applied correctly, srsUE commit is the same in both cases (i.e., commit ec29b0c1f) but maybe because the changes from patch are not commited.

CrashxZ commented 3 months ago

Hi I did apply it locally, but now I forked it and applied the patch (ensured the changes) and here is the new log file - gnb.log.txt ue.log.txt

pgawlowicz commented 3 months ago

hmm, PUSCH SNR is very small:

2024-07-18T22:36:24.696728 [PHY     ] [D] [  1010.6] PUSCH: rnti=0x4601 h_id=0 prb=[3, 6) symb=[0, 14) mod=QPSK rv=0 tbs=11 crc=KO iter=6.0 sinr=-30.3dB t=306.8us uci_t=0.0us ret_t=0.0us
2024-07-18T22:36:24.706472 [PHY     ] [D] [  1011.6] PUSCH: rnti=0x4601 h_id=0 prb=[3, 6) symb=[0, 14) mod=QPSK rv=0 tbs=11 crc=KO iter=6.0 sinr=-30.7dB t=148.4us uci_t=0.0us ret_t=0.0u

could you try to tune tx/rx gains on both gnb and ue? you could try the following config in both:

  tx_gain: 60
  rx_gain: 30
CrashxZ commented 3 months ago

Thanks for that , I tuned the tx/rx gains for both the ue and the gnb (tried a couple of configs). Whats seems to be working is -

[gnb] tx_gain: 80 rx_gain:50 [ue] tx_gain: 50 rx_gain:70

Now the UE connects for a file and then it fails the scheduling request - "Scheduling request failed: releasing RRC connection..."

The AMF log aslo shows that it recieves the request. Here are the logs -

amf.log.txt gnb.log.txt ue.log.txt

do i have to adjust the time_adv_nsamples on the ue side? If yes can you tell me how to calculate it correctly ?

CrashxZ commented 3 months ago

Update : I saw that there was thread where you suggested another user to add the inactivity timer in their gnb config- cu_cp: inactivity_timer: 7200

So I added it, and still the RRC is released.

pgawlowicz commented 3 months ago

PDSCH at UE seems to be more or less OK, but PUSCH at gnb has a huge cfo. Do you have a possibility to connect your b200s to a better clock source?

CrashxZ commented 3 months ago

I don't have access to a better clock source like a GPSDO, at the moment. Is there anything that I can tune to make the connection more stable to get a IP address ( and make it last for a few mins ) ?

Because if I do that , I can reasonably convince my lab to buy the external clocks.

CrashxZ commented 3 months ago

I managed to connect for a second, using different B210s. But it disconnected right after.

image
Erick5G commented 2 months ago

Hello, How can I add this patch to the SrsRan Project folder? I have the same error with X310 as gnb and a b210 as UE and I do not have an assigned IP or an interface created. I already tried downloading the BW, srate in both gnb and ue scripts, and applying the performance. Captura desde 2024-08-01 19-47-09

CrashxZ commented 2 months ago

Fork the code and use the git apply command.

Something like -> git apply name-of-file.patch

Or use a vs code extension, which does it for you. ( Its pretty simple)

Let me know if it works for you ? Are you using external clocks ?

It would be a great help if you can tell me about your setup and findings .

CrashxZ commented 2 months ago

@pgawlowicz Hey, Luckily we have ordered 2 units of the GPSDO for using as external reference clocks.

While it arrives, I wanted to get a more stable connection, but i left it running for a few hours, and now it wont connect at all - Can you please suggest what happened? and how to fix that ?

The UE log -

2024-08-03T18:12:13.715850 [UE ] [I] Using binary srsue with arguments: mainak_ue.conf 2024-08-03T18:12:13.744771 [UE ] [I] Built in Release mode using commit 4e49269f7 on branch CPS_LAB. 2024-08-03T18:12:13.780048 [RF ] [I] Configuring physical DL channel 0 with band-pass filter (0.0, 0.0) 2024-08-03T18:12:13.780049 [RF ] [I] Configuring physical UL channel 0 with band-pass filter (0.0, 0.0) 2024-08-03T18:12:15.302913 [STCK ] [E] Unknown PCAP option true 2024-08-03T18:12:15.305275 [PHY-SA ] [D] [ 0] SYNC: state=IDLE, tti=0 2024-08-03T18:12:15.305277 [PHY-SA ] [D] [ 0] Discarding samples and sending tx_end 2024-08-03T18:12:15.310992 [MAC ] [I] [ 0] Resetting MAC 2024-08-03T18:12:15.310996 [MAC ] [D] [ 0] Msg3 buffer flushed 2024-08-03T18:12:15.310998 [MAC ] [I] [ 0] Logical Channel Setup: LCID=0, LCG=0, priority=0, PBR=-1, BSD=50ms, bucket_size=0 2024-08-03T18:12:15.311001 [MAC ] [I] [ 0] Logical Channel Setup: LCID=1, LCG=0, priority=1, PBR=-1, BSD=50ms, bucket_size=0 2024-08-03T18:12:15.311012 [MAC ] [D] [ 0] After MAC reset:

CrashxZ commented 2 months ago

I am not even receiving any underflow/overflow messages on the UE side.

frankist commented 2 months ago

Isn't there a manual command on the UE side to set a starting frequency offset (I believe it is rf.freq_offset)? If the difference between gnb and ue is too high, the UE won't be able to compensate it

Erick5G commented 2 months ago

Hi @CrashxZ @pgawlowicz I have already made the changes of the new patch, but still no IP address is assigned. Now I get the following errors: 2024-08-05T19:51:57.158745 [RRC-NR ] [I] Proc "Cell Selection" - Completed with failure. 2024-08-05T19:51:57.158751 [RRC-NR ] [W] Could not finish setup request. Deallocating dedicatedInfoNAS PDU Do you know how can I fix them?

Attached logs: gnb_20M.log ue.log

@CrashxZ Yes, I'm using GPSDO for both radios X310 as GNB and B210 as UE. But I have also performed tests with the internal clock of both radios, and I have the same error result with and without it.

Late: 0; Underflow: 1; Overflow: 0;

pgawlowicz commented 2 months ago

@Erick5G could you share your configs?

Erick5G commented 2 months ago

Sure, @pgawlowicz

I've tested band n3 and n78 with this configuration for GNB and it's worked perfectly for Cost UE. Also with internal clock or gpsdo got the same result. But no sucess for SrsUE with B210.

Attached config: ue_rf.conf.txt gnb_x310_20MHz_UECOST.yml.txt

Erick5G commented 2 months ago

I've tried new config for GNB, adding the following lines: pdcch: dedicated: ss2_type: common
dci_format_0_1_and_1_1: false
common: ss0_index: 0
coreset0_index: 12
prach: prach_config_index: 1

Found Issues:

  1. [PHY-SA ] [E] [ 0] prach_cfg.freq_offset=1 is not compatible with LTE
  2. PDCCH DMRS Rx: Measure L=2; ncce=4; RSRP=-30.2 dBfs; EPRE=-5.7 dBfs; Corr=0.004 Discarded PDCCH candidate L=2;ncce=4; Correlation is too low (0.0<0.5); EPRE=-5.68; RSRP=-30.17;
  3. [PHY-SA ] [E] [ 6360] SYNC: detected out-of-sync... skipping slot ...
  4. [PHY0-NR] [E] [ 6380] PBCH-MIB: NR SFN (639) does not match current SFN (638)
  5. [PHY0-NR] [W] [ 6440] PBCH-MIB: CRC failed

Ubuntu Terminal for the UE Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0x39, tti=3694 Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0x39, tti=3854 Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0x39, tti=4014 Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0x39, tti=4174

Attached Conf & logs: gnb_x310_20MHz_SRSUE.yml.txt ue.conf.txt ue.log

As additional information: srsRan_Project is being used for the GNB and srsRan_4G for the UE. I hope this information is of some use. Regards

pgawlowicz commented 2 months ago

could you try to use this config with gnb running on x310

ru_sdr:
  device_driver: uhd                  
  device_args: type=x300,addr=X.X.X.X,master_clock_rate=184.32e6,send_frame_size=8000,recv_frame_size=8000        
  clock: gpsdo                        
  sync: gpsdo                         
  srate: 30.72                        
  tx_gain: 35                         
  rx_gain: 60                         

cell_cfg:
  dl_arfcn: 368640                    
  band: 3                             
  channel_bandwidth_MHz: 20           
  common_scs: 15                      
  plmn: "00101"                       
  tac: 7                              
  pci: 1
CrashxZ commented 2 months ago

Hey @pgawlowicz , I have a question , do you think I messed something up by letting it run for hours ? Because I don't see rf underflow /overflow messages anymore .

It gets stuck at attaching UE.

CrashxZ commented 2 months ago

How much should I set that value to ?

Isn't there a manual command on the UE side to set a starting frequency offset (I believe it is rf.freq_offset)? If the difference between gnb and ue is too high, the UE won't be able to compensate it

Erick5G commented 2 months ago

Hi @pgawlowicz

could you try to use this config with gnb running on x310

ru_sdr:
  device_driver: uhd                  
  device_args: type=x300,addr=X.X.X.X,master_clock_rate=184.32e6,send_frame_size=8000,recv_frame_size=8000        
  clock: gpsdo                        
  sync: gpsdo                         
  srate: 30.72                        
  tx_gain: 35                         
  rx_gain: 60                         

cell_cfg:
  dl_arfcn: 368640                    
  band: 3                             
  channel_bandwidth_MHz: 20           
  common_scs: 15                      
  plmn: "00101"                       
  tac: 7                              
  pci: 1

Hi @pgawlowicz With this config for X310, maximum values accepted for this radios are 31.5 & 37.5. Terminal UE Error: setting gain for transmitter 0. Tx gain (i.e., 35.0 dB) is out-of-range. Range is [0.0, 31.5] dB in steps of 0.5 dB. Error: setting gain for receiver 0. Rx gain (i.e., 60.0 dB) is out-of-range. Range is [0.0, 37.5] dB in steps of 0.5 dB. srsGNB ERROR: Unable to create radio session.

Applying the changes did not produce any improvement, only PDU errors.

gnb_x310_20MHz_UECOST.yml.txt uePLMNtac change.log gnb_20M.log

pgawlowicz commented 2 months ago

seems there are some issues on the gnb side as well:

2024-08-07T20:55:30.002583 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:55:34.002587 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:55:46.002601 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:55:53.002523 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:55:58.002561 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:56:22.002599 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:56:25.002658 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:56:30.002634 [RF      ] [W] Real-time failure in RF: underflow
2024-08-07T20:56:34.002560 [RF      ] [W] Real-time failure in RF: underflow

could you try to run this script on both PC? https://github.com/srsran/srsRAN_Project/blob/main/scripts/srsran_performance

Erick5G commented 2 months ago

Done! We still receive this info on terminal.

Late: 0; Underflow: 1; Overflow: 0; And the same issues of PDU.

pgawlowicz commented 2 months ago

could you try with uhd-4.3?

CrashxZ commented 2 months ago

I am using UHD 4.3 with a core i7 11900k for one of the UEs , I still get the underflow/late messages.

pgawlowicz commented 2 months ago

hmm, that is strange. Could you try with a smaller BW, like 10MHz and post your configs ang logs here?

pgawlowicz commented 1 week ago

@CrashxZ any updates on this issue?

CrashxZ commented 1 week ago

Hi @pgawlowicz ! We got 2 units of GPSDOs and 2 active GPS antennas for the B210s. I tried connecting it using them by setting clock=gpsdo and sync=gpsdo, but it didn't help. I have similar connection issues. It connects intermittently for a few seconds and disconnects (if it even connects). Since we have to use this as an indoor solution, I'm not sure on what to do .

I'll post the logs and the config files shortly .

pgawlowicz commented 1 week ago

could you try the patch from this comment https://github.com/srsran/srsRAN_Project/issues/826#issuecomment-2370588502

CrashxZ commented 6 days ago

Here are the logs for the indoor setup. Unfortunately, i cannot use clock=gpsdo and sync=gpsdo on the gNb since, its needs to be outdoors, for a gps signal. But after the from the comment #826 the connection is established every time (unlike intermittently like before). but only for a few times.

gnb.log.txt ue.log.txt

CrashxZ commented 6 days ago

I plan to test outdoors soon, but we need this for an indoor application, so I have to come up with a work around.

CrashxZ commented 5 days ago

also i found this the the open5gc log - image

10/24 16:46:53.468: [amf] INFO: UE Context Release [Action:2] (../src/amf/ngap-handler.c:1698)

pgawlowicz commented 4 days ago

there are some link issues and gnb sends a UE release request to the core:

2024-10-23T18:19:36.822680 [DU-MNG  ] [I] ue=0: RLF timer expired with cause="RLC max ReTxs reached". Requesting a UE release...
2024-10-23T18:19:36.822772 [DU-F1   ] [I] Tx PDU du=0 ue=0 du_ue=0: UEContextReleaseRequest

Please pull the newest srsRAN-Project version and try again.

CrashxZ commented 1 day ago

Hi, I pulled the newest srsRAN-Project, commit - 9d5dd742a. Updated to the latest UHD Drivers (4.7) and still running into the same issue - 

2024-10-28T23:00:39.845798 [DU-MNG  ] [I] ue=0: RLF timer expired with cause="RLC max ReTxs reached". Requesting a UE release...
2024-10-28T23:00:39.845802 [PHY     ] [D] [   421.7] PDCCH: rnti=0x4601 ss_id=2 format=1_0 cce=0 al=2 t=9.9us

I have also noticed that it says UE Context Release [Action:2] in the open5gc logs (There was once i saw it was action 1) - 

open5gs_5gc  | 10/29 00:00:38.433: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:401)
open5gs_5gc  | 10/29 00:00:38.433: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2550)
open5gs_5gc  | 10/29 00:00:38.433: [amf] INFO:     RAN_UE_NGAP_ID[0] AMF_UE_NGAP_ID[1] TAC[7] CellID[0x66c000] (../src/amf/ngap-handler.c:562)
open5gs_5gc  | 10/29 00:00:38.433: [amf] INFO: [suci-0-001-01-0000-0-0-0123456780] Unknown UE by SUCI (../src/amf/context.c:1835)
open5gs_5gc  | 10/29 00:00:38.433: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1616)
open5gs_5gc  | 10/29 00:00:38.433: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:1165)
open5gs_5gc  | 10/29 00:00:38.433: [gmm] INFO: [suci-0-001-01-0000-0-0-0123456780]    SUCI (../src/amf/gmm-handler.c:166)
open5gs_5gc  | 10/29 00:00:39.999: [amf] INFO: UE Context Release [Action:2] (../src/amf/ngap-handler.c:1698)
open5gs_5gc  | 10/29 00:00:39.999: [amf] INFO:     RAN_UE_NGAP_ID[0] AMF_UE_NGAP_ID[1] (../src/amf/ngap-handler.c:1699)
open5gs_5gc  | 10/29 00:00:39.999: [amf] INFO:     SUCI[suci-0-001-01-0000-0-0-0123456780] (../src/amf/ngap-handler.c:1702)
open5gs_5gc  | 10/29 00:00:39.999: [amf] INFO: [Removed] Number of gNB-UEs is now 0 (../src/amf/context.c:2557)

I believe that the TX and RX gains are good with the observed sinr, so can it be an authentication issue from the open5gc? I am using the dockerised version provided in the repository.

Here are the logs and configs -

gnb.yaml.txt mainak_ue.conf.txt ue.log3.txt gnb.log3.txt

pgawlowicz commented 21 hours ago

hmm, PUSCH cfo is high and variable. 

2024-10-28T23:00:38.563891 [PHY     ] [D] [   293.0] PUSCH: rnti=0x4601 h_id=0 prb=[3, 22) symb=[0, 14) mod=16QAM rv=0 tbs=800 crc=KO iter=6.0 sinr=3.9dB t=193.9us uci_t=0.0us ret_t=0.0us
  cfo=+1354.1Hz
2024-10-28T23:00:38.574148 [PHY     ] [D] [   294.0] PUSCH: rnti=0x4601 h_id=0 prb=[3, 22) symb=[0, 14) mod=16QAM rv=0 tbs=800 crc=KO iter=6.0 sinr=-2.6dB t=182.5us uci_t=0.0us ret_t=0.0us
  cfo=-1381.8Hz
2024-10-28T23:00:38.584059 [PHY     ] [D] [   295.0] PUSCH: rnti=0x4601 h_id=0 prb=[3, 22) symb=[0, 14) mod=16QAM rv=0 tbs=800 crc=KO iter=6.0 sinr=-2.2dB t=177.4us uci_t=0.0us ret_t=0.0us
  cfo=-1356.4Hz
2024-10-28T23:00:38.593971 [PHY     ] [D] [   296.0] PUSCH: rnti=0x4601 h_id=0 prb=[3, 22) symb=[0, 14) mod=16QAM rv=0 tbs=800 crc=KO iter=6.0 sinr=-1.8dB t=174.1us uci_t=0.0us ret_t=0.0us
  cfo=-1360.4Hz
2024-10-28T23:00:38.603893 [PHY     ] [D] [   297.0] PUSCH: rnti=0x4601 h_id=0 prb=[3, 22) symb=[0, 14) mod=16QAM rv=0 tbs=800 crc=KO iter=6.0 sinr=-3.1dB t=177.5us uci_t=0.0us ret_t=0.0us
  cfo=-1320.9Hz

Do you have an external clock to connect with both gnb and UE USRPs?