srsran / srsRAN_Project

Open source O-RAN 5G CU/DU solution from Software Radio Systems (SRS) https://docs.srsran.com/projects/project
https://www.srsran.com
GNU Affero General Public License v3.0
523 stars 178 forks source link

unstable srsue traffic #934

Open Waseem0808 opened 1 week ago

Waseem0808 commented 1 week ago

Issue Description

DL traffic is very unstable in SRSUE and is causing the connection release.

Setup Details

A linux machine where gNB runs thru UHD B200 SDR with 10 MHZ geobondar clock in FDD mode(band 3). In the same linux machine, open5gs is also running. An another linux machine, where srsUE runs.

gNB configuration: gNB.txt Note: The gNB config is modified so it serves for both SRSUE COTS UE.

srsUE configuration: srsUe.txt

I am using iperf tool to pump DL traffic (UDP): Command in Server (srsUE PC): iperf -i 60 -s -b 5M -u Command in Client (gNB/open5gs PC): iperf -i 30 -t 600 -c 10.45.0.2 -b 5M -u

nb_ngap file: nb_ngap.zip

Expected Behavior

The DL traffic runs without interpretation.

Actual Behaviour

The 5Mbps traffic runs for few minutes (around 5 minutes) and then connection suddenly releases because of high error rate. The error rate suddenly jumps to quite high. If I try higher data rates then connection release even much earlier. I am adding few gNB snapshots along with time.

snapshot1 snapshot2 snapshot3 snapshot4

srsueSnapshot:

srsueSnapshot

iperf snapshots: Server:

iperfServer

Client:

iperfClient

Additional Information

The performance script is run first. Enabling more logs is making connection to loose connection much faster.

pgawlowicz commented 1 week ago

Do you run both gnb and srsUE on the same PC? did you execute this script? https://github.com/srsran/srsRAN_Project/blob/main/scripts/srsran_performance

Waseem0808 commented 1 week ago

Do you run both gnb and srsUE on the same PC? did you execute this script? https://github.com/srsran/srsRAN_Project/blob/main/scripts/srsran_performance

No, the gNB and srsUE is run on two different linux machines. Yes, I have run the performance script.

pgawlowicz commented 1 week ago

Could you share your logs? does it work better with smaller BW? like 10mhz?

Waseem0808 commented 1 week ago

I tried lowering the bandwidth from 20 MHZ to 10 MHZ but the situation remains the same. I am adding the logs for a 20 MHZ test run. [Uploading nb.zip…]()

Waseem0808 commented 1 week ago

It looks logs are not getting uploaded. Please use this link to download the logs: https://www.dropbox.com/scl/fo/kx9qkwihb934d4w368s99/AHRidDmVz4efQMWiobaJvSQ?rlkey=qbgylqz755h7mvr4kwvyl9c05&st=ln419tjr&dl=0

pgawlowicz commented 3 days ago

could you try to apply the patch from this comment to srsUE, recompile and try again? https://github.com/srsran/srsRAN_Project/issues/826#issuecomment-2370588502

Waseem0808 commented 1 day ago

Thanks for the patch. I am facing an issue with my setup now, I am having high error rate just after attach. This issue seems after I was making srsue to connect to internet only thru tun_srsue, These are the changes I made in srsue PC:

  1. Delete Existing Default Route The current default route is pointing to 192.168.12.1 via wlo1. sudo ip route del default via 192.168.12.1 dev wlo1

  2. Add a New Default Route via tun_srsue Next, set the default route to tun_srsue. sudo ip route add default via 10.45.0.1 dev tun_srsue

  3. Verify the Routing Table Check if the new default route is set correctly: ip route show Ensure that the default route is now via tun_srsue.

  4. echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

I then reverted these changes REVERTING TO ORIGINAL DEFAULT ROUTE:

sudo ip route del default via 10.45.0.1 dev tun_srsue
sudo ip route add default via 192.168.12.1 dev wlo1

My routing table before modifying routing table looks as after srsue attach:

root@srsue# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.12.1    0.0.0.0         UG    600    0        0 wlo1
10.45.0.0       0.0.0.0         255.255.255.0   U     0      0        0 tun_srsue
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 tun_srsue
192.168.12.0    0.0.0.0         255.255.255.0   U     600    0        0 wlo1

After I reverted the changes back, my routing table looks like:

root@srsue# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.12.1    0.0.0.0         UG    600    0        0 wlo1
10.45.0.0       0.0.0.0         255.255.255.0   U     0      0        0 tun_srsue
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlo1
192.168.12.0    0.0.0.0         255.255.255.0   U     600    0        0 wlo1

There is difference in routing table. I am not sure, How its impacting the traffic & how to revert it back. Please guide.

For reference, my routing table in gNB machine looks as:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.12.1    0.0.0.0         UG    600    0        0 wlo1
10.45.0.0       0.0.0.0         255.255.255.0   U     0      0        0 ogstun
10.53.1.0       0.0.0.0         255.255.255.0   U     0      0        0 br-1f7bd8df2c69
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 br-1f7bd8df2c69
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.19.1.0      0.0.0.0         255.255.255.0   U     0      0        0 br-c314e7707126
192.168.12.0    0.0.0.0         255.255.255.0   U     600    0        0 wlo1
pgawlowicz commented 1 day ago

you can just reboot the machine, the routing changes you did are not persistent.

Waseem0808 commented 1 day ago

I rebooted both the PCs multiple times but still the same issue. The srsue is release as soon as its attached. I didn't made any change in my setup except the routing table change. Earlier I used to have two tun_srsue interfaces in srsue PC after attach. Why there were two interfaces? Are we expecting two interfaces? I am now having only 1 interface, Is it okay?

pgawlowicz commented 1 day ago

Could you share gnb and srsUE logs?