O-RAN NearRT-RIC and XApp Tutorial: Simulating UEs across multiple virtual machines

rcbarke commented 3 days ago

Issue Description

Hello, I am learning how to leverage SRSRAN with NearRT-RIC and xApp software. To that effect, I am building the below tutorial on virtual machines: https://docs.srsran.com/projects/project/en/latest/tutorials/source/near-rt-ric/source/index.html

I have successfully run through all steps in this tutorial on a single virtual machine instance, and would now like to spin up a second virtual machine to create larger networks.

In my end state, I should have:

VM1: Open5GS Core + OSC RIC + SRSRAN Project gNB + SRSRANUE1
VM2: SRSRAN UE2

Setup Details

Both machines run Ubuntu 22.04 LTS.

A virtual private network, SRSRAN Network is configured in virtual box:

10.0.2.0/24

Port Forwarding is enabled:

ssh to core TCP 127.0.0.1 22222 10.0.2.15 22
ssh to node TCP 127.0.0.1 22223 10.0.2.100 22
entry to core TCP 127.0.0.1 22224 10.0.2.15 22
entry to node TCP 127.0.0.1 22225 10.0.2.100 22

I then configure both machines with static IPs to each address, with a 10.0.2.2 gateway (virtual box's default).

Expected Behavior

UE2 should be able to subscribe and connect to the network on VM1. There should be end-to-end connectivity, both across VM1 and VM2 as well as from UE2 to the RIC/EPC and back (ping, iperf3, etc.)

I am able to achieve this with UE1, but UE2 is unreachable.

Actual Behaviour

I complete the steps in the tutorial to set up each package as indicated above. Note on VM2, the only binaries that are built are SRSRAN4G for building UEs. All other software is located on VM1.

I note that after VM1 is setup, a bridged adapter is setup on 10.0.2.1 on VM1. Note the bolded IP route below. This seems to be causing pings to fail from VM1 to VM2 or from VM2 to VM1. The machines are receiving responses from this bridged adapter that the opposite VM is unreachable, breaking IP layer connectivity.

ric@OSC-Core:~$ ip route default via 10.0.2.2 dev enp0s3 proto static metric 100 10.0.2.0/24 dev br-37b838d7204e proto kernel scope link src 10.0.2.1 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100 10.53.1.0/24 dev br-fedbac9791e2 proto kernel scope link src 10.53.1.1 169.254.0.0/16 dev enp0s3 scope link metric 1000 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

Steps to reproduce the problem

See above. When I test, I ensure to declare separate namespaces ue1 and ue2 across both VMs according to my architecture. I.E. - VM1: sudo ip netns add ue1 VM2: sudo ip netns add ue2

Running sudo ./srsue ue_zmq.conf on VM2 currently does not succeed since there is no IP layer connectivity between VM1 and VM2.

Additional Information

Is it possible to configure SRSRAN across multiple VMs in this configuration? Again, we simply want to split the UEs off to separate nodes to simulate busier network conditions. This way, we can build more than 8 UEs within our simulation.

If this is possible, how should I modify the configuration to achieve this? How do we resolve the issue with the bridged adapter? Does ue_zmq.conf need to be modified before running the srsue binary on VM2?

rcbarke commented 3 days ago

Update: I was able to get multiple UEs connected from a single VM, collocated with the RIC, leveraging gnuradio. Cross VM communication is still experiencing the error described above due to the presence of the 10.0.2.1 bridged adapter. I would appreciate any information available on what the purpose of this adapter is and how to facilitate cross VM communication.

pgawlowicz commented 2 days ago

Hi, If you want to run gnb and srsUE on two different machines, you must correctly set the IP addresses for the zmq-based RF driver endpoints. Please have a look here: https://docs.srsran.com/projects/project/en/latest/tutorials/source/ntn/source/index.html#running-gnb-and-amarisoftue-on-separate-machines It describes a setup with amarisoftUE but should be the same for srsUE.

Note that even when running multiple UEs on separate machines, it is not possible to connect them to a single gnb directly. Therefore, you still need to use a gnuradio signal broker that connects the gnb and all UEs.

rcbarke commented 2 days ago

Hi, If you want to run gnb and srsUE on two different machines, you must correctly set the IP addresses for the zmq-based RF driver endpoints. Please have a look here: https://docs.srsran.com/projects/project/en/latest/tutorials/source/ntn/source/index.html#running-gnb-and-amarisoftue-on-separate-machines It describes a setup with amarisoftUE but should be the same for srsUE.

Note that even when running multiple UEs on separate machines, it is not possible to connect them to a single gnb directly. Therefore, you still need to use a gnuradio signal broker that connects the gnb and all UEs.

Thank you! The instructions within the documentation are clear, but in attempting to recreate this scenario, I am running into a subnet overlap with our virtual private network for our virtualbox VMs and the docker containers spun up by Open5GS_5GC, OSC RIC, and SRSRAN. They're all on 10.0.2.0/24, and the VMs seem to collide with a bridged network adapter from docker located at 10.0.2.1 when I attempt to put the RIC VM on 10.0.2.15 and the UE node VM on 10.0.2.100. Shifting the docker subnets via a daemon JSON causes build errors, and moving the VMs off of the 10.0.2.0/24 subnet breaks communication once resources are built. Any ideas?

pgawlowicz commented 1 day ago

Open5GS and srsran containers are running within 10.53.1.0/24 network, see here

For the ORAN-SC-RIC, you would need to change the config files:

change the IP of the RIC network here
IPs of all containers in the docker-compose.yml
change the IPs of the RIC components here
RIC IP in the gnb config file here
any ip address inside RIC comopents configs in https://github.com/srsran/oran-sc-ric/tree/main/ric/configs
IPs in the xAppBase class: here

Probably, you can use this answer for the above: https://github.com/srsran/srsRAN_Project/issues/788#issuecomment-2311996007

I think that should be enough to eliminate the conflict with the VM network.

rcbarke commented 7 hours ago

Open5GS and srsran containers are running within 10.53.1.0/24 network, see here

For the ORAN-SC-RIC, you would need to change the config files:

change the IP of the RIC network here

IPs of all containers in the docker-compose.yml

change the IPs of the RIC components here

RIC IP in the gnb config file here

any ip address inside RIC comopents configs in https://github.com/srsran/oran-sc-ric/tree/main/ric/configs

IPs in the xAppBase class: here

Probably, you can use this answer for the above: #788 (comment)

I think that should be enough to eliminate the conflict with the VM network.

This worked perfectly, thank you! Thus I have closed: https://github.com/srsran/srsRAN_Project/issues/853

I now have connectivity across the E2 gNB/RIC link, KPIMon is capturing metrics at the core, and Grafana is capturing metrics at the RAN.

KPIMon seems to only capture metrics for two of my three UEs (missing UE3), which is peculiar, though Grafana shows all three.

However, when I attempt to add UE 4 and connect it across VMs, gnuradio is not attaching any of the UEs. I suspect my problem is there. I am following the prior instructions referenced at: https://docs.srsran.com/projects/project/en/latest/tutorials/source/ntn/source/index.html#running-gnb-and-amarisoftue-on-separate-machines

I have two VMs:

VM1 at 10.0.2.15 containing my core and UEs 1-3
VM2 at 10.0.2.100 containing UE4
The VMs are connected via a virtualbox private NAT and I have ping connectivity across both VMs
UE namespaces are declared for UE1-3 on VM1 and U4 on VM2

Within subscribers.csv on VM1:

I have added a row for ue4, initialized based on ue3's configuration. I incremented every UE unique ID (ex: IMSI) and left all the common IDs across all UEs the same.

Within ue4_zmq.conf on VM2:

tx_port is set to *:2401
rx_port is set to 10.0.2.15:2400 (VM1's IP)
All parameters common to subscriber.csv match
All log files now reference "ue4"

Within gnuradio on VM1:

Cloned UE3 TX/RX paths and connected them to each adder/disaggregator
Tx DL to UE updated to 0.0.0.0:2400
Rx UL from UE updated to 10.0.2.100:2401 (VM2's IP)
QTGUI Block added for ue4's pathloss parameter

I've attached each config file. Where is my configuration error? remote_ue.zip

srsran / srsRAN_Project