openaicellular / oaic

Core software for Open AI Cellular
https://openaicellular.github.io/oaic/
GNU General Public License v3.0
65 stars 23 forks source link

E2 connection failing to be established #103

Closed gth828r closed 1 year ago

gth828r commented 1 year ago

Hello, and thanks for your great project!

I am walking through the steps at https://openaicellular.github.io/oaic/oran_installation.html trying to set up a single machine to run the RIC, EPC, eNB, and UEs with ZMQ. I am able to see the E2 messages generated by the E2 agent on the eNB, but I never see a response. When I look at the E2 manager logs, I see the following:

{"crit":"INFO","ts":1688369013543,"id":"E2Manager","msg":"[RMR -> E2 Manager] #rmrCgoApi.RecvMsg - message { MType: 12001, Len: 1693, Meid: \"enB_macro_001_001_0019b0\", Xaction: &\"   874327858705\", Payload: [&31302e3130372e3235332e33393a33
383030307c3c453241502d5044553e3c696e6974696174696e674d6573736167653e3c70726f636564757265436f64653e313c2f70726f636564757265436f64653e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c4532736574757052657175
6573743e3c70726f746f636f6c4945733e3c45327365747570526571756573744945733e3c69643e333c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c476c6f62616c45326e6f64652d49443e3c654e423e3c676c6f62616c2d654e
422d49443e3c704c4d4e2d4964656e746974793e3030463131303c2f704c4d4e2d4964656e746974793e3c654e422d49443e3c6d6163726f2d654e422d49443e30303030303030303030303131303031313031313c2f6d6163726f2d654e422d49443e3c2f654e422d49443e3c2f676c6f62616c2d654e422d
49443e3c2f654e423e3c2f476c6f62616c45326e6f64652d49443e3c2f76616c75653e3c2f45327365747570526571756573744945733e3c45327365747570526571756573744945733e3c69643e31303c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e
3c76616c75653e3c52414e66756e6374696f6e732d4c6973743e3c50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c69643e383c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c52414e66756e6374696f6e2d49
74656d3e3c72616e46756e6374696f6e49443e303c2f72616e46756e6374696f6e49443e3c72616e46756e6374696f6e446566696e6974696f6e3e3030433034463532343134453244343533323533344432443442353034443030303031323331324533333245333632453331324533343245333132453331
32453331324533323245333230353030344235303444323036443646364536393734364637323630303030313031303338303534373236393637363736353732333130313031303030313036314538303446324434333535324435353530323034443635363137333735373236353644363536453734323034
333646364537343631363936453635373232303636364637323230373436383635323034353530343332303633364636453645363536333734363536343230363436353730364336463739364436353645373430313031303130313c2f72616e46756e6374696f6e446566696e6974696f6e3e3c72616e4675
6e6374696f6e5265766973696f6e3e303c2f72616e46756e6374696f6e5265766973696f6e3e3c2f52414e66756e6374696f6e2d4974656d3e3c2f76616c75653e3c2f50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c50726f746f636f6c49452d53696e676c65436f6e7461696e6572
3e3c69643e383c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c52414e66756e6374696f6e2d4974656d3e3c72616e46756e6374696f6e49443e313c2f72616e46756e6374696f6e49443e3c72616e46756e6374696f6e446566696e
6974696f6e3e36303038303034463532343134453244343533323533344432443637344534323244344535323534303030303230333132453333324533363245333132453334324533313245333233383334333533383245333933393245333032453332333132453333324533333245333132453333303330
3036373445343232443445353235343030303130313033383035343732363936373637363537323331303130313030303130363033303035323635373036463732373433313031303130303030303130323830353036313732363136443331313030313031303130313c2f72616e46756e6374696f6e446566
696e6974696f6e3e3c72616e46756e6374696f6e5265766973696f6e3e303c2f72616e46756e6374696f6e5265766973696f6e3e3c2f52414e66756e6374696f6e2d4974656d3e3c2f76616c75653e3c2f50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c2f52414e66756e6374696f6e
732d4c6973743e3c2f76616c75653e3c2f45327365747570526571756573744945733e3c2f70726f746f636f6c4945733e3c2f45327365747570526571756573743e3c2f76616c75653e3c2f696e6974696174696e674d6573736167653e3c2f453241502d5044553e] } has been received for transaction id: 874327858705","mdc":{"time":"2023-07-03 07:23:33.543"}}                                                                                                                                                                                 {"crit":"INFO","ts":1688369013544,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.Handle - RAN name: enB_macro_001_001_0019b0 - received E2_SETUP_REQUEST. Payload: 31302e3130372e3235332e33393a33383030307c3c453241502d5044553e3c696e6974696174696e674d6573736167653e3c70726f636564757265436f64653e313c2f70726f636564757265436f64653e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c45327365747570526571756573743e3c70726f746f636f6c4945733e3c45327365747570526571756573744945733e3c69643e333c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c476c6f62616c45326e6f64652d49443e3c654e423e3c676c6f62616c2d654e422d49443e3c704c4d4e2d4964656e746974793e3030463131303c2f704c4d4e2d4964656e746974793e3c654e422d49443e3c6d6163726f2d654e422d49443e30303030303030303030303131303031313031313c2f6d6163726f2d654e422d49443e3c2f654e422d49443e3c2f676c6f62616c2d654e422d49443e3c2f654e423e3c2f476c6f62616c45326e6f64652d49443e3c2f76616c75653e3c2f45327365747570526571756573744945733e3c45327365747570526571756573744945733e3c69643e31303c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c52414e66756e6374696f6e732d4c6973743e3c50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c69643e383c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c52414e66756e6374696f6e2d4974656d3e3c72616e46756e6374696f6e49443e303c2f72616e46756e6374696f6e49443e3c72616e46756e6374696f6e446566696e6974696f6e3e303043303446353234313445324434353332353334443244344235303444303030303132333132453333324533363245333132453334324533313245333132453331324533323245333230353030344235303444323036443646364536393734364637323630303030313031303338303534373236393637363736353732333130313031303030313036314538303446324434333535324435353530323034443635363137333735373236353644363536453734323034333646364537343631363936453635373232303636364637323230373436383635323034353530343332303633364636453645363536333734363536343230363436353730364336463739364436353645373430313031303130313c2f72616e46756e6374696f6e446566696e6974696f6e3e3c72616e46756e6374696f6e5265766973696f6e3e303c2f72616e46756e6374696f6e5265766973696f6e3e3c2f52414e66756e6374696f6e2d4974656d3e3c2f76616c75653e3c2f50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c69643e383c2f69643e3c637269746963616c6974793e3c72656a6563742f3e3c2f637269746963616c6974793e3c76616c75653e3c52414e66756e6374696f6e2d4974656d3e3c72616e46756e6374696f6e49443e313c2f72616e46756e6374696f6e49443e3c72616e46756e6374696f6e446566696e6974696f6e3e363030383030344635323431344532443435333235333444324436373445343232443445353235343030303032303331324533333245333632453331324533343245333132453332333833343335333832453339333932453330324533323331324533333245333332453331324533333033303036373445343232443445353235343030303130313033383035343732363936373637363537323331303130313030303130363033303035323635373036463732373433313031303130303030303130323830353036313732363136443331313030313031303130313c2f72616e46756e6374696f6e446566696e6974696f6e3e3c72616e46756e6374696f6e5265766973696f6e3e303c2f72616e46756e6374696f6e5265766973696f6e3e3c2f52414e66756e6374696f6e2d4974656d3e3c2f76616c75653e3c2f50726f746f636f6c49452d53696e676c65436f6e7461696e65723e3c2f52414e66756e6374696f6e732d4c6973743e3c2f76616c75653e3c2f45327365747570526571756573744945733e3c2f70726f746f636f6c4945733e3c2f45327365747570526571756573743e3c2f76616c75653e3c2f696e6974696174696e674d6573736167653e3c2f453241502d5044553e","mdc":{"time":"2023-07-03 07:23:33.544"}}{"crit":"INFO","ts":1688369013544,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.Handle - got general configuration from rnib - enableRic: true","mdc":{"time":"2023-07-03 07:23:33.544"}}{"crit":"INFO","ts":1688369013544,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.parseSetupRequest - payload: <E2AP-PDU><initiatingMessage><procedureCode>1</procedureCode><criticality><reject/></criticality><value><E2setupRequest><protocolIEs><E2setupRequestIEs><id>3</id><criticality><reject/></criticality><value><GlobalE2node-ID><eNB><global-eNB-ID><pLMN-Identity>00F110</pLMN-Identity><eNB-ID><macro-eNB-ID>00000000000110011011</macro-eNB-ID></eNB-ID></global-eNB-ID></eNB></GlobalE2node-ID></value></E2setupRequestIEs><E2setupRequestIEs><id>10</id><criticality><reject/></criticality><value><RANfunctions-List><ProtocolIE-SingleContainer><id>8</id><criticality><reject/></criticality><value><RANfunction-Item><ranFunctionID>0</ranFunctionID><ranFunctionDefinition>00C04F52414E2D4532534D2D4B504D000012312E332E362E312E342E312E312E312E322E3205004B504D206D6F6E69746F72600001010380547269676765723101010001061E804F2D43552D5550204D6561737572656D656E7420436F6E7461696E657220666F72207468652045504320636F6E6E6563746564206465706C6F796D656E7401010101</ranFunctionDefinition><ranFunctionRevision>0</ranFunctionRevision></RANfunction-Item></value></ProtocolIE-SingleContainer><ProtocolIE-SingleContainer><id>8</id><criticality><reject/></criticality><value><RANfunction-Item><ranFunctionID>1</ranFunctionID><ranFunctionDefinition>6008004F52414E2D4532534D2D674E422D4E5254000020312E332E362E312E342E312E32383435382E39392E302E32312E332E332E312E330300674E422D4E525400010103805472696767657231010100010603005265706F72743101010000010280506172616D311001010101</ranFunctionDefinition><ranFunctionRevision>0</ranFunctionRevision></RANfunction-Item></value></ProtocolIE-SingleContainer></RANfunctions-List></value></E2setupRequestIEs></protocolIEs></E2setupRequest></value></initiatingMessage></E2AP-PDU>","mdc":{"time":"2023-07-03 07:23:33.544"}}{"crit":"INFO","ts":1688369013545,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.Handle - E2T Address: 10.107.253.39:38000 - handling E2_SETUP_REQUEST","mdc":{"time":"2023-07-03 07:23:33.545"}}
{"crit":"INFO","ts":1688369013545,"id":"E2Manager","msg":"#E2TInstancesManager.GetE2TInstance - E2T Instance address: 10.107.253.39:38000 not found on DB","mdc":{"time":"2023-07-03 07:23:33.545"}}
{"crit":"ERROR","ts":1688369013546,"id":"E2Manager","msg":"#E2TermInitNotificationHandler.Handle - Failed retrieving E2TInstance. error: #rNibReader.getByKeyAndUnmarshalJson - entity of type *entities.E2TInstance not found. Key: E2TInstance:10.107.253.39:38000","mdc":{"time":"2023-07-03 07:23:33.546"}}

Is there something obvious that I have done wrong? Has anyone else seen this issue?

gth828r commented 1 year ago

For the record, I am running on GCP with an Ubuntu 20.04 VM with 8 cores, 32 GB of RAM, and a 32 GB disk. Below are my notes for the commands which I ran (largely just copied from the documentation, with a few fixes that seemed to help resolve setup issues after a few failed attempts)

# 1. Set up the RIC

git clone https://github.com/openaicellular/oaic.git
cd oaic
git submodule update --init --recursive

# Perform this to get example configs
git submodule update --init --recursive --remote
cd RIC-Deployment/RECIPE_EXAMPLE

cd ../..

cd RIC-Deployment/tools/k8s/bin
./gen-cloud-init.sh

sudo apt-get update
sudo ./k8s-1node-cloud-init-k_1_16-h_2_17-d_cur.sh

# The machine will reboot here

sudo kubectl create ns ricinfra

sudo helm install stable/nfs-server-provisioner --namespace ricinfra --name nfs-release-1
sudo kubectl patch storageclass nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
sudo apt install -y nfs-common

sudo docker run -d -p 5001:5000 --restart=always --name ric registry:2

cd oaic/ric-plt-e2/

cd RIC-E2-TERMINATION
sudo docker build -f Dockerfile -t localhost:5001/ric-plt-e2:5.5.0 .
sudo docker push localhost:5001/ric-plt-e2:5.5.0

cd ~/oaic/RIC-Deployment/bin
sudo ./deploy-ric-platform -f ../RECIPE_EXAMPLE/PLATFORM/example_recipe_oran_e_release_modified_e2.yaml

# 2. Set up srsRAN with an E2 agent

sudo apt-get install -y build-essential cmake libfftw3-dev libmbedtls-dev libboost-program-options-dev libconfig++-dev libsctp-dev libtool autoconf

sudo apt-get install -y libzmq3-dev

sudo add-apt-repository ppa:ettusresearch/uhd
sudo apt-get update
sudo apt-get install -y libuhd-dev libuhd4.4.0 uhd-host

cd ~/oaic/asn1c
sudo apt install libtool autoconf
git checkout velichkov_s1ap_plus_option_group
autoreconf -iv
./configure
make -j`nproc`
sudo make install
sudo ldconfig

cd ~/oaic/srsRAN-e2/
mkdir build
export SRS=`realpath .`
cd build
cmake ../ -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    -DRIC_GENERATED_E2AP_BINDING_DIR=${SRS}/e2_bindings/E2AP-v01.01 \
    -DRIC_GENERATED_E2SM_KPM_BINDING_DIR=${SRS}/e2_bindings/E2SM-KPM \
    -DRIC_GENERATED_E2SM_GNB_NRT_BINDING_DIR=${SRS}/e2_bindings/E2SM-GNB-NRT
make -j`nproc`
sudo make install
sudo ldconfig
sudo srsran_install_configs.sh user --force
cd ../../..
joshuamoorexyz commented 1 year ago

I am glad you were able to resolve. Sometimes restarting the pods can help and make sure all needed pods are in fact in the running state before trying. E2 Setup response can take a few minutes sometimes.

gth828r commented 1 year ago

Hello, thank you for your response. Sorry for the confusion, but I never resolved the reported error of:

Failed retrieving E2TInstance

What I listed in my follow-up comment previously was just the steps I had taken to get to that point. I have since been trying other projects, so this is not urgent on my end; however, it would be great to have multiple projects to experiment with to ensure things like compatibility and spec compliance. I'll be sure to try again in the future and see where things stand!