gNB is not scheduling uplink C-plane on release_24_04

[rejohnsleebac]

Issue Description

Uplink PUSCH C-plane packets are missing in release_24_04. We have captured the pcap from fronthaul and couldn't find any Uplink PUSCH cplane packets other than PRACH C-plane packets

Setup Details

SRS gNB DU ------> E810 -----> Fronthaul ------> tcpdump

Expected Behavior

Expecting Uplink C-plane other than PRACH

Actual Behaviour

There are only Cplane PRACH packets

Additional Information

amf: addr: 192.20.20.10 # The address or hostname of the AMF. bind_addr: 192.20.20.111 # A local IP that the gNB binds to for traffic from the AMF.

ru_ofh: ru_bandwidth_MHz: 100 # RU instantaneous bandwidth.

t1a_max_cp_dl: 500 # Maximum T1a on Control-Plane for Downlink in microseconds. t1a_min_cp_dl: 250 # Minimum T1a on Control-Plane for Downlink in microseconds. t1a_max_cp_ul: 465 # Maximum T1a on Control-Plane for Uplink in microseconds. t1a_min_cp_ul: 250 # Minimum T1a on Control-Plane for Uplink in microseconds. t1a_max_up: 250 # Maximum T1a on User-Plane in microseconds. t1a_min_up: 80 # Minimum T1a on User-Plane in microseconds. ta4_max: 500 # Maximum Ta4 on User-Plane in microseconds. ta4_min: 25
is_prach_cp_enabled: true # Configures if Control-Plane messages should be used to receive PRACH messages. compr_method_ul: bfp # Uplink compression method. compr_bitwidth_ul: 9 # Uplink IQ samples bitwidth after compression. compr_method_dl: bfp # Downlink compression method. compr_bitwidth_dl: 9 # Downlink IQ samples bitwidth after compression. compr_method_prach: bfp # PRACH compression method. compr_bitwidth_prach: 9 # PRACH IQ samples bitwidth after compression. enable_ul_static_compr_hdr: true # Configures if the compression header is present for uplink User-Plane messages (false) or not present (true). enable_dl_static_compr_hdr: true # Configures if the compression header is present for downlink User-Plane messages (false) or not present (true). iq_scaling: 5.5 # IQ samples scaling factor applied before compression, should be a positive value smaller than 10. cells:

• network_interface: 0000:51:00.0 ru_mac_addr: 98:ae:71:00:00:11 # RU MAC address. du_mac_addr: 00:11:22:33:44:66 # DU MAC address. vlan_tag_cp: 200 # VLAN tag value for C-Plane. vlan_tag_up: 200 # VLAN tag value for U-Plane. prach_port_id: [4, 5, 6, 7] # PRACH eAxC port value. dl_port_id: [0, 1, 2, 3] # Downlink eAxC port values. ul_port_id: [0, 1, 2, 3] # Uplink eAxC port values.

cell_cfg: dl_arfcn: 643452 # ARFCN of the downlink carrier (center frequency). band: 78 # The NR band. channel_bandwidth_MHz: 100 # Bandwith in MHz. Number of PRBs will be automatically derived. common_scs: 30 # Subcarrier spacing in kHz used for data. plmn: "00101" # PLMN broadcasted by the gNB. tac: 1 # Tracking area code (needs to match the core configuration). pci: 1 # Physical cell ID. nof_antennas_dl: 4 # Downlink number of antennas. nof_antennas_ul: 4 # Uplink number of anntennas. prach: prach_config_index: 159 # PRACH configuration index. prach_root_sequence_index: 0 # PRACH root sequence index. zero_correlation_zone: 0 # Zero correlation zone. prach_frequency_start: 0 # Offset in PRBs of lowest PRACH transmission occasion in frequency domain respective to PRB 0. pucch: sr_period_ms: 20 # This can be set either 20 or 40 ms. f1_nof_cell_res_sr: 15 f2_nof_cell_res_csi: 15 csi: csi_rs_period: 40 # This can be set either 20 or 40 ms. pusch: dmrs_additional_position: 2 tdd_ul_dl_cfg: nof_dl_symbols: 6 nof_ul_symbols: 4 nof_dl_slots: 7 nof_ul_slots: 2

log: filename: /tmp/gnb.log # Path of the log file. all_level: debug # Logging level applied to all layers.

pcap: mac_enable: false # Set to true to enable MAC-layer PCAPs. mac_filename: /tmp/gnb_mac.pcap # Path where the MAC PCAP is stored. ngap_enable: false # Set to true to enable NGAP PCAPs. ngap_filename: /tmp/gnb_ngap.pcap # Path where the NGAP PCAP is stored.

expert_phy: max_proc_delay: 5

hal: eal_args: "--lcores (0-1)@(1-23) -a 0000:51:00.0"

The PCAP and log file is also attached issue.zip

[andrepuschmann]

Hey - it seems there is no UE attaching to the cell. UL Cplane is only scheduled by the gNB if there is any UL scheduling. If you've just started the gNB and there is no UE attempting to attach, e.g. do PRACH, there is no PUSCH or PUCCH generated. The actual problem in the PCAP is that the RU is not sending any PRACH uplane message with the PRACH PRBs. You probably have to check the RU config and see if the packets arrive in time and that the RU responds properly.

Hope that helps.

[rejohnsleebac]

Hi @andrepuschmann,

Thanks for your quick response, Earlier we have tried to capture the frounthaul packets through a NIC card and now we have tried connecting the RU and tried to attach the UE. Now we are getting the uplink PUSCH C-plane at the time of RACH process.We are getting all packets on time in both radio and DU side,

Now the current issue is :

• Sometimes UE is sending PRACH and is received at the gNB but the TA value is very high and MSG3 CRC is failure. May be due to that RAR is not proper.

• SSB decoding is not proper while checking with the VSA (We did not see any PSS,SSS,etc in the block).

• We couldn't find the SIB message in XCAL which is supposed to be received after MIB.

-How do we increase the SSB Block Power?

• Configurations are: gnb_config.zip

[rejohnsleebac]

Hi @andrepuschmann

We have successfully connected the UE with the gNB after changing the slot format XCAL log -

So now we are using a third party Core and we wanted to know the procedure to the add the details of the NGU interface in the gNB. Can you help us on how we can add this? Is there any example configurations available with NGU?

I have attached the current gnb.log for reference. gnb_logs.zip

[rejohnsleebac]

We are able to connect the UE with the Open5gs gNB but the throughput we are getting is only a maximum of around 300Mbps. We are also using the Open5gs Core in our setup. Can you suggest why we could be observing low throughput and how we could increase the throughput further?

[ismagom]

Can you paste console trace and upload gnb log?

[rejohnsleebac]

I have attached the gNB log and trace console_log.txt gnb_logs.zip

[andrepuschmann]

Hey @rejohnsleebac - I suggest you start with UDP testing first, I guess you're using TCP right now.

For your reference, the SSB block power can be set like this:

cell_cfg:
  ssb:
    ssb_period: 10
    ssb_block_power_dbm: -21

Also, it seems you're using a VVDN RU. Would you mind telling what exact model it is and which firmware version you use? Was there anything special you set? You mentioned you changed the "slot format". You mean the TDD pattern? Was that a change in the RU or DU config?

[rejohnsleebac]

We are using band N78 radio with firmware version 3.0.5 The slot format was DDDSU and we changed it to DDDDDDDSUU and it was changed in the DU config.

[rejohnsleebac]

Hi @andrepuschmann and @ismagom

We did the throughput test and so far we have achieved downlink around 700-800 mbps and uplink 50-60 mpbs

We have some questions:

1. From the trace its clear that MCS is 28, Bler is 0-4% and rank is 4 but still throughput is 700 mbps, Where is it dropping? Do we need any parameters to be configured or any RLC mode settings to be done?
2. We are getting 700 mbps in UDP, but TCP performance is very poor and its around 100-200 Mbps in the same channel condition.

The config files are attached below: du_configs.zip

[ismagom]

Thanks @rejohnsleebac ,

From the console trace only, I would say the problem is not in the stack. It seems not enough data is arriving to the N3 interface. Can you share the file /tmp/gnb.log please? A short 10 seconds run is enough. Using UDP DL please.

Once UDP is achieving full rate can try TCP. To get full rate with TCP there are a few tweaks needed. Can you try adding the following options to the config:

cu_up: 
  gtpu_queue_size: 32768
  warn_on_drop: true
  gtpu_reordering_timer: 20

Then increase the NIC buffers of the N3 interface:

sudo ethtool -G <itf_name> tx 8160 rx 8160

and the UDP buffers:

sudo sysctl -w net.core.rmem_max=2147483647
sudo sysctl -w net.core.rmem_default=2147483647

Then send us the logs. We'll be able to tell where are the bottlenecks.

Actually, maybe setting the tweaks above also helps with UDP DL (depending where the packets are being dropped).

[rejohnsleebac]

Thanks @ismagom for your valuable information,

We have attached the gnb logs here: gnb_log.zip You can extract the .7z file inside the zip file with the command _7z e gnblogs.7z (since the size was very big)

Also with the command _sudo ethtool -G tx 8160 rx 8160_ did you mean the tunnel interface ogstun? When tried that we got an error :"netlink error: Operation not supported" Is there any option for using a physical interface instead of the tunnel interface?

Note: We got these logs by using the additional configurations you have provided without the ethtool command and we couldn't find any increase in the throughput. Also we observed an error in the gnb.log file that we send: 2024-05-14T06:17:05.308524 [GTPU ] [W] Dropped GTP-U PDU, queue is full. teid=0x000001 Will that be an issue?

[ismagom]

Hi @rejohnsleebac ,

you need to tell us a bit more when you send the logs. What were you doing, UDP or TCP? what throughput did you see?

Let's do first a run with UDP, copy and paste the console trace and the logs.

Then do another run with TCP, copy and paste console trace and the logs.

What core network are you using? Is it running in the same machine using local interface?

[brendan-mcauliffe]

Hi @rejohnsleebac ,

Can you also share the most up-to-date version of your gNB configuration file?

[vvdn5gtest]

Hi @ismagom and @brendan-mcauliffe We tried that test using UDP Initial we tried using the open5gs and core and now we are trying to test with another core with which we attained 1.5Gbps with another gNB stack.But the throughput issue still persisted. The gNB trace of the test using another core is given below: The iperf log is shown below: The block diagram of the test setup is as follows: Now we have tried with a simgle RU. The UE we are using is Oneplus 9 The latest configuration is: gnb_ru_ran_vvdn_lpru_tdd_n78_100mhz_4x4.zip

[pgawlowicz]

@vvdn5gtest any update on this issue?