search

srsran / srsRAN_Project

Open source O-RAN 5G CU/DU solution from Software Radio Systems (SRS) https://docs.srsran.com/projects/project
https://www.srsran.com
GNU Affero General Public License v3.0
527 stars 179 forks source link

Significant degradation in DL/UL performance between May21st and Jun10th versions of the "test" repository #675

Closed wozzamac closed 5 months ago

wozzamac commented 5 months ago

Issue Description

Whilst testing the srsRAN system for our customer Antevia I've seen a significant degradation in performance between the May 21st version of the "test" repository and the Jun 10th version.

I suspect a recent change/changes have adversely affected performance.

Setup Details

I'm using a Picocom 802R and running the same config file on both versions of the "test" repository.

I attach a Motorola MG62 and perform a DL/UL speed test and compare results.

The May 21st version works well, the Jun 10th version doesn't.

Expected Behavior

I would expect very little difference in performance.

Actual Behaviour

Very poor data rates and large error rates when using the Jun10th version.

Steps to reproduce the problem

Please find attached the config file, relevant gnb logs, git log details showing commit-ids and gnb t-trace info.

Additional Information

Attached zip file contains git log details for each srsRAN version srsRAN_test_May21vJun10.zip

ismagom commented 5 months ago

hi @wozzamac, you should not use the test branch. Do you see the same in the main branch?

wozzamac commented 5 months ago

I quite often use the "test" branch to try more recent changes.

I've noticed the "main" and "test" branch appear to be at the same commit-id point of Jun 10th "c33cacba7d940e734ac7bad08935cbc35578fad9"

I've just built the "main" branch and see the same poor DL/UL performance.

herlesupreeth commented 5 months ago

Hey @wozzamac , we have identified an issue in the scheduler which could affect the UL performance when there are lots of retranmissions and will be pushing out a fix soon. I will let you know once the fix is out so that you can give it a try.

wozzamac commented 5 months ago

Hey @wozzamac , we have identified an issue in the scheduler which could affect the UL performance when there are lots of retranmissions and will be pushing out a fix soon. I will let you know once the fix is out so that you can give it a try.

Ok thanks for the update

herlesupreeth commented 5 months ago

@wozzamac The fix has been pushed to test branch. Please give it a try and let me know if it fixes the issue. If it doesn't then please send the gNB (mac and rrc in debug) logs and console trace output

wozzamac commented 5 months ago

@wozzamac The fix has been pushed to test branch. Please give it a try and let me know if it fixes the issue. If it doesn't then please send the gNB (mac and rrc in debug) logs and console trace output

Hi,

Thanks for the quick response/turn-around.

Just grabbed the latest Jun17th "test" repo and built.

Unfortunately the DL/UL rates are still very poor compared to the May21st "test" repo version.

Jun17-test-version-results.zip

Please find attached the gnb.log file with rrc/mac trace level set to "debug" as requested.

Here's the gnb trace "t" output:


antevia@dusupermicr:~$ sudo taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ~/srsRAN_Project/build/apps/gnb/gnb -c ~/Downloads/gnb_ru_picocom_scb_tdd_n77_30mhz_May21.yml

--== srsRAN gNB (commit 40b17b429) ==--

The PRACH detector will not meet the performance requirements with the configuration {Format B4, ZCZ 0, SCS 30kHz, Rx ports 2}.
N2: Connection to AMF on 10.4.31.116:38412 completed
Cell pci=1, bw=30 MHz, 4T4R, dl_arfcn=678334 (n77), dl_freq=4175.01 MHz, dl_ssb_arfcn=677664, ul_freq=4175.01 MHz

Initializing the Open Fronthaul Interface for sector#0: ul_compr=[none,16], dl_compr=[none,16], prach_compr=[none,16], prach_cp_enabled=false, downlink_broadcast=false
Warning: Configured PRACH occasion collides with PUCCH RBs ([0..1) intersects [0..12)). Some interference between PUCCH and PRACH is expected.
Warning: Configured PRACH occasion collides with PUCCH RBs ([0..1) intersects [0..12)). Some interference between PUCCH and PRACH is expected.
==== gNB started ===
Type <h> to view help
t

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr     ta  phr
   1 4604 |   3   1    5   2.6k    2    0   0%      0 |  26.7  -5.2    9   4.2k    1    0   0%      0   -97n  -11
   1 4604 |   1   1    1   6.0k   13    4  23%      0 |   6.7 -44.0   23    18k    7   30  81%      0   -16n    0
   1 4604 |   1   1    0      0    0    0   0%      0 |  32.4  -4.6   28   4.4k    1    0   0%      0      0    5
   1 4604 |   1   1    0    112    1    1  50%      0 |  32.9  -4.7   28   8.7k    2    0   0%      0    24n    5
   1 4607 |   1   1    2   6.5k   15    5  25%      0 |   6.8 -44.4   23    22k    7   20  74%      0      0    5
   1 4607 |   1   1    4   350k  120  211  63%      0 |  -2.8 -49.2    5   206k   56  142  71%      0      0    3
   1 4607 |   1   1    1   232k  101  161  61%      0 |  -5.1 -52.3    9   244k   25   89  78%      0      0   -6
   1 4607 |   1   1    0   4.6k    2   23  92%      0 |   n/a   n/a    0      0    0    0   0%      0      0   -6
   1 4607 |   1   1    0    83k   41   80  66%      0 |  14.3 -30.1   23    76k   15   14  48%      0      0    0
   1 4607 |   1   1    1    63k   28   74  72%      0 |  33.2  -4.4   28    28k    7    0   0%      0      0    4
   1 4607 |   3   1    1    97k   51   91  64%      0 |  -7.7 -52.1    8    55k   14   59  80%    384  -260n   -6

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr     ta  phr
   1 4607 |   1   1    0    17k   13   28  68%      0 |   7.6 -41.5   15    36k    9   14  60%      0      0    2
   1 4607 |   1   1    0   123k   64   92  58%  2.27k |  34.0  -3.2   28    61k   14    0   0%      0      0    4
   1 4607 |   1   1    1   118k   40  110  73%      0 |  33.3  -4.2   28    17k    4    0   0%      0      0    4
   1 4607 |   1   1    0    67k   33   67  67%      0 |  34.8  -4.1   28   8.7k    2    0   0%      0      0    3
   1 4607 |   1   1    0    12k    6    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0      0    3
   1 4607 |   1   1    0   9.3k    5   22  81%      0 |   n/a   n/a    0      0    0    0   0%      0      0    3
   1 4607 |   1   1    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0      0    3
   1 4607 |   1   1    0   111k   57  119  67%      0 |  34.2  -4.2   28    22k    5    0   0%      0      0    3
   1 4607 |   1   1    1   284k  118  223  65%      0 |  34.0  -4.2   28    39k    9    0   0%      0      0    3
   1 4607 |   1   1    1   129k   51   73  58%      0 |  32.6  -4.1   28    17k    4    0   0%      0      0    3
   1 4607 |   1   1    1   327k  121  119  49%      0 |  33.2  -4.5   28    39k    9    0   0%      0      0    5

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr     ta  phr
   1 4607 |   3   1    1   418k  155  298  65%      0 |  33.6  -4.2   28    57k   13    0   0%      0  -764n    4
   1 4607 |   1   1    0   410k  169  264  60%      0 |  34.1  -3.2   28    61k   14    0   0%      0      0    4
   1 4607 |   1   1    1   263k  110  126  53%      0 |  19.3 -25.2   25    31k    8   10  55%      0      0   -8
   1 4607 |   1   1    0   250k  107  228  68%      0 |  35.7  -2.3   28    48k   11    0   0%      0   122n    3
   1 4607 |   2   1    0    74k   33  169  83%      0 |  36.0  -2.5   28   8.7k    2    0   0%      0      0    4
   1 4607 |   1   1    0    61k   32   90  73%      0 |  35.9  -1.2   28    17k    4    0   0%      0      0    2
   1 4607 |   1   1    0    33k   17   56  76%      0 |  34.5  -2.7   28   8.7k    2    0   0%      0      0    3
   1 4607 |   1   1    0    12k    6    0   0%      0 |  36.1  -1.3   28   4.4k    1    0   0%      0      0    2
   1 4607 |   1   1    0    12k    6    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0      0    2
   1 4607 |   1   1    0      0    0    0   0%      0 |   n/a   n/a    0      0    0    0   0%      0      0    2
   1 4607 |  12   4    7    43k   27   11  28%      0 |  -2.2 -44.8    7   962k   75  166  68%   150k  -244n   -6

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr     ta  phr
   1 4607 |   1   1   12    54k   26   17  39%      0 |  -2.9 -43.8    2   604k  198  402  67%   300k  -830n   -6
   1 4607 |  11   4    9    30k   20   16  44%      0 |  -2.9 -43.7    2   456k  200  400  66%   300k  -651n   -7
   1 4607 |  13   4    9    34k   21    6  22%      0 |  -2.8 -44.5    2   600k  196  404  67%   300k  -146n   -6
   1 4607 |  12   4   11    46k   25   18  41%      0 |  -2.3 -46.5    3   737k  189  411  68%   300k   341n   -5
   1 4607 |  13   4    7    60k   29   13  30%      0 |  -1.7 -45.8    3   451k  198  402  67%   300k   797n   -5
   1 4607 |  12   4    5   316k   61   51  45%      0 |  -1.3 -45.6    5   1.1M  195  405  67%   300k   699n   -4
   1 4607 |  12   4   10   182k   28   10  26%      0 |  -1.6 -46.1    4   451k  198  402  67%   300k   976n   -1
   1 4607 |  12   4    5   223k   40   61  60%      0 |  -2.6 -47.2    4   429k  188  412  68%   300k  -260n   -3
   1 4607 |   1   1    9    72k   26   12  31%      0 |  -2.3 -47.1    4   438k  192  408  68%   300k     1u   -5
   1 4607 |  12   4   12    92k   27    7  20%      0 |  -2.4 -46.9    3   746k  193  407  67%   300k   781n   -6
   1 4607 |  12   4    8    36k   28    8  22%      0 |  -2.3 -47.1    4   598k  195  405  67%   300k  -813n   -6

          |--------------------DL---------------------|-------------------------UL------------------------------
 pci rnti | cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  rsrp  mcs  brate   ok  nok  (%)    bsr     ta  phr
   1 4607 |  11   4   12    34k   23   12  34%      0 |  -2.2 -46.8    3   753k  196  404  67%   300k   789n   -6
   1 4607 |   1   1    8    20k   17    7  29%      0 |  -2.5 -47.2    2   435k  191  409  68%   300k   -56n   -4
   1 4607 |  13   3    7    37k   24   19  44%      0 |  -1.8 -46.1    5   607k  199  401  66%   300k   -81n   -6
   1 4607 |   2   1   11    17k   15    5  25%      0 |  -2.3 -46.2    2   442k  194  406  67%   300k   423n   -5
q

Here's the "git log" output:


antevia@dusupermicr:~$ cd ~/srsRAN_Project
antevia@dusupermicr:~/srsRAN_Project$ git log
commit 40b17b429dce2627921d099e8a57674e6f2d82d1 (HEAD -> test, origin/test)
Merge: c33cacba7 480e33432
Author: codebot <group_11283275_bot_29fb07365fb84a160b31ccf01e2e2aa2@noreply.gitlab.com>
Date:   Mon Jun 17 10:23:05 2024 +0000

    Update main

    # Conflicts:
    #       README.md
    #       apps/examples/du/CMakeLists.txt
    #       apps/examples/du/du_example.cpp
    #       apps/examples/du/fapi_factory.cpp
    #       apps/examples/du/fapi_factory.h
    #       apps/examples/du/phy_factory.cpp
    #       apps/examples/du/phy_factory.h
    #       apps/examples/du/radio_notifier_sample.h
    #       apps/services/console_helper.cpp
    #       apps/services/console_helper.h
    #       include/srsran/pdcp/pdcp_tx_pdu.h
    #       include/srsran/support/signal_handler.h
    #       lib/support/signal_handler.cpp

commit 480e33432e20990f4555209ea8f357a3b7cd0148
Author: faluco <borja@srs.io>
Date:   Mon Jun 17 10:38:14 2024 +0200

    Disable array-bounds checking without assertions to avoid false positives detected when compiling with GCC

commit 0dd6428f5a6a4b4230e076a2778c5fde88b42ff1
herlesupreeth commented 5 months ago

Thank you for the logs. In the latest logs I dont see any issue from scheduler perspective as I see the following warnings in the OFH. I will let my colleagues to take a look at what the issue is.

@sauka can you please take a look what below warning means?

2024-06-17T11:37:39.002282 [OFH     ] [W] Real-time timing worker woke up late, sleep time has been '571us', or equivalently, '16' symbols
2024-06-17T11:37:39.002303 [SCHED   ] [D] [   564.9] Slot decisions pci=1 t=4us (0 PDSCHs, 0 PUSCHs):
- PUCCH: c-rnti=0x4607 format=1 prb=[77..78) symb=[0..14) cs=0 occ=1 uci: harq_bits=0 sr=1
2024-06-17T11:37:39.002871 [OFH     ] [W] Real-time timing worker woke up late, sleep time has been '607us', or equivalently, '17' symbols
ismagom commented 5 months ago

This is not a bug. Please use the discussion forum to get advice from the community on how to tune the configuration to achieve high performance.