search

srsran / srsRAN_4G

Open source SDR 4G software suite from Software Radio Systems (SRS) https://docs.srsran.com/projects/4g
https://www.srsran.com
GNU Affero General Public License v3.0
3.42k stars 1.13k forks source link

make test errors #983

Open beruysaliz opened 1 year ago

beruysaliz commented 1 year ago

Issue Description

I downloaded and installed srsRAN as in doc, but while making test took 2 errors. 876:benchmark_radio_multi_rf 877:test_radio_rt_gain_zmq LastTest.log

Setup Details

[Specify details of the test setup. This would help us reproduce the problem reliably] e.g. Network configuration, Operation System, Hardware, RF front-end, library and driver versions Ubuntu 18.04, uhd 3.15.0 RF front-end installed

Expected Behavior

[What you expect to happen] without any errors

Actual Behaviour

[What happens instead e.g. error message]

Steps to reproduce the problem

[Tell us how to reproduce this issue e.g. RF setup, application config files]

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue]

robertfalkenberg commented 1 year ago

Could you please provide the output of cmake and the build process when building in a clean directory?

beruysaliz commented 1 year ago

CMakeOutput.log CMakeError.log build.log I attached the files robert.

robertfalkenberg commented 1 year ago

Thank you very much. Could you please also provide me with the output of the following commands, executed in the build directory:

echo $LD_LIBRARY_PATH

readelf -d lib/src/phy/rf/libsrsran_rf.so | head -20

readelf -d lib/src/radio/test/benchmark_radio | head -20

Note that readelf is part of the binutils-common package: https://manpages.ubuntu.com/manpages/bionic/man1/aarch64-linux-gnu-readelf.1.html

beruysaliz commented 1 year ago

1) echo $LD_LIBRARY_PATH ==> /usr/local/lib 2) Dynamic section at offset 0x33da0 contains 30 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000000e (SONAME) Library soname: [libsrsran_rf.so.0] 0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/:] 0x000000000000000c (INIT) 0x5798 0x000000000000000d (FINI) 0x2f894 0x0000000000000019 (INIT_ARRAY) 0x233d68 0x000000000000001b (INIT_ARRAYSZ) 16 (bytes) 0x000000000000001a (FINI_ARRAY) 0x233d78 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x1f0 0x0000000000000005 (STRTAB) 0x2928 0x0000000000000006 (SYMTAB) 0xb40 0x000000000000000a (STRSZ) 6545 (bytes)

3) Dynamic section at offset 0x7b6f8 contains 36 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libsrsgui.so] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libsrsran_rf.so.0] 0x0000000000000001 (NEEDED) Shared library: [libfftw3f.so.3] 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2] 0x000000000000001d (RUNPATH) Library runpath: [/home/hiwi-pc/Documents/srsRAN-master/build/lib/src/phy/rf:/usr/local/lib] 0x000000000000000c (INIT) 0x85d8 0x000000000000000d (FINI) 0x6c0b4 0x0000000000000019 (INIT_ARRAY) 0x27ad88 0x000000000000001b (INIT_ARRAYSZ) 32 (bytes) 0x000000000000001a (FINI_ARRAY) 0x27ada8 0x000000000000001c (FINI_ARRAYSZ) 16 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x298

robertfalkenberg commented 1 year ago

It looks like for some reason $LD_LIBRARY_PATH was set globally. Is this intentional?

The search paths under $LD_LIBRARY_PATH have priority over all other directories. In addition, there also appears to be an older version of srsRAN installed at the specified search path.

This causes a wrong version from /usr/local/lib to be loaded instead of the RF library in the build directory. Fortunately the versions are not compatible, otherwise even stranger errors would be expected.

I recommend either removing old the installation or (better) resetting the $LD_LIBRARY_PATH variable.

beruysaliz commented 1 year ago

for UHD version 3.15 installation I followed that guide https://kb.ettus.com/Building_and_Installing_the_USRP_Open-SourceToolchain(UHD_and_GNU_Radio)_on_Linux and it says Finally, make sure that the LD_LIBRARY_PATH environment variable is defined and includes the folder under which UHD was installed. Most commonly, you can add the line below to the end of your $HOME/.bashrc file: I added the environment variable at the end of the bashrc file.

  1. how can I completely delete the old version ?
  2. do you mean with "resetting the $LD_LIBRARY_PATH variable" basically delete that line from bashrc file ?
robertfalkenberg commented 1 year ago

1.

To uninstall the old version, go to the build directory of the old version and run sudo make uninstall. If the build directory is gone, just download the old version, build, install, and finally uninstall it.

In your case, most probably it will be sufficient to just delete the files /usr/local/lib/libsrsran_rf*.

In case you are working on a shared workstation, please note that this will uninstall/break the old version for all other users of your system.

2.

do you mean with "resetting the $LD_LIBRARY_PATH variable" basically delete that line from bashrc file ?

Exactly, remove that line from bashrc. But in case that your system requires that variable to be set as is, we cannot go this way.

3.

A last option is to compile the current version of srsRAN with the following flag (in a clean build directory): cmake .... -DENABLE_RF_PLUGINS=False ...., which replaces dynamic RF plugins by fixed ones.

aniee01 commented 1 year ago

I have installed srsRAN on ubuntu 22.04. while running "make test" i got the error

858: test_radio_rt_gain_zmq

***Exception: SegFault 0.48 sec

  1. the command you asked to run echo $LD_LIBRARY_PATH did nothing. i didnt got anything in its output.

Below is the output of other two commands.

  1. root@srsue:/usr/src/srsRAN/build# readelf -d lib/src/phy/rf/libsrsran_rf.so | head -20 Dynamic section at offset 0x2fda8 contains 30 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 2 \4

  3. 4510263+.

  4. 0x000000000000000e (SONAME) Library soname: [libsrsran_rf.so.0] 0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/:] 0x000000000000000c (INIT) 0x6000 0x000000000000000d (FINI) 0x2ade8 0x0000000000000019 (INIT_ARRAY) 0x30d78 0x000000000000001b (INIT_ARRAYSZ) 16 (bytes) 0x000000000000001a (FINI_ARRAY) 0x30d88 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x2f0 0x0000000000000005 (STRTAB) 0x2898 0x0000000000000006 (SYMTAB) 0xb28 0x000000000000000a (STRSZ) 6515 (bytes)

  5. root@srsue:/usr/src/srsRAN/build# readelf -d lib/src/radio/test/benchmark_radio | head -20 Dynamic section at offset 0x7c6f0 contains 35 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libsrsran_rf.so.0] 0x0000000000000001 (NEEDED) Shared library: [libfftw3f.so.3] 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2] 0x000000000000001d (RUNPATH) Library runpath: [/usr/src/srsRAN/build/lib/src/phy/rf] 0x000000000000000c (INIT) 0x9000 0x000000000000000d (FINI) 0x6adf8 0x0000000000000019 (INIT_ARRAY) 0x7cd80 0x000000000000001b (INIT_ARRAYSZ) 32 (bytes) 0x000000000000001a (FINI_ARRAY) 0x7cda0 0x000000000000001c (FINI_ARRAYSZ) 16 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x3a0 0x0000000000000005 (STRTAB) 0x2ce0

how to resolve this issue. kindly guide me. Regards

robertfalkenberg commented 1 year ago

Could you please provide the output of cmake and the build process when building in a clean directory? Please upload LastTest.log of the failed testrun.

Also please share the output of

free -h

and the file cpuinfo.txt produced by this command

cat /proc/cpuinfo > cpuinfo.txt
aniee01 commented 1 year ago

thank you for helping me out. i just re-run the cmake command and everything started working fine.

humble Regards

chaohwa4 commented 1 year ago

I ran with SRS RAN Release 22.10.0. I ran "make test" with ADI PackRF using 2021-07-28-ADI-Kuiper. I have the following errors:

The following tests FAILED: 556 - rf_file_test (Failed) 857 - benchmark_radio_multi_rf (Child killed) 860 - npdsch_npdcch_file1 (Failed) 861 - npdsch_npdcch_file2 (Failed) 863 - npdsch_npdcch_file3 (Failed) 874 - ngap_asn1_test (Child aborted) 1369 - file_sink_test (Failed) 1370 - file_utils_test (Failed)

What should I do now?

chaohwa4 commented 1 year ago

Here are the three files I have. CMakeOutput.log CMakeError.log build.log

chaohwa4 commented 1 year ago

I ran another "make test" on my second PackRF and got different results:

The following tests FAILED: 556 - rf_file_test (Failed) 860 - npdsch_npdcch_file1 (Failed) 861 - npdsch_npdcch_file2 (Failed) 863 - npdsch_npdcch_file3 (Failed) 874 - ngap_asn1_test (Child aborted) 887 - network_utils_test (Child aborted) 1369 - file_sink_test (Failed) 1370 - file_utils_test (Failed) 1462 - s1ap_test (Child aborted) 1470 - ngap_test (Child aborted)

CMakeError.log CMakeOutput.log

The build.log is attached in my previous response.

robertfalkenberg commented 1 year ago

Please also share the remaining infos listed in https://github.com/srsran/srsRAN_4G/issues/983#issuecomment-1283526535

robertfalkenberg commented 1 year ago

And please also

ulimit -a
chaohwa4 commented 1 year ago

Here are the files you requested: lastTest.log

root@analog:~/srsRAN/build# free -h total used free shared buff/cache available Mem: 997Mi 64Mi 729Mi 6.0Mi 204Mi 869Mi Swap: 49Mi 0B 49Mi

cpuinfo.txt

root@analog:~/srsRAN/build# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 6948 max locked memory (kbytes, -l) 65536 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 6948 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

robertfalkenberg commented 1 year ago

Thanks for the feedback. max locked memory is restricted to 65536. Please try this: https://github.com/srsran/srsRAN_4G/issues/881#issuecomment-1247971384

robertfalkenberg commented 1 year ago

If tests are still failing, please share the results of:

ctest --rerun-failed --output-on-failure
chaohwa4 commented 1 year ago

I typed "/etc/security/limits.conf" to add "* - memlock unlimited" and saved it. After typing "ulimit -l", the number is still 65536. I also rebooted the PackRF and the number doesn't change at all after typing "ulimit -l".

chaohwa4 commented 1 year ago

I type "ulimit -l unlimited" to make it unlimited.

After typing "ulimit -a", root@analog:~/srsRAN/build# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 6948 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 6948 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

chaohwa4 commented 1 year ago

After making the maximum locked memory memory, I ran "make test" and there were 5 failed tests. The following tests FAILED: 857 - benchmark_radio_multi_rf (Child killed) 874 - ngap_asn1_test (Child aborted) 887 - network_utils_test (Child aborted) 1462 - s1ap_test (Child aborted) 1470 - ngap_test (Child aborted) Errors while running CTest

The details are in lastTest_022823.log.

As requested, I ran "ctest --rerun-failed --output-on-failure". The details are in ctestRerun_022823.log

robertfalkenberg commented 1 year ago

It appears that the remaining failures are related to missing sctp kernel modules. We noticed, that on some ARM-based systems, sctp kernel modules are not included by default and have to be installed separately. Here is an example for RaspberryPi: https://docs.srsran.com/projects/4g/en/latest/app_notes/source/pi4/source/index.html?highlight=linux-modules-extra-raspi#running-on-ubuntu-22-04-lts

chaohwa4 commented 1 year ago

Currently I am using the Kuiper Linux provided by Analog Devices. The version is 2019_R2. I tried to use "sudo apt-get install linux-modules-extra-raspi", but it does not work.

root@analog:~# sudo apt-get install linux-modules-extra-raspi Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package linux-modules-extra-raspi

So I built the Linux kernels with sctp kernel modules using adi-5.10 to generate the new uImage and devicetree.dtb. Now testing after changing both the uImage and devicetree.dtb files.

chaohwa4 commented 1 year ago

In my PackRF, the OS is Analog Devices Kuiper Linux 2019_R2, which is the fork of Raspberry Pi OS 10 (Buster) with Linux Kernel 4.19 and GCC version 8.3. The PackRF is Zynq Z7035 with AD9361. PackRF is the multichannel version of Pluto.

In order to add SCTP, I changed both uImage and devicetree.dtb and used Linux Kernel 5.10. After setting ulimit -l as unlimited, I ran "make test" and still had 8 errors. The following tests FAILED: 556 - rf_file_test (Failed) 857 - benchmark_radio_multi_rf (Child killed) 860 - npdsch_npdcch_file1 (Failed) 861 - npdsch_npdcch_file2 (Failed) 863 - npdsch_npdcch_file3 (Failed) 874 - ngap_asn1_test (Child aborted) 1369 - file_sink_test (Failed) 1370 - file_utils_test (Failed) 1462 - s1ap_test (Child aborted) Errors while running CTest make: *** [Makefile:152: test] Error 8

The details are in lastTest_030223.log.

robertfalkenberg commented 1 year ago

I think the following tests are related to max locked memory again:

556 - rf_file_test (Failed)
860 - npdsch_npdcch_file1 (Failed)
861 - npdsch_npdcch_file2 (Failed)
863 - npdsch_npdcch_file3 (Failed)
1369 - file_sink_test (Failed)
1370 - file_utils_test (Failed)

for the others, please share

ctest --rerun-failed --output-on-failure

since the default output does not contain sufficient info on the cause of the failure.

robertfalkenberg commented 1 year ago

According to your previous post about the RAM:

root@analog:~/srsRAN/build# free -h
total used free shared buff/cache available
Mem: 997Mi 64Mi 729Mi 6.0Mi 204Mi 869Mi
Swap: 49Mi 0B 49Mi

you might run out of memory in the other tests.

For example, the memory footprint of benchmark_radio_multi_rf (tested on a local x86 machine) is as follows:

Maximum resident set size (kbytes): 983460

Measured via:

/usr/bin/time -v ./benchmark_radio "-d" "zmq" "-a" "tx_port=tcp://*:2000,rx_port=tcp://localhost:2000;tx_port=tcp://*:2001,rx_port=tcp://localhost:2001;tx_port=tcp://*:2002,rx_port=tcp://localhost:2002;tx_port=tcp://*:2003,rx_port=tcp://localhost:2003;" "-p" "4"
chaohwa4 commented 1 year ago

After typing ctest --rerun-failed --output-on-failure The results are shown in ctestRerun_0510_030523.log

The maximum locked memory is unlimited. As shown in root@analog:~/srsRAN/build# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 6921 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 6921 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

chaohwa4 commented 1 year ago

Since my PackRF (AD9361/Zynq 7035) has only 1GB of memory, what should I do to prevent the out-of-memory issue?

robertfalkenberg commented 1 year ago

You could disable/skip that test. However, I'm afraid that srsue or srsenb require similar amounts of memory.

chaohwa4 commented 1 year ago

So if I run srsRAN on a Linux destop with at least 8GB of RAM and use my PackRFs as RF devices, will it be doable?

chaohwa4 commented 1 year ago

I want to run a Ubuntu laptop as EPC, which has 8GB of RAM. After installing srsRAN 4G and running "make test", there are no errors at all.

robertfalkenberg commented 1 year ago

So if I run srsRAN on a Linux destop with at least 8GB of RAM and use my PackRFs as RF devices, will it be doable?

Still, srsue or srsenb will most likely crash due to OOM after random time when run on your PackRFs.