search

srsran / srsRAN_4G

Open source SDR 4G software suite from Software Radio Systems (SRS) https://docs.srsran.com/projects/4g
https://www.srsran.com
GNU Affero General Public License v3.0
3.37k stars 1.11k forks source link

srsLTE crashed #623

Closed avalori1 closed 3 years ago

avalori1 commented 3 years ago

srsLTE crashes after fairly long time running OK

Issue Description

crash

Setup Details

~/srsLTE/srsepc$ srsenb --version --- Software Radio Systems LTE eNodeB --- Version 20.10.1

But strangely, can't get the version of srsepc?

~/srsLTE/srsepc$ srsepc --version Built in Release mode using commit 45486b6 on branch master. --- Software Radio Systems EPC --- Reading configuration file /home/bts/.config/srslte/epc.conf...

Expected Behavior

keep running

Actual Behaviour

crash

Steps to reproduce the problem

Just wait, runs for a fairly long time but then crash

Additional Information

srsLTE.backtrace.crash

--- command='srsenb' version=20.10.1 signal=11 date='05/02/2021 08:03:33' --- srsenb(+0x17bc33) [0x55cc1d3eac33] /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f4138143040] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base+0x13) [0x7f4138b54fa3] srsenb(+0x186d11) [0x55cc1d3f5d11] srsenb(+0x5ea1e) [0x55cc1d2cda1e] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4139deb6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f413822571f]

--- command='srsenb' version=20.10.1 signal=6 date='05/02/2021 08:03:34' --- srsenb(+0x17bc33) [0x55cc1d3eac33] /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f4138143040] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f4138142fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f4138144921] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957) [0x7f4138b37957] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7f4138b3dae6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7f4138b3db21] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x938ff) [0x7f4138b3e8ff] srsenb(+0xd6e43) [0x55cc1d345e43] srsenb(+0xcc70f) [0x55cc1d33b70f] srsenb(+0xc7bf2) [0x55cc1d336bf2] srsenb(+0xc7de0) [0x55cc1d336de0] srsenb(+0xb76b1) [0x55cc1d3266b1] srsenb(+0x826f5) [0x55cc1d2f16f5] srsenb(+0x189ba1) [0x55cc1d3f8ba1] srsenb(+0x5ea1e) [0x55cc1d2cda1e] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4139deb6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f413822571f]

--- command='srsenb' version=20.10.1 signal=11 date='05/02/2021 12:51:47' --- srsenb(+0x17bc33) [0x56300b261c33] /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f8e47ab7040] srsenb(+0xb8868) [0x56300b19e868] srsenb(+0x8274c) [0x56300b16874c] srsenb(+0x189ba1) [0x56300b26fba1] srsenb(+0x5ea1e) [0x56300b144a1e] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f8e4975f6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8e47b9971f]

andrepuschmann commented 3 years ago

Thanks for reporting the issue. Unfortunately, without debug symbols it's hard to tell what's going on. Could you build with debug symbols perhaps and run again. If it crashes, please share the backtrace.

avalori1 commented 3 years ago

Dear Andre, sorry for my ignorance... a novice on this. How can I do that? Is an option to be added at the make step in this instructions? https://docs.srslte.com/en/latest/general/source/1_installation.html

Thanks

On Mon, 8 Feb 2021 at 21:23, Andre Puschmann notifications@github.com wrote:

Thanks for reporting the issue. Unfortunately, without debug symbols it's hard to tell what's going on. Could you build with debug symbols perhaps and run again. If it crashes, please share the backtrace.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/srsLTE/srsLTE/issues/623#issuecomment-775424625, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARPP3JEQLT2A7MAWE3U7DXDS6BB4VANCNFSM4XEYLTFA .

avalori1 commented 3 years ago

@andrepuschmann would be possible for you to point me in the direction of instruction to do so, so I can collect some more information?

Thanks

andrepuschmann commented 3 years ago

We work on instructions for that. I hope to get something out shortly. Quick tip is to use cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ... Clean up the build folder before that and run a clean build. Then when it crashes, post the ./srsLTE_backtrace.log file

avalori1 commented 3 years ago

Thanks for your tip!I think I managed to build as instructed. It seems that there is a bit more info coming up but could not see the ./srsLTE_backtrace.log file

The info I got are: In the terminal

RACH: tti=5331, cc=0, preamble=3, offset=0, temp_crnti=0x46 User 0x46 connected terminate called after throwing an instance of 'std::length_error' what(): basic_string::_M_create srsLTE crashed... backtrace saved in './srsLTE.backtrace.crash'... --- exiting ---

And in the srsLTE.backtrace.crash file --- command='srsenb' version=20.10.1 signal=6 date='11/02/2021 21:27:46' --- srsenb(+0x162327) [0x55f0252d4327] /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f2c8f8e5040] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f2c8f8e4fb7] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f2c8f8e6921] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957) [0x7f2c902d9957] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7f2c902dfae6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7f2c902dfb21] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d54) [0x7f2c902dfd54] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8e811) [0x7f2c902db811] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x124799) [0x7f2c90371799] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_mutateEmmPKcm+0x6b) [0x7f2c90371b8b] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm+0x1cb) [0x7f2c9037294b] /usr/local/lib/libsrslte_rf.so.0(+0x29f12) [0x7f2c90a0cf12] /usr/local/lib/libsrslte_rf.so.0(rf_uhd_send_timed_multi+0x328) [0x7f2c909f8998] /usr/local/lib/libsrslte_rf.so.0(srslte_rf_send_timed_multi+0x29) [0x7f2c909f6ab9] srsenb(+0x245678) [0x55f0253b7678] srsenb(+0x2479f1) [0x55f0253b99f1] srsenb(+0x76ae0) [0x55f0251e8ae0] srsenb(+0x7f5e6) [0x55f0251f15e6] srsenb(+0x16e389) [0x55f0252e0389] srsenb(+0x5f59e) [0x55f0251d159e] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2c918216db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f2c8f9c771f]

avalori1 commented 3 years ago

Dear @andrepuschmann

Please find another example of crash... If you have recommendations on how to gather more useful data, I will be happy to do so. Not sure if this is of any usefulness, but I noticed that in the terminal I see twice the crash message...

srsLTE crashed... backtrace saved in './srsLTE.backtrace.crash'...
srsLTE crashed... backtrace saved in './srsLTE.backtrace.crash'...
--- exiting ---
--- exiting ---

In the srsLTE.backtrace.crash file

--- command='srsenb' version=20.10.1 signal=11 date='13/02/2021 16:15:04' ---
    srsenb(+0x162327) [0x561ed2eb6327]
    /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f2f2123e040]
    /lib/x86_64-linux-gnu/libusb-1.0.so.0(libusb_submit_transfer+0x1c5) [0x7f2f1b330a95]
    /usr/local/lib/libuhd.so.3.15.0(+0x86922c) [0x7f2f1f29122c]
    /usr/local/lib/libuhd.so.3.15.0(+0x869548) [0x7f2f1f291548]
    /usr/local/lib/libuhd.so.3.15.0(+0x868515) [0x7f2f1f290515]
    /usr/local/lib/libuhd.so.3.15.0(+0x5ad1f3) [0x7f2f1efd51f3]
    /usr/local/lib/libsrslte_rf.so.0(+0x29eee) [0x7f2f22365eee]
    /usr/local/lib/libsrslte_rf.so.0(rf_uhd_send_timed_multi+0x328) [0x7f2f22351998]
    /usr/local/lib/libsrslte_rf.so.0(srslte_rf_send_timed_multi+0x29) [0x7f2f2234fab9]
    srsenb(+0x245678) [0x561ed2f99678]
    srsenb(+0x2479f1) [0x561ed2f9b9f1]
    srsenb(+0x76ae0) [0x561ed2dcaae0]
    srsenb(+0x7f5e6) [0x561ed2dd35e6]
    srsenb(+0x16e389) [0x561ed2ec2389]
    srsenb(+0x5f59e) [0x561ed2db359e]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2f2317a6db]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f2f2132071f]
andrepuschmann commented 3 years ago

The crash seems to come from the PHY layer interacting with the radio. But it's hard to tell how from the logs/info we have, the backtrace isn't very useful either. Could you perhaps try the following: cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_ASAN=True .., again wipe the build folder and compile again. Please send stdout when you run cmake. This command enables AddressSanitizer. This will slow down the execution and maybe the eNB doesn't work anymore. But you said the eNB crashed eventually without doing stuff, right? If so, just leave it running like before and see what happens. Thanks

avalori1 commented 3 years ago

Andre, The one below is the output while running cmake using the option you emailed me.

I am not sure what you mean with "But you said the eNB crashed eventually without doing stuff, right?"... I mean, there was always a UE connected doing some traffic... I did not try to leave it running with no UE connected.

As you guessed, the eNB seems not working now. I see the network on the UE but can't attach to it. I will leave the eNB running overnight and will let you know tomorrow if it crashed or not

Regards

bts@bts-ThinkPad-L460:~/srsLTE/build$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_ASAN=True ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_SYSTEM: Linux-5.4.0-65-generic
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- CMAKE_CXX_COMPILER: /usr/bin/c++
-- Enabling ccache for C
-- Enabling ccache for CXX
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
-- Checking for module 'fftw3f >= 3.0'
--   Found fftw3f , version 3.3.7
-- FFTW3F LIBRARIES: /usr/lib/x86_64-linux-gnu/libfftw3f.so
-- FFTW3F STATIC LIBRARIES: /usr/lib/x86_64-linux-gnu/libfftw3f.a
-- FFTW3F INCLUDE DIRS: /usr/include
-- Found fftw3f: /usr/lib/x86_64-linux-gnu/libfftw3f.so  
-- FFT_LIBRARIES: /usr/lib/x86_64-linux-gnu/libfftw3f.so
-- Checking for module 'polarssl'
--   No package 'polarssl' found
-- POLARSSL LIBRARIES: POLARSSL_LIBRARIES-NOTFOUND
-- POLARSSL STATIC LIBRARIES: POLARSSL_STATIC_LIBRARIES-NOTFOUND
-- POLARSSL INCLUDE DIRS: POLARSSL_INCLUDE_DIRS-NOTFOUND
-- Could NOT find POLARSSL (missing: POLARSSL_LIBRARIES POLARSSL_INCLUDE_DIRS) 
-- Checking for module 'mbedtls'
--   No package 'mbedtls' found
-- MBEDTLS LIBRARIES: /usr/lib/x86_64-linux-gnu/libmbedcrypto.so
-- MBEDTLS STATIC LIBRARIES: /usr/lib/x86_64-linux-gnu/libmbedcrypto.a
-- MBEDTLS INCLUDE DIRS: /usr/include
-- Found MBEDTLS: /usr/lib/x86_64-linux-gnu/libmbedcrypto.so  
-- Checking for module 'libpcsclite'
--   Found libpcsclite, version 1.8.23
-- Found PCSCLITE: /usr/lib/x86_64-linux-gnu/libpcsclite.so  
-- PCSC LIBRARIES: /usr/lib/x86_64-linux-gnu/libpcsclite.so
-- PCSC INCLUDE DIRS: /usr/include/PCSC
-- Building with PCSC support.
-- UHD LIBRARIES /usr/local/lib/libuhd.so
-- UHD INCLUDE DIRS /usr/local/include
-- Found UHD: /usr/local/lib/libuhd.so  
-- Performing Test UHD_ENABLE_X300_FW_RESET
-- Performing Test UHD_ENABLE_X300_FW_RESET - Failed
-- Performing Test UHD_ENABLE_RFNOC
-- Performing Test UHD_ENABLE_RFNOC - Success
-- Performing Test UHD_ENABLE_CUSTOM_RFNOC
-- Performing Test UHD_ENABLE_CUSTOM_RFNOC - Failed
-- Checking for module 'libbladeRF'
--   Found libbladeRF, version 2.2.0-2018.12-rc3-2-ppabionic
-- Looking for bladerf_get_board_name in bladeRF
-- Looking for bladerf_get_board_name in bladeRF - found
-- Found libbladeRF: /usr/include, /usr/lib/x86_64-linux-gnu/libbladeRF.so
-- FINDING SOAPY.
-- Checking for module 'SoapySDR'
--   No package 'SoapySDR' found
-- libSOAPYSDR not found.
-- FINDING ZEROMQ.
-- Checking for module 'ZeroMQ'
--   No package 'ZeroMQ' found
-- Found libZEROMQ: /usr/include, /usr/lib/x86_64-linux-gnu/libzmq.so
-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   program_options
-- SRSGUI LIBRARIES /usr/local/lib/libsrsgui.so
-- SRSGUI INCLUDE DIRS /usr/local/include
-- Found SRSGUI: /usr/local/lib/libsrsgui.so  
-- Performing Test HAVE_WNO_UNUSED_BUT_SET_VARIABLE
-- Performing Test HAVE_WNO_UNUSED_BUT_SET_VARIABLE - Success
-- Performing Test HAVE_SSE
-- Performing Test HAVE_SSE - Success
-- SSE4.1 is enabled - target CPU must support it
-- Performing Test HAVE_AVX
-- Performing Test HAVE_AVX - Success
-- AVX is enabled - target CPU must support it
-- Performing Test HAVE_AVX2
-- Performing Test HAVE_AVX2 - Success
-- AVX2 is enabled - target CPU must support it
-- Performing Test HAVE_FMA
-- Performing Test HAVE_FMA - Success
-- FMA is enabled - target CPU must support it
-- Performing Test HAVE_AVX512
-- Performing Test HAVE_AVX512 - Failed
-- Performing Test HAVE_VISIBILITY_HIDDEN_CXX
-- Performing Test HAVE_VISIBILITY_HIDDEN_CXX - Success
-- Performing Test HAVE_ERROR_INCOMPATIBLE
-- Performing Test HAVE_ERROR_INCOMPATIBLE - Success
-- SSE4.1 is enabled - target CPU must support it
-- AVX is enabled - target CPU must support it
-- AVX2 is enabled - target CPU must support it
-- FMA is enabled - target CPU must support it
-- Performing Test HAVE_VISIBILITY_HIDDEN_C
-- Performing Test HAVE_VISIBILITY_HIDDEN_C - Success
-- CMAKE_C_FLAGS is  -Wno-unused-but-set-variable -Werror=incompatible-pointer-types -Wall -Wno-comment -Wno-write-strings -Wno-unused-result -Wformat -Wmissing-field-initializers -Wtype-limits -std=c99 -fno-strict-aliasing -D_GNU_SOURCE -ggdb -DBUILD_TYPE_RELWITHDEBINFO -march=native -mfpmath=sse -mavx2 -DLV_HAVE_AVX2 -DLV_HAVE_AVX -DLV_HAVE_SSE -mfma -DLV_HAVE_FMA -Ofast -funroll-loops -fvisibility=hidden -fsanitize=address -fno-omit-frame-pointer -Werror --param large-function-growth=1600
-- CMAKE_CXX_FLAGS is  -Wall -Wno-comment -Wno-reorder -Wno-unused-variable -Wtype-limits -std=c++11 -fno-strict-aliasing -Wno-unused-but-set-variable -march=native -mfpmath=sse -mavx2 -DLV_HAVE_AVX2 -DLV_HAVE_AVX -DLV_HAVE_SSE -fvisibility=hidden -ggdb -DBUILD_TYPE_RELWITHDEBINFO -fsanitize=address -fno-omit-frame-pointer -Werror
-- Using install prefix: /usr/local
-- Building for version: 20.10.1
--    examples will be installed.
-- Checking for module 'sctp'
--   No package 'sctp' found
-- SCTP LIBRARIES: /usr/lib/x86_64-linux-gnu/libsctp.so
-- SCTP INCLUDE DIRS: /usr/include
-- Found SCTP: /usr/lib/x86_64-linux-gnu/libsctp.so  
-- No post-build command defined
-- Building with srsUE
-- No post-build-UE command defined
-- No post-build command defined
-- Building with srsENB
-- Found LibConfig++: /usr/lib/x86_64-linux-gnu/libconfig++.so
-- static LibConfig++ path: /usr/lib/x86_64-linux-gnu/libconfig++.a
-- Found LibConfig: /usr/lib/x86_64-linux-gnu/libconfig.so
-- static LibConfig path: /usr/lib/x86_64-linux-gnu/libconfig.a
-- Checking for module 'sctp'
--   No package 'sctp' found
-- SCTP LIBRARIES: /usr/lib/x86_64-linux-gnu/libsctp.so
-- SCTP INCLUDE DIRS: /usr/include
-- No post-build-ENB command defined
-- Building with srsEPC
-- Found LibConfig++: /usr/lib/x86_64-linux-gnu/libconfig++.so
-- static LibConfig++ path: /usr/lib/x86_64-linux-gnu/libconfig++.a
-- Found LibConfig: /usr/lib/x86_64-linux-gnu/libconfig.so
-- static LibConfig path: /usr/lib/x86_64-linux-gnu/libconfig.a
-- Checking for module 'sctp'
--   No package 'sctp' found
-- SCTP LIBRARIES: /usr/lib/x86_64-linux-gnu/libsctp.so
-- SCTP INCLUDE DIRS: /usr/include
-- No post-build-EPC command defined
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bts/srsLTE/build
avalori1 commented 3 years ago

@andrepuschmann

the eNB crashed (fairly quickly) and this time indeed spitted out a lot of info! I copied as HTML to keep the formatting (although the colour coding was lost).

Regards

 264 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 265 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 266 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 281 0.10   0 2.0     0    0   89 100%    0  0.0   0     0    0    0   0%   0.0
 283 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 285 3.0   0 0.0     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 286 10.0   0 2.7     0    0   97 100%    0  0.0   0     0    0    0   0%   0.0
 391 0.10   0 2.0     0    0   89 100%    0  0.0   0     0    0    0   0%   0.0
 3b1  11   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 494  12   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_M_create
srsLTE crashed... backtrace saved in './srsLTE.backtrace.crash'...
---  exiting  ---
=================================================================
==26751==ERROR: AddressSanitizer: heap-use-after-free on address 0x612000000a20 at pc 0x558355248a37 bp 0x7f4ccc0527a0 sp 0x7f4ccc052790
READ of size 1 at 0x612000000a20 thread T25
    #0 0x558355248a36 in srslog::detail::shared_variable<bool>::operator bool() const /home/bts/srsLTE/lib/include/srslte/srslog/detail/support/thread_utils.h:99
    #1 0x558355248a36 in srslog::log_channel::enabled() const /home/bts/srsLTE/lib/include/srslte/srslog/log_channel.h:87
    #2 0x558355248a36 in void srslog::log_channel::operator()<char*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char*&&) /home/bts/srsLTE/lib/include/srslte/srslog/log_channel.h:104
    #3 0x558355248a36 in srslte::srslog_wrapper::log(std::unique_ptr<srslte::logger::log_str, srslte::logger::log_str_deleter>) /home/bts/srsLTE/lib/src/common/logger_srslog_wrapper.cc:29
    #4 0x558355236fdb in srslte::log_filter::all_log(srslte::LOG_LEVEL_ENUM, unsigned int, char*, unsigned char const*, int, bool) /home/bts/srsLTE/lib/src/common/log_filter.cc:138
    #5 0x55835523cf8a in srslte::log_filter::info(char const*, ...) /home/bts/srsLTE/lib/src/common/log_filter.cc:181
    #6 0x5583554ffeb1 in srslte::radio::handle_rf_msg(srslte_rf_error_t) /home/bts/srsLTE/lib/src/radio/radio.cc:957
    #7 0x7f4cddb2ac68 in log_underflow /home/bts/srsLTE/lib/src/phy/rf/rf_uhd_imp.cc:220
    #8 0x7f4cddb2ac68 in async_thread /home/bts/srsLTE/lib/src/phy/rf/rf_uhd_imp.cc:263
    #9 0x7f4cddb3c315 in void* std::__invoke_impl<void*, void* (*)(void*), rf_uhd_handler_t*>(std::__invoke_other, void* (*&&)(void*), rf_uhd_handler_t*&&) /usr/include/c++/7/bits/invoke.h:60
    #10 0x7f4cddb3c315 in std::__invoke_result<void* (*)(void*), rf_uhd_handler_t*>::type std::__invoke<void* (*)(void*), rf_uhd_handler_t*>(void* (*&&)(void*), rf_uhd_handler_t*&&) /usr/include/c++/7/bits/invoke.h:96
    #11 0x7f4cddb3c315 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)())) std::thread::_Invoker<std::tuple<void* (*)(void*), rf_uhd_handler_t*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/7/thread:234
    #12 0x7f4cddb3c315 in std::thread::_Invoker<std::tuple<void* (*)(void*), rf_uhd_handler_t*> >::operator()() /usr/include/c++/7/thread:243
    #13 0x7f4cddb3c315 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void* (*)(void*), rf_uhd_handler_t*> > >::_M_run() /usr/include/c++/7/thread:186
    #14 0x7f4cdd42f6de  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd6de)
    #15 0x7f4cde9fa6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #16 0x7f4cdcaec71e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)

0x612000000a20 is located 224 bytes inside of 280-byte region [0x612000000940,0x612000000a58)
freed by thread T26 (WORKER0) here:
    #0 0x7f4cdecf32c0 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe12c0)
    #1 0x5583551dd529 in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> >::deallocate(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true>*, unsigned long) /usr/include/c++/7/ext/new_allocator.h:125
    #2 0x5583551dd529 in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> > >::deallocate(std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> >&, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true>*, unsigned long) /usr/include/c++/7/bits/alloc_traits.h:462
    #3 0x5583551dd529 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true>*) /usr/include/c++/7/bits/hashtable_policy.h:2086
    #4 0x5583551dd529 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true>*) /usr/include/c++/7/bits/hashtable_policy.h:2097
    #5 0x5583551dd529 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() /usr/include/c++/7/bits/hashtable.h:2032
    #6 0x5583551dd529 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable() /usr/include/c++/7/bits/hashtable.h:1358
    #7 0x5583551ddacb in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, srslog::log_channel, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel> > >::~unordered_map() /usr/include/c++/7/bits/unordered_map.h:101
    #8 0x5583551ddacb in srslog::object_repository<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, srslog::log_channel>::~object_repository() /home/bts/srsLTE/lib/src/srslog/object_repository.h:36
    #9 0x5583551ddacb in srslog::srslog_instance::~srslog_instance() /home/bts/srsLTE/lib/src/srslog/srslog_instance.h:33
    #10 0x7f4cdca0e160  (/lib/x86_64-linux-gnu/libc.so.6+0x43160)

previously allocated by thread T0 here:
    #0 0x7f4cdecf2448 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0448)
    #1 0x5583551c331a in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> >::allocate(unsigned long, void const*) /usr/include/c++/7/ext/new_allocator.h:111
    #2 0x5583551c331a in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> > >::allocate(std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> >&, unsigned long) /usr/include/c++/7/bits/alloc_traits.h:436
    #3 0x5583551c331a in std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true>* std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, true> > >::_M_allocate_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&>&&) /usr/include/c++/7/bits/hashtable_policy.h:2060
    #4 0x5583551c331a in std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_emplace<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&> >(std::integral_constant<bool, true>, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&>&&) /usr/include/c++/7/bits/hashtable.h:1644
    #5 0x5583551c331a in std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::emplace<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&>&&) /usr/include/c++/7/bits/hashtable.h:736
    #6 0x5583551c331a in std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel>, false, true>, bool> std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, srslog::log_channel, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, srslog::log_channel> > >::emplace<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&>&&) /usr/include/c++/7/bits/unordered_map.h:387
    #7 0x5583551c331a in srslog::log_channel& srslog::object_repository<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, srslog::log_channel>::emplace<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&, srslog::detail::log_backend&>&&) /home/bts/srsLTE/lib/src/srslog/object_repository.h:60
    #8 0x5583551c331a in create_and_register_log_channel /home/bts/srsLTE/lib/src/srslog/srslog.cpp:230
    #9 0x5583551c331a in srslog::create_log_channel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, srslog::sink&) /home/bts/srsLTE/lib/src/srslog/srslog.cpp:262
RF status: O=0, U=1, L=0

------DL--------------------------------UL------------------------------------
rnti cqi  ri mcs brate   ok  nok  (%)  snr  phr mcs brate   ok  nok  (%)   bsr
  76 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 247 0.10   0 3.0     0    0   11 100%    0  0.0   0     0    0    0   0%   0.0
 263 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 264 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 265 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 266 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 281 0.10   0 2.0     0    0   11 100%    0  0.0   0     0    0    0   0%   0.0
 283 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 285 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 286 0.10   0 3.0     0    0   11 100%    0  0.0   0     0    0    0   0%   0.0
 391 0.10   0 2.0     0    0   11 100%    0  0.0   0     0    0    0   0%   0.0
 3b1 4.0   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 494 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
    #10 0x558354d88a1c in main /home/bts/srsLTE/srsenb/src/main.cc:479
    #11 0x7f4cdc9ecbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

Thread T25 created by T0 here:
    #0 0x7f4cdec49d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x7f4cdd42f994 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd994)
    #2 0x7f4cddb39699 in rf_uhd_open_multi /home/bts/srsLTE/lib/src/phy/rf/rf_uhd_imp.cc:826
    #3 0x7f4cddb2591d in srslte_rf_open_devname /home/bts/srsLTE/lib/src/phy/rf/rf_imp.c:123
    #4 0x5583554f8ff9 in srslte::radio::open_dev(unsigned int const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/bts/srsLTE/lib/src/radio/radio.cc:453
    #5 0x558355503ed7 in srslte::radio::init(srslte::rf_args_t const&, srslte::phy_interface_radio*) /home/bts/srsLTE/lib/src/radio/radio.cc:145
    #6 0x558354de8e65 in srsenb::enb::init(srsenb::all_args_t const&, srslte::logger*) /home/bts/srsLTE/srsenb/src/enb.cc:88
    #7 0x558354d88d86 in main /home/bts/srsLTE/srsenb/src/main.cc:496
    #8 0x7f4cdc9ecbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

Thread T26 (WORKER0) created by T0 here:
    #0 0x7f4cdec49d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x55835526cc51 in threads_new_rt_cpu /home/bts/srsLTE/lib/src/common/threads.c:148
    #2 0x558355262d72 in srslte::thread_pool::init_worker(unsigned int, srslte::thread_pool::worker*, unsigned int, unsigned int) /home/bts/srsLTE/lib/src/common/thread_pool.cc:86
    #3 0x558354dff9d8 in srsenb::phy::init(srsenb::phy_args_t const&, srsenb::phy_cfg_t const&, srslte::radio_interface_phy*, srsenb::stack_interface_phy_lte*) /home/bts/srsLTE/srsenb/src/phy/phy.cc:122
    #4 0x558354de8ee9 in srsenb::enb::init(srsenb::all_args_t const&, srslte::logger*) /home/bts/srsLTE/srsenb/src/enb.cc:95
    #5 0x558354d88d86 in main /home/bts/srsLTE/srsenb/src/main.cc:496
    #6 0x7f4cdc9ecbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

SUMMARY: AddressSanitizer: heap-use-after-free /home/bts/srsLTE/lib/include/srslte/srslog/detail/support/thread_utils.h:99 in srslog::detail::shared_variable<bool>::operator bool() const
Shadow bytes around the buggy address:
  0x0c247fff80f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c247fff8100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c247fff8110: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c247fff8120: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c247fff8130: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c247fff8140: fd fd fd fd[fd]fd fd fd fd fd fd fa fa fa fa fa
  0x0c247fff8150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c247fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c247fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c247fff8180: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c247fff8190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==26751==ABORTING
bts@bts-ThinkPad-L460:~/srsLTE/build$
andrepuschmann commented 3 years ago

Thanks for the logs. I have the feeling that there is something else going wrong before the issue that ASAN reports. I think we had a similar problem before internally and fixed that in our tree already. But it would be good to double check and backport (if that's the case) to 20.10. For that I think it would be best is if we could reproduce the issue ourselves. What exactly are you doing before the crash happens? I see the RNTIs of the UEs is quite high, are you attaching many UEs? What system is this running on? Can you please provide the info that we've asked for in the issue template? Thanks

andrepuschmann commented 3 years ago

A bit of a shot in the blue, but could you perhaps give https://github.com/srsLTE/srsLTE/tree/fix_enb_crash a try?

avalori1 commented 3 years ago

Thanks for the quick feedback. This is what I have done (I will try to be as detailed as possible). I am running the versions mentioned above (20.10.1). I start both srsenb and srsepc (without a specific order). I have a total of 4 SIMs that I programmed and used, I have placed them at different times in 4 different devices, swapping them several times:

At the moment of the latest test (the one that you were guessing might have not worked) there was only one active device with a programmed SIM in it (the Samsung). One other of the custom SIMs was in the Iphone which was, however, switched off. The Samsung phone was able to see the network but was unable to connect to it (with all the previous builds the phone was connecting happily and able to surf the internet).

About the system, does this answer to your questions?:

SDR: Ettus B200

PC:

bts@bts-ThinkPad-L460:~$ sudo neofetch --stdout
root@bts-ThinkPad-L460 

OS: Ubuntu 18.04.5 LTS x86_64 
Host: 20FVS1A400 ThinkPad L460 
Kernel: 5.4.0-65-generic 
Uptime: 2 days, 5 hours, 27 mins 
Packages: 2404 
Shell: bash 4.4.20 
Resolution: 1920x1080 
WM: GNOME Shell 
WM Theme: Adwaita 
Theme: Adwaita [GTK3] 
Icons: Adwaita [GTK3] 
Terminal: gnome-terminal 
CPU: Intel i5-6200U (4) @ 2.800GHz 
GPU: Intel Skylake GT2 [HD Graphics 520] 
Memory: 2871MiB / 7395MiB
avalori1 commented 3 years ago

@andrepuschmann

Dear Andre, one more update:

I Run the version fix_enb_crash compiled with the cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_ASAN=True .. option

---  Software Radio Systems LTE eNodeB  ---
Reading configuration file /home/bts/.config/srslte/enb.conf...
Built in RelWithDebInfo mode using commit 812f2571d on branch fix_enb_crash.

which crashed after about 5 hours with not much info in the stout (compared to all the info that came before)

 4fb 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 4fc 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
terminate called recursively
srsLTE crashed... backtrace saved in './srsLTE.backtrace.crash'...
---  exiting  ---

in the srsLTE.backtrace.crash there are two sections with the right timestamp:

--- command='srsenb' version=20.10.1 signal=6 date='16/02/2021 04:48:45' ---
    srsenb(+0x67f480) [0x55da1841f480]
    srsenb(+0x63c40e) [0x55da183dc40e]
    /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7fd38aec8040]
    /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fd38aec7fb7]
    /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fd38aec9921]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957) [0x7fd38b8bc957]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7fd38b8c2ae6]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7fd38b8c2b21]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d54) [0x7fd38b8c2d54]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8e811) [0x7fd38b8be811]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x124799) [0x7fd38b954799]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_mutateEmmPKcm+0x6b) [0x7fd38b954b8b]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEmmPKcm+0x1cb) [0x7fd38b95594b]
    /usr/local/lib/libsrslte_rf.so.0(+0x5b00f) [0x7fd38c02100f]
    /usr/local/lib/libsrslte_rf.so.0(rf_uhd_recv_with_time_multi+0x6ae) [0x7fd38bfee6fe]
    srsenb(+0x90874a) [0x55da186a874a]
    srsenb(+0x90ac47) [0x55da186aac47]
    srsenb(+0x246bdc) [0x55da17fe6bdc]
    srsenb(+0x1aeb1e) [0x55da17f4eb1e]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fd38ceb86db]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fd38afaa71f]

--- command='srsenb' version=20.10.1 signal=6 date='16/02/2021 04:48:47' ---
    srsenb(+0x67f480) [0x55da1841f480]
    srsenb(+0x63c40e) [0x55da183dc40e]
    /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7fd38aec8040]
    /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7fd38aec7fb7]
    /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7fd38aec9921]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x94c42) [0x7fd38b8c4c42]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7fd38b8c2ae6]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7fd38b8c2b21]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d54) [0x7fd38b8c2d54]
    /usr/local/lib/libuhd.so.3.15.0(+0x2164ae) [0x7fd3884bc4ae]
    /usr/local/lib/libuhd.so.3.15.0(_Z25set_c_global_error_stringRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x141) [0x7fd388b820f1]
    /usr/local/lib/libsrslte_rf.so.0(+0x5bbb3) [0x7fd38c021bb3]
    /usr/local/lib/libsrslte_rf.so.0(+0x22513) [0x7fd38bfe8513]
    /usr/local/lib/libsrslte_rf.so.0(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFPvS3_EP16rf_uhd_handler_tEEEEE6_M_runEv+0x36) [0x7fd38bffa316]
    /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6df) [0x7fd38b8ed6df]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fd38ceb86db]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fd38afaa71f]
andrepuschmann commented 3 years ago

Ok, I'll try to reproduce here myself. What else do you do during the 5h of the test with the UEs? You do traffic? Attach/detach? I see the RNTIs 4fb is quite high? Also, you use srsEPC by default with the default config, right? Can you perhaps share all configs you use so I can setup the exact same here?

avalori1 commented 3 years ago

@andrepuschmann

during the 5 hours there was a single UE (Samsung) which was trying to connect (but I think never managed to reach the internet). So, probably indeed may (at least attempts) to attach-detach. To be 100% detailed (although I am pretty sure is not relevant), there was one more of the custom SIMs registered in the user_db.csv which was in an IPhone, which was turned off) and one other phone with a commercial SIM in it (maybe it was trying to connect to the eNB even if the SIM was not of course allowed?) I am attaching the .conf files (removed the user_db.csv). In the user_db.csv, apart the IMSI and key information, the only difference from the default is that I assigned staic IPs to try to solve the issue #585 which I was facing in the past with several UEs (this test was with only one UE online, as said. Can you please elaborate a bit more on the RNTIs? I understand that are basically the "UE identifiers"... however, this are many more than the number of allowed IMSIs... Is a RNTI created also for interactions with non allowed UEs? Is there a way to "reset" them? Would it be of any help?

Thanks and regards.

srslte.zip

avalori1 commented 3 years ago

@andrepuschmann Hoping to be useful rather than annoying, one more example... I also saved the files:

enb.log
sb.pcap
enb_s1ap.pcap
epc.log
epc.pcap

Let me know if you need them (in total are almost 5GB...)

Cheers, Andrea

 160 4.5   0 1.1     0    0   96 100%    0  0.0   0     0    0    0   0%   0.0
 163 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 1b7 8.0   00.10     0    0    8 100%    0  0.0   0     0    0    0   0%   0.0
 1b8 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 1b9 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 1ba 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 215 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 216 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 217  11   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 225 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 243  14   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 244 6.5   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 245 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 26d  12   0 0.0     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 26e 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 270 7.0   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 29c 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 2df 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 2f8 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 311 0.10   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 312 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 313 1.0   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 315  14   0 0.8     0    0   93 100%    0  0.0   0     0    0    0   0%   0.0
 316 8.0   00.10     0    0    8 100%    0  0.0   0     0    0    0   0%   0.0
 317 0.10   00.10     0    0    3 100%    0  0.0   0     0    0    0   0%   0.0
 319 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 31a 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 349 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 34a 4.0   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
 34b 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 34c 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 34d 0.10   0 0.0     0    0    0   0%    0  0.0   0     0    0    0   0%   0.0
 34f 3.0   00.10     0    0    4 100%    0  0.0   0     0    0    0   0%   0.0
RACH:  tti=4361, cc=0, preamble=30, offset=1, temp_crnti=0x350
RACH:  tti=4451, cc=0, preamble=5, offset=31, temp_crnti=0x351
RACH:  tti=4471, cc=0, preamble=4, offset=31, temp_crnti=0x352
RACH:  tti=4491, cc=0, preamble=22, offset=31, temp_crnti=0x353
RACH:  tti=4511, cc=0, preamble=8, offset=31, temp_crnti=0x354
Disconnecting rnti=0x351.
RACH:  tti=4531, cc=0, preamble=54, offset=31, temp_crnti=0x355
Disconnecting rnti=0x352.
RACH:  tti=4551, cc=0, preamble=12, offset=31, temp_crnti=0x356
Disconnecting rnti=0x353.
RACH:  tti=4571, cc=0, preamble=10, offset=31, temp_crnti=0x357
Disconnecting rnti=0x354.
RACH:  tti=4591, cc=0, preamble=39, offset=31, temp_crnti=0x358
Disconnecting rnti=0x355.
RACH:  tti=4611, cc=0, preamble=44, offset=31, temp_crnti=0x359
RACH:  tti=4631, cc=0, preamble=29, offset=31, temp_crnti=0x35a
Disconnecting rnti=0x356.
Disconnecting rnti=0x357.
RACH:  tti=4651, cc=0, preamble=51, offset=31, temp_crnti=0x35b
RACH:  tti=4671, cc=0, preamble=10, offset=31, temp_crnti=0x35c
Disconnecting rnti=0x358.
Disconnecting rnti=0x359.
RACH:  tti=4691, cc=0, preamble=21, offset=31, temp_crnti=0x35d
Disconnecting rnti=0x35a.
RACH:  tti=4711, cc=0, preamble=44, offset=31, temp_crnti=0x35e
Disconnecting rnti=0x35b.
RACH:  tti=4731, cc=0, preamble=46, offset=31, temp_crnti=0x35f
Disconnecting rnti=0x35c.
RACH:  tti=4751, cc=0, preamble=44, offset=31, temp_crnti=0x360
Disconnecting rnti=0x35d.
RACH:  tti=4771, cc=0, preamble=41, offset=31, temp_crnti=0x361
Disconnecting rnti=0x35e.
RACH:  tti=4791, cc=0, preamble=50, offset=31, temp_crnti=0x362
Disconnecting rnti=0x35f.
RACH:  tti=4811, cc=0, preamble=20, offset=18, temp_crnti=0x363
RACH:  tti=4831, cc=0, preamble=13, offset=18, temp_crnti=0x364
Disconnecting rnti=0x360.
Disconnecting rnti=0x361.
RACH:  tti=4851, cc=0, preamble=23, offset=18, temp_crnti=0x365
Disconnecting rnti=0x362.
RACH:  tti=4871, cc=0, preamble=8, offset=18, temp_crnti=0x366
Disconnecting rnti=0x363.
RACH:  tti=4891, cc=0, preamble=8, offset=18, temp_crnti=0x367
Disconnecting rnti=0x364.
RACH:  tti=4911, cc=0, preamble=6, offset=18, temp_crnti=0x368
Disconnecting rnti=0x365.
RACH:  tti=4931, cc=0, preamble=19, offset=18, temp_crnti=0x369
Disconnecting rnti=0x366.
RACH:  tti=4951, cc=0, preamble=6, offset=18, temp_crnti=0x36a
Disconnecting rnti=0x367.
RACH:  tti=4971, cc=0, preamble=50, offset=18, temp_crnti=0x36b
Disconnecting rnti=0x368.
RACH:  tti=4991, cc=0, preamble=37, offset=18, temp_crnti=0x36c
Disconnecting rnti=0x369.
RACH:  tti=5011, cc=0, preamble=0, offset=31, temp_crnti=0x36d
Disconnecting rnti=0x36a.
RACH:  tti=5031, cc=0, preamble=31, offset=18, temp_crnti=0x36e
Disconnecting rnti=0x36b.
RACH:  tti=5051, cc=0, preamble=11, offset=18, temp_crnti=0x36f
Disconnecting rnti=0x36c.
RACH:  tti=5071, cc=0, preamble=0, offset=31, temp_crnti=0x370
Disconnecting rnti=0x36d.
RACH:  tti=5091, cc=0, preamble=24, offset=18, temp_crnti=0x371
Disconnecting rnti=0x36e.
RACH:  tti=5111, cc=0, preamble=11, offset=45, temp_crnti=0x372
RACH:  tti=5111, cc=0, preamble=13, offset=3, temp_crnti=0x373
Disconnecting rnti=0x36f.
RACH:  tti=5131, cc=0, preamble=13, offset=45, temp_crnti=0x374
RACH:  tti=5131, cc=0, preamble=15, offset=3, temp_crnti=0x375
RACH:  tti=5151, cc=0, preamble=5, offset=10, temp_crnti=0x376
Disconnecting rnti=0x370.
RACH:  tti=5151, cc=0, preamble=8, offset=16, temp_crnti=0x377
Disconnecting rnti=0x371.
RACH:  tti=5171, cc=0, preamble=5, offset=10, temp_crnti=0x378
RACH:  tti=5171, cc=0, preamble=8, offset=16, temp_crnti=0x379
Disconnecting rnti=0x372.
RACH:  tti=5191, cc=0, preamble=49, offset=3, temp_crnti=0x37a
Disconnecting rnti=0x373.
RACH:  tti=5191, cc=0, preamble=52, offset=9, temp_crnti=0x37b
Disconnecting rnti=0x374.
RACH:  tti=5211, cc=0, preamble=24, offset=45, temp_crnti=0x37c
RACH:  tti=5211, cc=0, preamble=26, offset=3, temp_crnti=0x37d
Disconnecting rnti=0x375.
Disconnecting rnti=0x376.
RACH:  tti=5231, cc=0, preamble=48, offset=3, temp_crnti=0x37e
RACH:  tti=5231, cc=0, preamble=51, offset=9, temp_crnti=0x37f
Disconnecting rnti=0x377.
Disconnecting rnti=0x378.
RACH:  tti=5251, cc=0, preamble=10, offset=45, temp_crnti=0x380
RACH:  tti=5251, cc=0, preamble=12, offset=3, temp_crnti=0x381
Disconnecting rnti=0x379.
Disconnecting rnti=0x37a.
RACH:  tti=5271, cc=0, preamble=15, offset=45, temp_crnti=0x382
Disconnecting rnti=0x37b.
RACH:  tti=5271, cc=0, preamble=17, offset=3, temp_crnti=0x383
=================================================================
==6565==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f65e900bf54 at pc 0x556fa0ba7399 bp 0x7f65e900b6e0 sp 0x7f65e900b6d0
READ of size 4 at 0x7f65e900bf54 thread T33 (METRICS_HUB)
Disconnecting rnti=0x37c.
RACH:  tti=5291, cc=0, preamble=43, offset=3, temp_crnti=0x384
Disconnecting rnti=0x37d.
RACH:  tti=5291, cc=0, preamble=46, offset=9, temp_crnti=0x385
Disconnecting rnti=0x37e.
RACH:  tti=5311, cc=0, preamble=44, offset=3, temp_crnti=0x386
Disconnecting rnti=0x37f.
RACH:  tti=5311, cc=0, preamble=47, offset=9, temp_crnti=0x387
Disconnecting rnti=0x380.
RACH:  tti=5331, cc=0, preamble=1, offset=16, temp_crnti=0x388
Disconnecting rnti=0x381.
RACH:  tti=5331, cc=0, preamble=31, offset=45, temp_crnti=0x389
Disconnecting rnti=0x382.
RACH:  tti=5351, cc=0, preamble=47, offset=3, temp_crnti=0x38a
Disconnecting rnti=0x383.
RACH:  tti=5351, cc=0, preamble=50, offset=9, temp_crnti=0x38b
Disconnecting rnti=0x384.
RACH:  tti=5371, cc=0, preamble=56, offset=3, temp_crnti=0x38c
RACH:  tti=5371, cc=0, preamble=59, offset=9, temp_crnti=0x38d
Disconnecting rnti=0x385.
Disconnecting rnti=0x386.
RACH:  tti=5391, cc=0, preamble=23, offset=45, temp_crnti=0x38e
Disconnecting rnti=0x387.
RACH:  tti=5391, cc=0, preamble=25, offset=3, temp_crnti=0x38f
RACH:  tti=5411, cc=0, preamble=4, offset=10, temp_crnti=0x390
RACH:  tti=5411, cc=0, preamble=7, offset=16, temp_crnti=0x391
Disconnecting rnti=0x388.
Disconnecting rnti=0x389.
Disconnecting rnti=0x38a.
RACH:  tti=5431, cc=0, preamble=60, offset=3, temp_crnti=0x392
Disconnecting rnti=0x38b.
RACH:  tti=5431, cc=0, preamble=63, offset=9, temp_crnti=0x393
Disconnecting rnti=0x38c.
RACH:  tti=5451, cc=0, preamble=47, offset=3, temp_crnti=0x394
Disconnecting rnti=0x38d.
RACH:  tti=5451, cc=0, preamble=50, offset=9, temp_crnti=0x395
Disconnecting rnti=0x38e.
RACH:  tti=5471, cc=0, preamble=45, offset=3, temp_crnti=0x396
Disconnecting rnti=0x38f.
RACH:  tti=5471, cc=0, preamble=48, offset=9, temp_crnti=0x397
Disconnecting rnti=0x390.
Disconnecting rnti=0x391.
RACH:  tti=5491, cc=0, preamble=29, offset=45, temp_crnti=0x398
RACH:  tti=5491, cc=0, preamble=31, offset=3, temp_crnti=0x399
SCHED: Could not transmit RAR within the window (RA=5491, Window=[5494..5504], RAR=5508)
Disconnecting rnti=0x392.
RACH:  tti=5511, cc=0, preamble=1, offset=32, temp_crnti=0x39a
Disconnecting rnti=0x393.
RACH:  tti=5511, cc=0, preamble=27, offset=9, temp_crnti=0x39b
RACH:  tti=5511, cc=0, preamble=30, offset=14, temp_crnti=0x39c
Disconnecting rnti=0x394.
Disconnecting rnti=0x395.
RACH:  tti=5531, cc=0, preamble=49, offset=9, temp_crnti=0x39d
RACH:  tti=5531, cc=0, preamble=52, offset=14, temp_crnti=0x39e
RACH:  tti=5531, cc=0, preamble=55, offset=20, temp_crnti=0x39f
Disconnecting rnti=0x396.
Disconnecting rnti=0x397.
RACH:  tti=5551, cc=0, preamble=12, offset=9, temp_crnti=0x3a0
RACH:  tti=5551, cc=0, preamble=15, offset=14, temp_crnti=0x3a1
SCHED: Could not transmit RAR within the window (RA=5551, Window=[5554..5564], RAR=5570)
    #0 0x556fa0ba7398 in srsenb::phy::get_metrics(srsenb::phy_metrics_t*) /home/bts/srsLTE/srsenb/src/phy/phy.cc:202
RACH:  tti=5551, cc=0, preamble=18, offset=20, temp_crnti=0x3a2
Disconnecting rnti=0x398.
Disconnecting rnti=0x399.
SCHED: Could not transmit RAR within the window (RA=5551, Window=[5554..5564], RAR=5582)
RACH:  tti=5571, cc=0, preamble=32, offset=27, temp_crnti=0x3a3
SCHED: Could not transmit RAR within the window (RA=5571, Window=[5574..5584], RAR=5590)
RACH:  tti=5571, cc=0, preamble=35, offset=32, temp_crnti=0x3a4
SCHED: Could not transmit RAR within the window (RA=5571, Window=[5574..5584], RAR=5594)
RACH:  tti=5571, cc=0, preamble=61, offset=9, temp_crnti=0x3a5
Disconnecting rnti=0x39a.
SCHED: Could not transmit RAR within the window (RA=5571, Window=[5574..5584], RAR=5595)
Disconnecting rnti=0x39b.
RACH:  tti=5591, cc=0, preamble=56, offset=9, temp_crnti=0x3a6
Disconnecting rnti=0x39c.
    #1 0x556fa0b8984d in srsenb::enb::get_metrics(srsenb::enb_metrics_t*) /home/bts/srsLTE/srsenb/src/enb.cc:206
RACH:  tti=5591, cc=0, preamble=59, offset=14, temp_crnti=0x3a7
SCHED: Could not transmit RAR within the window (RA=5591, Window=[5594..5604], RAR=5615)
RACH:  tti=5591, cc=0, preamble=62, offset=20, temp_crnti=0x3a8
SCHED: Could not transmit RAR within the window (RA=5591, Window=[5594..5604], RAR=5616)
RACH:  tti=5611, cc=0, preamble=15, offset=9, temp_crnti=0x3a9
Disconnecting rnti=0x39d.
Disconnecting rnti=0x39e.
Disconnecting rnti=0x39f.
RACH:  tti=5611, cc=0, preamble=18, offset=14, temp_crnti=0x3aa
RACH:  tti=5611, cc=0, preamble=21, offset=20, temp_crnti=0x3ab
SCHED: Could not transmit RAR within the window (RA=5611, Window=[5614..5624], RAR=5637)
RACH:  tti=5631, cc=0, preamble=12, offset=9, temp_crnti=0x3ac
SCHED: Could not transmit RAR within the window (RA=5631, Window=[5634..5644], RAR=5645)
RACH:  tti=5631, cc=0, preamble=15, offset=14, temp_crnti=0x3ad
SCHED: Could not transmit RAR within the window (RA=5631, Window=[5634..5644], RAR=5646)
RACH:  tti=5631, cc=0, preamble=18, offset=20, temp_crnti=0x3ae
Disconnecting rnti=0x3a0.
Disconnecting rnti=0x3a1.
SCHED: Could not transmit RAR within the window (RA=5631, Window=[5634..5644], RAR=5648)
Disconnecting rnti=0x3a2.
RACH:  tti=5651, cc=0, preamble=33, offset=27, temp_crnti=0x3af
RACH:  tti=5651, cc=0, preamble=36, offset=32, temp_crnti=0x3b0
SCHED: Could not transmit RAR within the window (RA=5651, Window=[5654..5664], RAR=5667)
RACH:  tti=5651, cc=0, preamble=62, offset=9, temp_crnti=0x3b1
Disconnecting rnti=0x3a3.
Disconnecting rnti=0x3a4.
Disconnecting rnti=0x3a5.
SCHED: Could not transmit RAR within the window (RA=5651, Window=[5654..5664], RAR=5670)
Disconnecting rnti=0x3a6.
Disconnecting rnti=0x3a7.
Disconnecting rnti=0x3a8.
Disconnecting rnti=0x3a9.
Disconnecting rnti=0x3aa.
Disconnecting rnti=0x3ab.
Disconnecting rnti=0x3ac.
Disconnecting rnti=0x3ad.
Disconnecting rnti=0x3ae.
Disconnecting rnti=0x3af.
Disconnecting rnti=0x3b0.
Disconnecting rnti=0x3b1.
    #2 0x556fa0b61bef in srslte::metrics_hub<srsenb::enb_metrics_t>::run_period() /home/bts/srsLTE/lib/include/srslte/common/metrics_hub.h:95
    #3 0x556fa0b54e28 in srslte::periodic_thread::run_thread() /home/bts/srsLTE/lib/include/srslte/common/threads.h:152
    #4 0x556fa0b54b1d in srslte::thread::thread_function_entry(void*) /home/bts/srsLTE/lib/include/srslte/common/threads.h:111
    #5 0x7f6604c0b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #6 0x7f6602cfd71e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)

Address 0x7f65e900bf54 is located in stack of thread T33 (METRICS_HUB) at offset 2084 in frame
    #0 0x556fa0ba6baf in srsenb::phy::get_metrics(srsenb::phy_metrics_t*) /home/bts/srsLTE/srsenb/src/phy/phy.cc:194

  This frame has 1 object(s):
    [32, 2080) 'metrics_tmp' <== Memory access at offset 2084 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
Thread T33 (METRICS_HUB) created by T0 here:
    #0 0x7f6604e5ad2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x556fa1020371 in threads_new_rt_cpu /home/bts/srsLTE/lib/src/common/threads.c:148
    #2 0x556fa0b3adcf in srslte::thread::start(int) /home/bts/srsLTE/lib/include/srslte/common/threads.h:72
    #3 0x556fa0b3adcf in srslte::periodic_thread::start_periodic(int, int) /home/bts/srsLTE/lib/include/srslte/common/threads.h:128
    #4 0x556fa0b3adcf in srslte::metrics_hub<srsenb::enb_metrics_t>::init(srslte::metrics_interface<srsenb::enb_metrics_t>*, float) /home/bts/srsLTE/lib/include/srslte/common/metrics_hub.h:64
    #5 0x556fa0b3adcf in main /home/bts/srsLTE/srsenb/src/main.cc:502
    #6 0x7f6602bfdbf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)

SUMMARY: AddressSanitizer: stack-buffer-overflow /home/bts/srsLTE/srsenb/src/phy/phy.cc:202 in srsenb::phy::get_metrics(srsenb::phy_metrics_t*)
Shadow bytes around the buggy address:
  0x0fed3d1f9790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fed3d1f97a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fed3d1f97b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fed3d1f97c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fed3d1f97d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fed3d1f97e0: 00 00 00 00 00 00 00 00 00 00[f3]f3 f3 f3 00 00
  0x0fed3d1f97f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fed3d1f9800: 00 00 00 00 00 00 f1 f1 f1 f1 00 f2 f2 f2 f2 f2
  0x0fed3d1f9810: f2 f2 f8 f2 f2 f2 f2 f2 f2 f2 f8 f2 f2 f2 f2 f2
  0x0fed3d1f9820: f2 f2 f8 f2 f2 f2 f2 f2 f2 f2 f8 f2 f2 f2 f2 f2
  0x0fed3d1f9830: f2 f2 f8 f2 f2 f2 f2 f2 f2 f2 f8 f2 f2 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==6565==ABORTING
bts@bts-ThinkPad-L460:~$
andrepuschmann commented 3 years ago

@avalori1 We've been able to reproduce the issue and work on a fix for it. I'll keep you posted as soon as we have something that you can try. Thanks again for the reports

avalori1 commented 3 years ago

Great, many thanks. Looking forward to try your solution.

andrepuschmann commented 3 years ago

Hey, we've pushed a new branch to https://github.com/srsLTE/srsLTE/tree/fixing_enb_20_10 that contains a few backported fixes for the eNB. Let me know if they improve the situation for you.

Thanks Andre

avalori1 commented 3 years ago

Hello, I run some tests and this are the first impressions. It seems to be way better. I did try to install with the cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_ASAN=True .. option but after that I was unable to start my node (using a B200). The Tx and Rx LED on the board were off or flickering... After re-compiling without the option the enode seems working well and with no crash. Will want to also increase the number of UE but will need some time for that. On the topic, is there a way to put in relationship the rnti shown by the trace of the srsenb with the IMSI?

Thanks

andrepuschmann commented 3 years ago

That's good to know. Thanks for checking. The ASAN build is not meant for normal operation. It's just for testing, etc. There is 2x performance penalty when having ASAN for all the checks it does. Therefore, you see the flickering LEDs.

Regarding your other question, no there is no relationship. The RNTI is assigned by the eNB without knowing the IMSI

liumaolin-931 commented 3 years ago

Hi I have some similar problems and the terminal interface is as follows:

sudo srsepc epc.conf Built in Release mode using commit 45486b6 on branch master.

--- Software Radio Systems EPC ---

Reading configuration file epc.conf... HSS Initialized. Error binding SCTP socket Error initializing GTP-C

The strange thing is that it can work normally after boot and restart, but when I turn off the current terminal and open a new one, this error will appear. Can someone help me?Thanks!

avalori1 commented 3 years ago

Seems that you still have a srsepc running in the background... did you start it with & at the end? Try $ pidof srsepc if you get a number means that epc is still running...

and then: $ sudo kill -9

hope it helps

On Wed, 14 Apr 2021 at 08:52, liumaolin-931 @.***> wrote:

Hi I have some similar problems and the terminal interface is as follows:

sudo srsepc epc.conf Built in Release mode using commit 45486b6 https://github.com/srsLTE/srsLTE/commit/45486b6e2c622ab1d47906521662f087816d7d03 on branch master.

--- Software Radio Systems EPC ---

Reading configuration file epc.conf... HSS Initialized. Error binding SCTP socket Error initializing GTP-C

The strange thing is that it can work normally after boot and restart, but when I turn off the current terminal and open a new one, this error will appear. Can someone help me?Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/srsLTE/srsLTE/issues/623#issuecomment-819275876, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARPP3JEEIVWVMNLPH2FUFQ3TIU3TLANCNFSM4XEYLTFA .

liumaolin-931 commented 3 years ago

thank you very much!The problem is solved！

------------------ 原始邮件 ------------------ 发件人: "srsLTE/srsLTE" @.>; 发送时间: 2021年4月14日(星期三) 下午4:28 @.>; @.**@.>; 主题: Re: [srsLTE/srsLTE] srsLTE crashed (#623)

Seems that you still have a srsepc running in the background... did you start it with & at the end? Try $ pidof srsepc if you get a number means that epc is still running...

and then: $ sudo kill -9 <PID_you_got_above>

hope it helps

On Wed, 14 Apr 2021 at 08:52, liumaolin-931 @.***> wrote:

> Hi > I have some similar problems and the terminal interface is as follows: > > sudo srsepc epc.conf > Built in Release mode using commit 45486b6 > <https://github.com/srsLTE/srsLTE/commit/45486b6e2c622ab1d47906521662f087816d7d03&gt; > on branch master. > > --- Software Radio Systems EPC --- > > Reading configuration file epc.conf... > HSS Initialized. > Error binding SCTP socket > Error initializing GTP-C > > The strange thing is that it can work normally after boot and restart, but > when I turn off the current terminal and open a new one, this error will > appear. Can someone help me?Thanks! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/srsLTE/srsLTE/issues/623#issuecomment-819275876&gt;, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ARPP3JEEIVWVMNLPH2FUFQ3TIU3TLANCNFSM4XEYLTFA&gt; > . >

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

andrepuschmann commented 3 years ago

Very good. Closing the issue.