eNB crashes in srsenb::rrc::ue::parse_ul_dcch

[rriggio]

Issue Description

The eNB compiled from the latest master crashes during attach or detach. The behavior is random. Sometimes the UE (a nexus 5) is attaching without problems other times the eNB crashes.

Setup Details

I am using ubuntu 20.04.02, an ettus B210mini, a Nexus 5 as UE, and a sysmocom USIM-SJS1 sim card

Expected Behavior

The eNB should not crash

Actual Behaviour

The eNB crashes

Steps to reproduce the problem

Compile from master edit the enb.conf to point to the core network and try to attach a UE.

Additional Information

This is the backtraces of the crash:

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

1 0x00007ffff7673859 in __GI_abort () at abort.c:79

2 0x00005555556780da in srsran::optional::value() [clone .part.0] ()

3 0x000055555567e7a5 in srsenb::s1ap::ue::send_ulnastransport(std::unique_ptr<srsran::byte_buffer_t, std::default_delete >) ()

4 0x000055555567e87a in srsenb::s1ap::write_pdu(unsigned short, std::unique_ptr<srsran::byte_buffer_t, std::default_delete >) ()

5 0x000055555570e9d9 in srsenb::rrc::ue::parse_ul_dcch(unsigned int, std::unique_ptr<srsran::byte_buffer_t, std::default_delete >) ()

6 0x00005555556f4a7a in srsenb::rrc::parse_ul_dcch(unsigned short, unsigned int, std::unique_ptr<srsran::byte_buffer_t, std::default_delete >) ()

7 0x00005555556f6cb4 in srsenb::rrc::tti_clock() ()

8 0x000055555564f3e4 in srsran::task_details::smallbuffer_table_t<srsenb::enb_stack_lte::tti_clock()::{lambda()#1}, void>::call(void*) const ()

9 0x000055555564eb4a in srsenb::enb_stack_lte::run_thread() ()

10 0x00005555556066fd in srsran::thread::thread_function_entry(void*) ()

11 0x00007ffff7f97609 in start_thread (arg=) at pthread_create.c:477

12 0x00007ffff7770293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

And this is the crash log:

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:42:25' --- ./srsenb(+0x296a5f) [0x55c57886ca5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f61c683a210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f61c683a18b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f61c6819859] ./srsenb(+0x1240da) [0x55c5786fa0da] ./srsenb(+0x12a7a5) [0x55c5787007a5] ./srsenb(+0x12a87a) [0x55c57870087a] ./srsenb(+0x1ba9d9) [0x55c5787909d9] ./srsenb(+0x1a0a7a) [0x55c578776a7a] ./srsenb(+0x1a2cb4) [0x55c578778cb4] ./srsenb(+0xfb3e4) [0x55c5786d13e4] ./srsenb(+0xfab4a) [0x55c5786d0b4a] ./srsenb(+0xb26fd) [0x55c5786886fd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f61c713d609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f61c6916293]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:42:26' --- ./srsenb(+0x296a5f) [0x55c57886ca5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f61c683a210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f61c683a18b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f61c6819859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e951) [0x7f61c6bee951] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7f61c6bfa47c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa4e7) [0x7f61c6bfa4e7] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x1b338) [0x7f61c6f5e338] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x3dc02) [0x7f61c6f80c02] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xce) [0x7f61c683e15e] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x122b7) [0x7f61c6f552b7]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:42:26' --- --- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:42:53' --- ./srsenb(+0x296a5f) [0x55acd3411a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f0bb930a210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f0bb930a18b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f0bb92e9859] ./srsenb(+0x1240da) [0x55acd329f0da] ./srsenb(+0x12a7a5) [0x55acd32a57a5] ./srsenb(+0x12a87a) [0x55acd32a587a] ./srsenb(+0x1ba9d9) [0x55acd33359d9] ./srsenb(+0x1a0a7a) [0x55acd331ba7a] ./srsenb(+0x1a2cb4) [0x55acd331dcb4] ./srsenb(+0xfb3e4) [0x55acd32763e4] ./srsenb(+0xfab4a) [0x55acd3275b4a] ./srsenb(+0xb26fd) [0x55acd322d6fd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f0bb9c0d609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f0bb93e6293]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:42:53' --- ./srsenb(+0x296a5f) [0x55acd3411a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f0bb930a210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f0bb930a18b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f0bb92e9859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e951) [0x7f0bb96be951] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7f0bb96ca47c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa4e7) [0x7f0bb96ca4e7] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x1b338) [0x7f0bb9a2e338] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x3dc02) [0x7f0bb9a50c02] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xce) [0x7f0bb930e15e] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x122b7) [0x7f0bb9a252b7]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:42:53' --- --- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:43:18' --- ./srsenb(+0x296a5f) [0x562686ddaa5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f529b1fd210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f529b1fd18b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f529b1dc859] ./srsenb(+0x1240da) [0x562686c680da] ./srsenb(+0x12a7a5) [0x562686c6e7a5] ./srsenb(+0x12a87a) [0x562686c6e87a] ./srsenb(+0x1ba9d9) [0x562686cfe9d9] ./srsenb(+0x1a0a7a) [0x562686ce4a7a] ./srsenb(+0x1a2cb4) [0x562686ce6cb4] ./srsenb(+0xfb3e4) [0x562686c3f3e4] ./srsenb(+0xfab4a) [0x562686c3eb4a] ./srsenb(+0xb26fd) [0x562686bf66fd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f529bb00609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f529b2d9293]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:43:19' --- ./srsenb(+0x296a5f) [0x562686ddaa5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f529b1fd210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f529b1fd18b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f529b1dc859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e951) [0x7f529b5b1951] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7f529b5bd47c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa4e7) [0x7f529b5bd4e7] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x1b338) [0x7f529b921338] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x3dc02) [0x7f529b943c02] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xce) [0x7f529b20115e] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x122b7) [0x7f529b9182b7]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:43:19' --- --- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:43:38' --- ./srsenb(+0x296a5f) [0x558383789a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f4b7d980210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f4b7d98018b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f4b7d95f859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e951) [0x7f4b7dd34951] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7f4b7dd4047c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa9459) [0x7f4b7dd3f459] /lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x2a1) [0x7f4b7dd3fe11] /lib/x86_64-linux-gnu/libgcc_s.so.1(+0x10bdf) [0x7f4b7db3cbdf] /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x12a) [0x7f4b7db3d59a] /lib/x86_64-linux-gnu/libuhd.so.3.15.0(+0x981921) [0x7f4b7d628921] /lib/x86_64-linux-gnu/libuhd.so.3.15.0(+0x97f6c9) [0x7f4b7d6266c9] /lib/x86_64-linux-gnu/libuhd.so.3.15.0(+0x3b930d) [0x7f4b7d06030d] /lib/x86_64-linux-gnu/libuhd.so.3.15.0(+0x718dd0) [0x7f4b7d3bfdd0] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x25d92) [0x7f4b7e0aed92] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(rf_uhd_send_timed_multi+0x6e0) [0x7f4b7e09f750] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(rf_uhd_send_timed+0x92) [0x7f4b7e09ff42] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(srsran_rf_send_timed2+0x2d) [0x7f4b7e09bb3d] ./srsenb(+0x443637) [0x558383936637] ./srsenb(+0x443c13) [0x558383936c13] ./srsenb(+0xdf849) [0x5583835d2849] ./srsenb(+0xed1e7) [0x5583835e01e7] ./srsenb(+0x2a7cc9) [0x55838379acc9] ./srsenb(+0xb26fd) [0x5583835a56fd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f4b7e283609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f4b7da5c293]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:43:39' --- ./srsenb(+0x296a5f) [0x558383789a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f4b7d980210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f4b7d98018b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f4b7d95f859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xac63a) [0x7f4b7dd4263a] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7f4b7dd4047c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa4e7) [0x7f4b7dd404e7] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x1b338) [0x7f4b7e0a4338] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x3dc02) [0x7f4b7e0c6c02] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xce) [0x7f4b7d98415e] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x122b7) [0x7f4b7e09b2b7]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:44:18' --- ./srsenb(+0x296a5f) [0x5618ffb68a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fd4d2a64210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fd4d2a6418b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fd4d2a43859] ./srsenb(+0x1240da) [0x5618ff9f60da] ./srsenb(+0x12a7a5) [0x5618ff9fc7a5] ./srsenb(+0x12a87a) [0x5618ff9fc87a] ./srsenb(+0x1ba9d9) [0x5618ffa8c9d9] ./srsenb(+0x1a0a7a) [0x5618ffa72a7a] ./srsenb(+0x1a2cb4) [0x5618ffa74cb4] ./srsenb(+0xfb3e4) [0x5618ff9cd3e4] ./srsenb(+0xfab4a) [0x5618ff9ccb4a] ./srsenb(+0xb26fd) [0x5618ff9846fd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fd4d3367609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fd4d2b40293]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:44:19' --- ./srsenb(+0x296a5f) [0x5618ffb68a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fd4d2a64210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fd4d2a6418b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fd4d2a43859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e951) [0x7fd4d2e18951] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7fd4d2e2447c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa4e7) [0x7fd4d2e244e7] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x1b338) [0x7fd4d3188338] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x3dc02) [0x7fd4d31aac02] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xce) [0x7fd4d2a6815e] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x122b7) [0x7fd4d317f2b7]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:44:19' --- --- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:44:47' --- ./srsenb(+0x296a5f) [0x55734ae92a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f4c55052210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f4c5505218b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f4c55031859] ./srsenb(+0x1240da) [0x55734ad200da] ./srsenb(+0x12a7a5) [0x55734ad267a5] ./srsenb(+0x12a87a) [0x55734ad2687a] ./srsenb(+0x1ba9d9) [0x55734adb69d9] ./srsenb(+0x1a0a7a) [0x55734ad9ca7a] ./srsenb(+0x1a2cb4) [0x55734ad9ecb4] ./srsenb(+0xfb3e4) [0x55734acf73e4] ./srsenb(+0xfab4a) [0x55734acf6b4a] ./srsenb(+0xb26fd) [0x55734acae6fd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f4c55955609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f4c5512e293]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:44:48' --- ./srsenb(+0x296a5f) [0x55734ae92a5f] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7f4c55052210] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f4c5505218b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f4c55031859] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e951) [0x7f4c55406951] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa47c) [0x7f4c5541247c] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa4e7) [0x7f4c554124e7] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x1b338) [0x7f4c55776338] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x3dc02) [0x7f4c55798c02] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xce) [0x7f4c5505615e] /home/empower/TMP/srsRAN/build/lib/src/phy/rf/libsrsran_rf.so.0(+0x122b7) [0x7f4c5576d2b7]

--- command='./srsenb enb.conf' version=21.04.0 signal=6 date='08/05/2021 07:44:48' ---

[yagoda]

Hey, Can you send an email to srslte-users@lists.softwareradiosystems.com with the enb.log, epc.log and the console output?

Additionally, can you provide information about the SIM card used?

Regards, Justin

[rriggio]

I have added the brand and model of SIM card I am using. I am now having some difficulties in replicating the crash however if it can help, the crash was due to a failed assert at this line in s1ap::ue::send_ulnastransport:

container.mme_ue_s1ap_id.value = ctxt.mme_ue_s1ap_id.value();

[frankist]

Thanks @rriggio . It seems that UL NAS Transport is being sent to the MME for a UE whose S1 context is not yet fully created. We will need enb logs to further debug this issue.

[rriggio]

I will try to replicate the bug and provide the log @frankist. what level of logging should I enable in the eNB?

[frankist]

RRC and S1AP in Debug and all the other layers in Info should be sufficient. Thanks. We will soon provide a fix in the public release so that at least the app doesn't crash when this happens.

[rriggio]

Attached please find the logs. enb.log.zip

[frankist]

Thanks @rriggio. The branch https://github.com/srsran/srsRAN/tree/release_21_04_final should be able to resolve the crash. Can you confirm?

I noticed based on the enb.log that the EPC is taking a very long time to reply, or even not replying at all, to the InitialUEMessage. Are you using the srsEPC? If so, could you also send the epc.log file?

EDIT: Actually, it is not yet fixed. We were able to reproduce the issue with the srsEPC and are trying to fix it. I will update this issue when that's done.

[andrepuschmann]

Closing the issue. This should be fixed in master.