search

paullouisageneau / libdatachannel

C/C++ WebRTC network library featuring Data Channels, Media Transport, and WebSockets
https://libdatachannel.org/
Mozilla Public License 2.0
1.82k stars 365 forks source link

Segfault in SctptTansport #357

Closed harbu closed 3 years ago

harbu commented 3 years ago

Hello!

First of all, thanks for great library and project!

I'm reporting an issue we discovered while investigating #354 (potentially related to this).

For context: we've been studying the issue of connection speeds decreasing over time, originally via the node-datachannel library https://github.com/murat-dogan/node-datachannel/issues/32 as our main project is written in JavaScript.

We rebuilt our Node.js "issue reproduce project" in C++ to understand if the issue occurs also in C++ clients. While working on this, we noticed that when running connections for a long period of time, a segfault can occur. The following is a stack trace retrieved with gdb from a core dump:

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/examples/multi-node-benchmark/multi-node-benchmark...

warning: core file may not match specified executable file.
[New LWP 45760]
[New LWP 45765]
[New LWP 45802]
[New LWP 45745]
[New LWP 45767]
[New LWP 45758]
[New LWP 45819]
[New LWP 45803]
[New LWP 45801]
[New LWP 45811]
[New LWP 45759]
[New LWP 45761]
[New LWP 45821]
[New LWP 45817]
[New LWP 45820]
[New LWP 45773]
[New LWP 45770]
[New LWP 45799]
[New LWP 45822]
[New LWP 45800]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./build/examples/multi-node-benchmark/multi-node-benchmark'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000056136ce87920 in Peer::<lambda()>::operator()(void) const (__closure=0x7f2688005920) at /home/ubuntu/libdatachannel/examples/multi-node-benchmark/peer.cpp:81
81          cout << now() << logId << " DataChannel " << peerId << " LOW buffer " <<  dc->bufferedAmount()<< endl;
[Current thread is 1 (Thread 0x7f2698598700 (LWP 45760))]
(gdb) bt
#0  0x000056136ce87920 in Peer::<lambda()>::operator()(void) const (__closure=0x7f2688005920) at /home/ubuntu/libdatachannel/examples/multi-node-benchmark/peer.cpp:81
#1  0x000056136ce8a090 in std::_Function_handler<void(), Peer::setUpDataChannel(std::shared_ptr<rtc::DataChannel>)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/9/bits/std_function.h:300
#2  0x00007f269d1799b6 in std::function<void ()>::operator()() const (this=0x7f2680003750) at /usr/include/c++/9/bits/std_function.h:688
#3  0x00007f269d179a0d in rtc::synchronized_callback<>::operator()() const (this=0x7f2680003750) at /home/ubuntu/libdatachannel/include/rtc/utils.hpp:91
#4  0x00007f269d17973e in rtc::impl::Channel::triggerBufferedAmount(unsigned long) (this=0x7f26800035e0, amount=1048000) at /home/ubuntu/libdatachannel/src/impl/channel.cpp:45
#5  0x00007f269d199397 in rtc::impl::PeerConnection::forwardBufferedAmount(unsigned short, unsigned long) (this=0x7f269006af80, stream=1, amount=1048000) at /home/ubuntu/libdatachannel/src/impl/peerconnection.cpp:539
#6  0x00007f269d1d567b in std::__invoke_impl<void, void (rtc::impl::PeerConnection::* const&)(unsigned short, unsigned long), rtc::impl::PeerConnection* const&, unsigned short&, unsigned long&>(std::__invoke_memfun_deref, void (rtc::impl::PeerConnection::* const&)(unsigned short, unsigned long), rtc::impl::PeerConnection* const&, unsigned short&, unsigned long&) (__f=@0x7f2650015610: (void (rtc::impl::PeerConnection::*)(rtc::impl::PeerConnection * const, unsigned short, unsigned long)) 0x7f269d199324 <rtc::impl::PeerConnection::forwardBufferedAmount(unsigned short, unsigned long)>, __t=@0x7f2650015620: 0x7f269006af80)
    at /usr/include/c++/9/bits/invoke.h:73
#7  0x00007f269d1d3f6d in std::__invoke<void (rtc::impl::PeerConnection::* const&)(unsigned short, unsigned long), rtc::impl::PeerConnection* const&, unsigned short&, unsigned long&>(void (rtc::impl::PeerConnection::* const&)(unsigned short, unsigned long), rtc::impl::PeerConnection* const&, unsigned short&, unsigned long&) (__fn=@0x7f2650015610: (void (rtc::impl::PeerConnection::*)(rtc::impl::PeerConnection * const, unsigned short, unsigned long)) 0x7f269d199324 <rtc::impl::PeerConnection::forwardBufferedAmount(unsigned short, unsigned long)>) at /usr/include/c++/9/bits/invoke.h:95
#8  0x00007f269d1d1a13 in std::_Bind<void (rtc::impl::PeerConnection::*(rtc::impl::PeerConnection*, std::_Placeholder<1>, std::_Placeholder<2>))(unsigned short, unsigned long)>::__call_c<void, unsigned short&, unsigned long&, 0ul, 1ul, 2ul>(std::tuple<unsigned short&, unsigned long&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) const (this=0x7f2650015610, __args=...) at /usr/include/c++/9/functional:410
#9  0x00007f269d1cfb33 in std::_Bind<void (rtc::impl::PeerConnection::*(rtc::impl::PeerConnection*, std::_Placeholder<1>, std::_Placeholder<2>))(unsigned short, unsigned long)>::operator()<unsigned short&, unsigned long&, void>(unsigned short&, unsigned long&) const (this=0x7f2650015610)
    at /usr/include/c++/9/functional:495
#10 0x00007f269d1cc194 in _ZZN3rtc9weak_bindIMNS_4impl14PeerConnectionEFvtmES2_JRKSt12_PlaceholderILi1EERKS5_ILi2EEEEEDaOT_PT0_DpOT1_ENKUlDpOT_E_clIJtmEEEDaSL_ (this=0x7f2650015610) at /home/ubuntu/libdatachannel/include/rtc/utils.hpp:37
#11 0x00007f269d1d5822 in _ZNSt17_Function_handlerIFvtmEZN3rtc9weak_bindIMNS1_4impl14PeerConnectionEFvtmES4_JRKSt12_PlaceholderILi1EERKS7_ILi2EEEEEDaOT_PT0_DpOT1_EUlDpOT_E_E9_M_invokeERKSt9_Any_dataOtOm (__functor=..., __args#0=@0x7f2698596a34: 1, __args#1=@0x7f2698596a28: 1048000)
    at /usr/include/c++/9/bits/std_function.h:300
#12 0x00007f269d1f15c3 in std::function<void (unsigned short, unsigned long)>::operator()(unsigned short, unsigned long) const (this=0x7f265000f5f0, __args#0=1, __args#1=1048000) at /usr/include/c++/9/bits/std_function.h:688
#13 0x00007f269d1edba4 in rtc::impl::SctpTransport::triggerBufferedAmount(unsigned short, unsigned long) (this=0x7f265000f220, streamId=1, amount=1048000) at /home/ubuntu/libdatachannel/src/impl/sctptransport.cpp:543
#14 0x00007f269d1edb2c in rtc::impl::SctpTransport::updateBufferedAmount(unsigned short, long) (this=0x7f265000f220, streamId=1, delta=-800) at /home/ubuntu/libdatachannel/src/impl/sctptransport.cpp:538
#15 0x00007f269d1ed02e in rtc::impl::SctpTransport::trySendQueue() (this=0x7f265000f220) at /home/ubuntu/libdatachannel/src/impl/sctptransport.cpp:438
#16 0x00007f269d1ec2fe in rtc::impl::SctpTransport::flush() (this=0x7f265000f220) at /home/ubuntu/libdatachannel/src/impl/sctptransport.cpp:343
#17 0x00007f269d1ee23f in rtc::impl::SctpTransport::safeFlush() (this=0x7f265000f220) at /home/ubuntu/libdatachannel/src/impl/sctptransport.cpp:578
#18 0x00007f269d1f641e in std::__invoke_impl<bool, bool (rtc::impl::SctpTransport::*&)(), rtc::impl::SctpTransport*&>(std::__invoke_memfun_deref, bool (rtc::impl::SctpTransport::*&)(), rtc::impl::SctpTransport*&)
    (__f=@0x7f2650001020: (bool (rtc::impl::SctpTransport::*)(rtc::impl::SctpTransport * const)) 0x7f269d1ee20a <rtc::impl::SctpTransport::safeFlush()>, __t=@0x7f2650001030: 0x7f265000f220) at /usr/include/c++/9/bits/invoke.h:73
#19 0x00007f269d1f54a3 in std::__invoke<bool (rtc::impl::SctpTransport::*&)(), rtc::impl::SctpTransport*&>(bool (rtc::impl::SctpTransport::*&)(), rtc::impl::SctpTransport*&)
    (__fn=@0x7f2650001020: (bool (rtc::impl::SctpTransport::*)(rtc::impl::SctpTransport * const)) 0x7f269d1ee20a <rtc::impl::SctpTransport::safeFlush()>) at /usr/include/c++/9/bits/invoke.h:95
#20 0x00007f269d1f3ebf in std::_Bind<bool (rtc::impl::SctpTransport::*(rtc::impl::SctpTransport*))()>::__call<bool, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7f2650001020, __args=...) at /usr/include/c++/9/functional:400
#21 0x00007f269d1f27db in std::_Bind<bool (rtc::impl::SctpTransport::*(rtc::impl::SctpTransport*))()>::operator()<, bool>() (this=0x7f2650001020) at /usr/include/c++/9/functional:484
#22 0x00007f269d1f19e1 in rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}::operator()() (this=0x7f2650001018) at /home/ubuntu/libdatachannel/src/impl/processor.hpp:68
#23 0x00007f269d1f759e in std::__invoke_impl<bool, rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}&>(std::__invoke_other, rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#24 0x00007f269d1f6eda in std::__invoke<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}&>(bool (rtc::impl::SctpTransport::*&&)()) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#25 0x00007f269d1f6478 in std::_Bind<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1} ()>::__call<bool>(std::tuple<>&&, std::_Index_tuple<>) (this=0x7f2650001018, __args=...)
    at /usr/include/c++/9/functional:400
#26 0x00007f269d1f552b in std::_Bind<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1} ()>::operator()<, bool>() (this=0x7f2650001018) at /usr/include/c++/9/functional:484
#27 0x00007f269d1f3f2f in rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}::operator()() (this=0x7f2650001018)
    at /home/ubuntu/libdatachannel/src/impl/threadpool.hpp:109
#28 0x00007f269d1fafa7 in std::__invoke_impl<bool, rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}&>(std::__invoke_other, rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#29 0x00007f269d1faa6a in std::__invoke<rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}&>(bool (rtc::impl::SctpTransport::*&&)()) (__fn=...)
    at /usr/include/c++/9/bits/invoke.h:95
#30 0x00007f269d1fa2a1 in std::__future_base::_Task_state<rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}, std::allocator<int>, bool ()>::_M_run()::{lambda()#1}::operator()() const (this=0x7f26985978a8) at /usr/include/c++/9/future:1421
#31 0x00007f269d1fb7e8 in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}, std::allocator<int>, bool ()>::_M_run()::{lambda()#1}, bool>::operator()() const (this=0x7f26985978b0) at /usr/include/c++/9/future:1339
#32 0x00007f269d1fb067 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<bool>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}, std::allocator<int>, bool ()>::_M_run()::{lambda()#1}, bool> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/9/bits/std_function.h:286
#33 0x000056136ceacd00 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x7f26985978b0) at /usr/include/c++/9/bits/std_function.h:688
--Type <RET> for more, q to quit, c to continue without paging--
#34 0x000056136ceac1ac in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x7f2650000ff0, __f=0x7f26985978b0, __did_set=0x7f269859780f) at /usr/include/c++/9/future:561
#35 0x000056136ceb3e1d in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=
    @0x7f2698597850: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x56136ceac172 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __t=@0x7f2698597820: 0x7f2650000ff0) at /usr/include/c++/9/bits/invoke.h:73
#36 0x000056136ceaea55 in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=
    @0x7f2698597850: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x56136ceac172 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/9/bits/invoke.h:95
#37 0x000056136ceaca64 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=0x7f26985977a0) at /usr/include/c++/9/mutex:671
#38 0x000056136ceaca93 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=0x0) at /usr/include/c++/9/mutex:676
#39 0x000056136ceacaa8 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/9/mutex:676
#40 0x00007f269ca8a47f in __pthread_once_slow (once_control=0x7f2650001008, init_routine=0x7f269cb8bb80 <__once_proxy>) at pthread_once.c:116
#41 0x000056136ceaaae0 in __gthread_once(__gthread_once_t*, void (*)()) (__once=0x7f2650001008, __func=0x7f269cb8bb80 <__once_proxy>) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:700
#42 0x000056136ceacb52 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=
    @0x7f2698597850: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x56136ceac172 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/9/mutex:683
#43 0x000056136ceabf7f in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x7f2650000ff0, __res=..., __ignore_failure=false) at /usr/include/c++/9/future:401
#44 0x00007f269d1fa313 in std::__future_base::_Task_state<rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#1}, std::allocator<int>, bool ()>::_M_run()
    (this=0x7f2650000ff0) at /usr/include/c++/9/future:1423
#45 0x00007f269d1f5745 in std::packaged_task<bool ()>::operator()() (this=0x7f265001a850) at /usr/include/c++/9/future:1551
#46 0x00007f269d1f40f7 in rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#2}::operator()() const (this=0x7f266029d890)
    at /home/ubuntu/libdatachannel/src/impl/threadpool.hpp:117
#47 0x00007f269d1f66c8 in std::_Function_handler<void (), rtc::impl::ThreadPool::schedule<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>(std::chrono::time_point<std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::_V2::steady_clock, std::future<std::invoke_result<std::decay<rtc::impl::Processor::enqueue<bool (rtc::impl::SctpTransport::*)(), rtc::impl::SctpTransport*>(bool (rtc::impl::SctpTransport::*&&)(), rtc::impl::SctpTransport*&&)::{lambda()#1}>::type>::type>::duration<long, std::ratio<1l, 1000000000l> > >, bool (rtc::impl::SctpTransport::*&&)())::{lambda()#2}>::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/9/bits/std_function.h:300
#48 0x00007f269d1799b6 in std::function<void ()>::operator()() const (this=0x7f26985979a0) at /usr/include/c++/9/bits/std_function.h:688
#49 0x00007f269d1fc869 in rtc::impl::ThreadPool::runOne() (this=0x56136d247550) at /home/ubuntu/libdatachannel/src/impl/threadpool.cpp:75
#50 0x00007f269d1fc80e in rtc::impl::ThreadPool::run() (this=0x56136d247550) at /home/ubuntu/libdatachannel/src/impl/threadpool.cpp:69
#51 0x00007f269d1ff60b in std::__invoke_impl<void, void (rtc::impl::ThreadPool::*&)(), rtc::impl::ThreadPool*&>(std::__invoke_memfun_deref, void (rtc::impl::ThreadPool::*&)(), rtc::impl::ThreadPool*&)
    (__f=@0x56136d247c38: (void (rtc::impl::ThreadPool::*)(rtc::impl::ThreadPool * const)) 0x7f269d1fc7f2 <rtc::impl::ThreadPool::run()>, __t=@0x56136d247c48: 0x56136d247550) at /usr/include/c++/9/bits/invoke.h:73
#52 0x00007f269d1ff557 in std::__invoke<void (rtc::impl::ThreadPool::*&)(), rtc::impl::ThreadPool*&>(void (rtc::impl::ThreadPool::*&)(), rtc::impl::ThreadPool*&) (__fn=@0x56136d247c38: (void (rtc::impl::ThreadPool::*)(rtc::impl::ThreadPool * const)) 0x7f269d1fc7f2 <rtc::impl::ThreadPool::run()>)
    at /usr/include/c++/9/bits/invoke.h:95
#53 0x00007f269d1ff4c1 in std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x56136d247c38, __args=...) at /usr/include/c++/9/functional:400
#54 0x00007f269d1ff44f in std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()>::operator()<, void>() (this=0x56136d247c38) at /usr/include/c++/9/functional:484
#55 0x00007f269d1ff3f6 in std::__invoke_impl<void, std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()>>(std::__invoke_other, std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()>&&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#56 0x00007f269d1ff39f in std::__invoke<std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()>>(std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()>&&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#57 0x00007f269d1ff340 in std::thread::_Invoker<std::tuple<std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x56136d247c38) at /usr/include/c++/9/thread:244
#58 0x00007f269d1ff312 in std::thread::_Invoker<std::tuple<std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()> > >::operator()() (this=0x56136d247c38) at /usr/include/c++/9/thread:251
#59 0x00007f269d1ff2f2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (rtc::impl::ThreadPool::*(rtc::impl::ThreadPool*))()> > > >::_M_run() (this=0x56136d247c30) at /usr/include/c++/9/thread:195
#60 0x00007f269cb8cd84 in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#61 0x00007f269ca81609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#62 0x00007f269c9a8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Wondering if this could provide some insight into #354 or whether this is a separate issue altogether.

For context, we built libdatachannel under Ubuntu 20.04.2 LTS with flags 

cmake -B build -DUSE_GNUTLS=0 -DUSE_NICE=1 -DCMAKE_BUILD_TYPE=Debug

I can provide more details on how to reproduce this if necessary. Shortly put: we forked this repo to implement our C++-based example: https://github.com/streamr-dev/libdatachannel/tree/master/examples/multi-node-benchmark but reused our nodejs-based signaling server.

harbu commented 3 years ago

Update: problem could be on our side as well, as the source of the error is our code. Maybe our logging method is thread unsafe?

cout << now() << logId << " DataChannel " << peerId << " LOW buffer " <<  dc->bufferedAmount()<< endl;
ptesavol commented 3 years ago

The complete code of our onBufferedAmountLow callback where the crash happens is 

dc->onBufferedAmountLow([obj = this, logId = this->logId, peerId = this->peerId, dc= this->dc] ()
        {
        obj->paused = false;
        cout << now() << logId << " DataChannel " << peerId << " LOW buffer " <<  dc->bufferedAmount()<< endl;
        });

I wonder if we are using the closure correctly..

paullouisageneau commented 3 years ago

It looks like there is indeed an issue with the log line itself. Is the captured dc set at this point? this->dc it assigned only later line 93.

ptesavol commented 3 years ago

Well spotted! Apparently we followed the typescript version too faithfully, without noticing that a C++ smart pointer works quite differently compared to a Javascript reference.