search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.94k stars 1.06k forks source link

[CDC] SIGABRT encountered during stress test #2735

Open ajcaldera1 opened 4 years ago

ajcaldera1 commented 4 years ago

Encountered during stress test in tserver.err file (NO FATAL):

pure virtual method called
terminate called without an active exception
*** Aborted at 1572023210 (unix time) try "date -d @1572023210" if you are using GNU date ***
PC: @     0x7f06d40f1817 (unknown)
*** SIGABRT (@0x3e500000d1b) received by PID 3355 (TID 0x7f062bce2700) from PID 3355; stack trace: ***
    @     0x7f06d42ad360 pthread_timedjoin_np
    @     0x7f06d40f1817 (unknown)
    @     0x7f06d40f2b7a (unknown)
    @     0x7f06d466b5cc  __gnu_cxx::__verbose_terminate_handler()
    @     0x7f06d4669636 (unknown)
    @     0x7f06d4669680  std::terminate()
    @     0x7f06d466a12f __cxa_pure_virtual
    @     0x7f06d795aee1 (unknown)
    @     0x7f06d61d1f9c (unknown)
    @     0x7f06d61d21d9 (unknown)
    @     0x7f06d621f9a7  yb::rpc::Connection::HandleTimeout()
    @     0x7f06d5bd4442  yb::(anonymous namespace)::PosixRWFile::Sync()
    @     0x7f06d42a4fc4 (unknown)
    @     0x7f06d41a49bd build_trtable
    @                0x0 (unknown)

Backtrace from core file found:

#0  0x00007f06d40f1817 in raise () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libc.so.6
#1  0x00007f06d40f2b7a in abort () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libc.so.6
#2  0x00007f06d466b5cd in __gnu_cxx::__verbose_terminate_handler() () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#3  0x00007f06d4669636 in ?? () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#4  0x00007f06d4669681 in std::terminate() () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#5  0x00007f06d466a12f in __cxa_pure_virtual () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libstdc++.so.6
#6  0x00007f06d795aee1 in operator()<yb::Status::OK&, void> (__object=<optimized out>, this=<optimized out>) at /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:600
#7  __call<void, 0ul, 1ul> (__args=<optimized out>, this=<optimized out>) at /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:1074
#8  operator()<, void> (this=<optimized out>) at /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:1133
#9  std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (yb::cdc::CDCReadRpc::*)(yb::Status const&)> (yb::cdc::CDCReadRpc*, yb::Status::OK)> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:1871
#10 0x00007f06d61d1f9c in operator() (this=<optimized out>) at /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:2267
#11 yb::rpc::OutboundCall::InvokeCallbackSync (this=0x260358d0) at ../../src/yb/rpc/outbound_call.cc:384
#12 0x00007f06d61d21d9 in yb::rpc::InvokeCallbackTask::Run (this=<optimized out>) at ../../src/yb/rpc/outbound_call.cc:165
#13 0x00007f06d621f9a8 in yb::rpc::(anonymous namespace)::Worker::Execute (this=<optimized out>) at ../../src/yb/rpc/thread_pool.cc:99
#14 0x00007f06d5bd4443 in operator() (this=0x1d57ec8) at /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-/Cellar/gcc/5.5.0_4/include/c++/5.5.0/functional:2267
#15 yb::Thread::SuperviseThread (arg=0x1d57ea0) at ../../src/yb/util/thread.cc:741
#16 0x00007f06d42a4fc4 in start_thread () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libpthread.so.0
#17 0x00007f06d41a49bd in clone () from /home/yugabyte/yb-software/yugabyte-2.0.3.0-bbbd34d12fb1255aeee5b61720c8250031d78d84-release-centos-x86_64/linuxbrew/lib/libc.so.6
nspiegelberg commented 4 years ago

9 std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (yb::cdc::CDCReadRpc::)(yb::Status const&)> (yb::cdc::CDCReadRpc, yb::Status::OK)>

That's the lambda in the new LeaderGetChanges read path in cdc_service.cc