Closed Tryneus closed 8 years ago
Just got another crash, probably the same problem but a slightly different code path:
error: Error in ../src/arch/runtime/thread_pool.cc at line 359:
error: Segmentation fault from reading the address (nil).
error: Backtrace:
error: Tue Oct 20 16:32:42 2015
1 [0x1d1fbf8]: rethinkdb_backtrace(void**, int) at rethinkdb_backtrace.cc:101
2 [0x19085c1]: backtrace_t::backtrace_t() at backtrace.cc:203
3 [0x1908c2f]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:283
4 [0x190850f]: format_backtrace(bool) at backtrace.cc:198
5 [0x18fc3ce]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:83
6 [0x172eac8]: linux_thread_pool_t::fatal_signal_handler(int, siginfo*, void*) at thread_pool.cc:359
7 [0x7f2b3fd7acb0]: /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f2b3fd7acb0] at 0x7f2b3fd7acb0 (/lib/x86_64-linux-gnu/libpthread.so.0)
8 [0x18aebd0]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x18aebd0] at 0x18aebd0 ()
9 [0x18ae5a4]: mailbox_manager_t::on_message(connectivity_cluster_t::connection_t*, auto_drainer_t::lock_t, read_stream_t*) at mailbox.cc:245
10 [0x183cb14]: connectivity_cluster_t::run_t::handle(keepalive_tcp_conn_stream_t*, boost::optional<peer_id_t>, boost::optional<peer_address_t>, auto_drainer_t::lock_t, bool*) at cluster.cc:1192
11 [0x183a850]: connectivity_cluster_t::run_t::connect_to_peer(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*) at cluster.cc:333
12 [0x1852f88]: std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)>::operator()(connectivity_cluster_t::run_t*, peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*) const at functional:551
13 [0x1850eab]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>::__call_c<void, long&, 0, 1, 2, 3, 4, 5, 6>(std::tuple<long&>&&, std::_Index_tuple<0, 1, 2, 3, 4, 5, 6>) const at functional:1155
14 [0x184e267]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>::operator()<long&, void>(long&) const at functional:1220
15 [0x184a611]: pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long>::operator()() at pmap.hpp:21
16 [0x1855c10]: callable_action_instance_t<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long> >::run_action() at callable_action.hpp:31
17 [0x1729928]: callable_action_wrapper_t::run() at runtime_utils.cc:43
18 [0x1731b2f]: coro_t::run() at coroutines.cc:207
19 [0x184a5db]: void coro_t::spawn_now_dangerously<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long> >(pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)>, long>&&) at coroutines.hpp:53
20 [0x1846e03]: void pmap<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> >(long, long, std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> const&) at pmap.hpp:43
21 [0x1842da5]: void pmap<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> >(long, std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t const*, int, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, co_semaphore_t*)> (connectivity_cluster_t::run_t*, peer_address_t const*, std::_Placeholder<1>, boost::optional<peer_id_t>, auto_drainer_t::lock_t, bool*, static_semaphore_t*)> const&) at pmap.hpp:54
22 [0x183ad0b]: connectivity_cluster_t::run_t::join_blocking(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t) at cluster.cc:376
23 [0x1856969]: std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)>::operator()(connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t) const at functional:551
24 [0x18563aa]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> (connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)>::__call<void, , 0, 1, 2, 3>(std::tuple<>&&, std::_Index_tuple<0, 1, 2, 3>) at functional:1146
25 [0x185604c]: void std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> (connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)>::operator()<, void>() at functional:1206
26 [0x1855c2e]: callable_action_instance_t<std::_Bind<std::_Mem_fn<void (connectivity_cluster_t::run_t::*)(peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> (connectivity_cluster_t::run_t*, peer_address_t, boost::optional<peer_id_t>, auto_drainer_t::lock_t)> >::run_action() at callable_action.hpp:31
27 [0x1729928]: callable_action_wrapper_t::run() at runtime_utils.cc:43
28 [0x1731b2f]: coro_t::run() at coroutines.cc:207
gdb analysis of the core file shows a completely different backtrace:
#3 0x0000000001b0b8c0 in operator() (this=0x7f2b0c7f1700, sub=0x7f2b28187c00) at ../src/rdb_protocol/changefeed.cc:2308
#4 0x0000000001b26f20 in std::_Function_handler<void(ql::changefeed::range_sub_t*), ql::changefeed::msg_visitor_t::operator()(const ql::changefeed::msg_t::change_t&) const::<lambda(ql::changefeed::range_sub_t*)> >::_M_invoke(const std::_Any_data &, ql::changefeed::range_sub_t *) (__functor=..., __args#0=0x7f2b28187c00) at /usr/include/c++/4.6/functional:1778
#5 0x0000000001b3a071 in std::function<void(ql::changefeed::range_sub_t*)>::operator()(ql::changefeed::range_sub_t *) const (this=0x7f2b0ab04e90, __args#0=0x7f2b28187c00)
at /usr/include/c++/4.6/functional:2161
#6 0x0000000001b28ca8 in ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) (this=0x7f2b0d7ff200, f=...,
vec=std::vector of length 25, capacity 25 = {...}, subscription_threads=std::vector of length 1, capacity 1 = {...}, i=0) at ../src/rdb_protocol/changefeed.cc:2933
#7 0x0000000001b5ec3b in std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>::operator()(ql::changefeed::feed_t *, const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) const (this=0x7f2b0ab04800,
__object=0x7f2b0d7ff200, __args#0=..., __args#1=std::vector of length 25, capacity 25 = {...}, __args#2=std::vector of length 1, capacity 1 = {...}, __args#3=0)
at /usr/include/c++/4.6/functional:551
#8 0x0000000001b58a07 in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::__call_c<void, long&, 0, 1, 2, 3, 4>(std::tuple<long&> &, std::_Index_tuple<0, 1, 2, 3, 4>) const (this=0x7f2b0ab04800,
__args=empty std::tuple) at /usr/include/c++/4.6/functional:1155
#9 0x0000000001b512c5 in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::operator()<long int&, void>(void) const (this=0x7f2b0ab04800) at /usr/include/c++/4.6/functional:1219
#10 0x0000000001b46f09 in pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>, long int>::operator()(void) (this=0x7f2b0ab04750) at ../src/concurrency/pmap.hpp:20
#11 0x0000000001b3a333 in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (begin=0,
end=1, c=...) at ../src/concurrency/pmap.hpp:47
#12 0x0000000001b28e00 in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (count=1, c=...)
at ../src/concurrency/pmap.hpp:53
#13 0x0000000001b1a01a in ql::changefeed::feed_t::each_sub_in_vec<ql::changefeed::range_sub_t>(const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, rwlock_in_line_t *, const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7f2b0d7ff200,
vec=std::vector of length 25, capacity 25 = {...}, spot=0x7f2b0ab048f0, lock=..., f=...) at ../src/rdb_protocol/changefeed.cc:2916
#14 0x0000000001afd95c in ql::changefeed::feed_t::each_range_sub(const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7f2b0d7ff200, lock=...,
f=...) at ../src/rdb_protocol/changefeed.cc:2942
This looks a little more useful (although it's hard to say if it's more correct than the other backtraces). Inspecting sub
shows an almost certainly corrupted (or already freed) object. The binary, core file, and server folders are available on newton
in ~grey/issue_4972
.
Are these crashes on next
or 2.1.x
?
These were on next
(153a96a). The test framework was recently reworked a bit, and it appears the version in v2.1.x
deadlocks with this version of the table fuzzer. Still, I ran the test against v2.1.x
and could not reproduce the crash. I'll keep trying, as I've run into other problems.
I got one of these on the v2.1.x
build last night (a007331), core file and server files available on newton in ~grey/issue_4972_2/
.
Backtrace from gdb:
#0 0x0000000001604ba5 in linux_thread_pool_t::fatal_signal_handler (signum=11, info=0x7fb00903e570, data=0x7fb00903e440)
at ../src/arch/runtime/thread_pool.cc:359
#1 <signal handler called>
#2 0x00000000019017b5 in operator() (this=0x7fafe9d63860, sub=0x7fb009217800)
at ../src/rdb_protocol/changefeed.cc:2306
#3 0x000000000191deae in std::_Function_handler<void(ql::changefeed::range_sub_t*), ql::changefeed::msg_visitor_t::operator()(const ql::changefeed::msg_t::change_t&) const::<lambda(ql::changefeed::range_sub_t*)> >::_M_invoke(const std::_Any_data &, ql::changefeed::range_sub_t *) (__functor=..., __args#0=0x7fb009217800)
at /usr/include/c++/4.6/functional:1778
#4 0x0000000001931cfb in std::function<void(ql::changefeed::range_sub_t*)>::operator()(ql::changefeed::range_sub_t *) const (this=0x7faff6d84e90, __args#0=0x7fb009217800)
at /usr/include/c++/4.6/functional:2161
#5 0x000000000191fc36 in ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) (this=0x7fafe2bfcf00, f=..., vec=std::vector of length 25, capacity 25 = {...}, subscription_threads=std::vector of length 1, capacity 1 = {...}, i=0)
at ../src/rdb_protocol/changefeed.cc:2932
#6 0x00000000019555bb in std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>::operator()(ql::changefeed::feed_t *, const std::function<void(ql::changefeed::range_sub_t*)> &, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, const std::vector<int, std::allocator<int> > &, int) const (this=0x7faff6d84800, __object=0x7fafe2bfcf00, __args#0=..., __args#1=std::vector of length 25, capacity 25 = {...}, __args#2=std::vector of length 1, capacity 1 = {...}, __args#3=0)
at /usr/include/c++/4.6/functional:551
#7 0x000000000194fb4b in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::__call_c<void, long&, 0, 1, 2, 3, 4>(std::tuple<long&> &, std::_Index_tuple<0, 1, 2, 3, 4>) const (this=0x7faff6d84800, __args=empty std::tuple)
at /usr/include/c++/4.6/functional:1155
#8 0x0000000001948a71 in std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>::operator()<long int&, void>(void) const (this=0x7faff6d84800)
at /usr/include/c++/4.6/functional:1219
#9 0x000000000193f2e3 in pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)>, long int>::operator()(void) (this=0x7faff6d84750)
at ../src/concurrency/pmap.hpp:20
#10 0x0000000001931fbd in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (begin=0, end=1, c=...)
at ../src/concurrency/pmap.hpp:47
#11 0x000000000191fd8e in pmap<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*> >&, const std::vector<int>&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*> > >, std::reference_wrapper<const std::vector<int> >, std::_Placeholder<1>)> >(int64_t, const std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(const std::function<void(ql::changefeed::range_sub_t*)>&, const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > >&, const std::vector<int, std::allocator<int> >&, int)>(ql::changefeed::feed_t*, std::reference_wrapper<const std::function<void(ql::changefeed::range_sub_t*)> >, std::reference_wrapper<const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > >, std::reference_wrapper<const std::vector<int, std::allocator<int> > >, std::_Placeholder<1>)> &) (count=1, c=...)
at ../src/concurrency/pmap.hpp:53
#12 0x00000000019106f0 in ql::changefeed::feed_t::each_sub_in_vec<ql::changefeed::range_sub_t>(const std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > &, rwlock_in_line_t *, const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7fafe2bfcf00, vec=std::vector of length 25, capacity 25 = {...}, spot=0x7faff6d848f0, lock=..., f=...)
at ../src/rdb_protocol/changefeed.cc:2915
#13 0x00000000018f386e in ql::changefeed::feed_t::each_range_sub(const auto_drainer_t::lock_t &, const std::function<void(ql::changefeed::range_sub_t*)> &) (this=0x7fafe2bfcf00, lock=..., f=...)
at ../src/rdb_protocol/changefeed.cc:2941
#14 0x0000000001902d5d in ql::changefeed::msg_visitor_t::operator() (this=0x7faff6d85380, change=...)
at ../src/rdb_protocol/changefeed.cc:2392
#15 0x0000000001958f9b in boost::detail::variant::invoke_visitor<ql::changefeed::msg_visitor_t>::internal_visit<ql::changefeed::msg_t::change_t const> (this=0x7faff6d85120, operand=...)
at /usr/local/include/boost/variant/variant.hpp:858
...
This is on the same sub->active()
call that I got from the last backtrace on next
, and inspection of the sub
object once again shows corruption of some kind. The segfault is probably due to following the corrupted vtable pointer.
Assigning @mlucy . Since as far as we know nobody has run into this in production and it might require rapid restarting of servers, I'm going to move the issue to the 2.2 milestone though. We should still probably backport the fix to v2.1.x if it's not too hard.
This looks like https://github.com/rethinkdb/rethinkdb/issues/4878 to me, so I guess that probably wasn't fixed by the changes in the patch.
I think I found it (and the fuzzer doesn't seem to be producing this crash for me any more). This is in CR 3296 by @danielmewes .
A fix for this is in next and v2.1.x.
(Or what I believe is a fix. @Tryneus, let me know if you encounter this again.)
I was testing the new
--kill
option fortable_fuzzer.py
, and ran into this crash with the command linetable_fuzzer.py --servers 7 --threads 48 --changefeeds --kill