LeakSanitizer: detected memory leaks from seastar::net::conntrack::load_balancer

[cyx1231st]

Some critical tasks can be missed during engine exit, causing LeakSanitizer failure. Notably, there is a chance to lose the destruction task of foreign_ptr<lw_shared_ptr<conntrack::load_balancer>> submitted by conntrack::~handle() because its future is ignored, see https://github.com/scylladb/seastar/blob/65a8bedb02eaa0c82ae2ee3e47085bc73fb05382/include/seastar/net/posix-stack.hh#L87-L90

Detailed leak report:

==16836==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x7f9d1e86bc7f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dc7f)
    #1 0x55be9e4dc737 in std::_MakeUniq<seastar::smp_message_queue::async_work_item<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1}> >::__single_object std::make_unique<seastar::smp_message_queue::async_work_item<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1}>, seastar::smp_message_queue&, seastar::smp_service_group&, {lambda()#1}>(seastar::smp_message_queue&, seastar::smp_service_group&, {lambda()#1}&&) /usr/include/c++/9/bits/unique_ptr.h:849
    #2 0x55be9e4d058a in seastar::futurize<std::result_of<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1} ()>::type>::type seastar::smp_message_queue::submit<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, std::result_of&&) /root/ceph/src/seastar/include/seastar/core/smp.hh:243
    #3 0x55be9e4c1668 in seastar::futurize<std::result_of<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1} ()>::type>::type seastar::smp::submit_to<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, std::result_of&&) /root/ceph/src/seastar/include/seastar/core/smp.hh:329
    #4 0x55be9e4af502 in seastar::futurize<std::result_of<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1} ()>::type>::type seastar::smp::submit_to<seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int)::{lambda()#1}>(unsigned int, std::result_of&&) /root/ceph/src/seastar/include/seastar/core/smp.hh:348
    #5 0x55be9e4997c3 in seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::destroy(seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>, unsigned int) /root/ceph/src/seastar/include/seastar/core/sharded.hh:778
    #6 0x55be9e4893a9 in seastar::foreign_ptr<seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> >::~foreign_ptr() /root/ceph/src/seastar/include/seastar/core/sharded.hh:809
    #7 0x55be9e472e2c in seastar::net::conntrack::handle::~handle()::{lambda()#1}::~handle() /root/ceph/src/seastar/include/seastar/net/posix-stack.hh:88
    #8 0x55be9e4d1080 in seastar::smp_message_queue::async_work_item<seastar::net::conntrack::handle::~handle()::{lambda()#1}>::~async_work_item() /root/ceph/src/seastar/include/seastar/core/smp.hh:192
    #9 0x55be9e4d1119 in seastar::smp_message_queue::async_work_item<seastar::net::conntrack::handle::~handle()::{lambda()#1}>::~async_work_item() /root/ceph/src/seastar/include/seastar/core/smp.hh:192
    #10 0x55be9dd7c74a in operator() /root/ceph/src/seastar/src/core/reactor.cc:3042
    #11 0x55be9ddc2d20 in process_queue<4, seastar::smp_message_queue::process_completions(seastar::shard_id)::<lambda(seastar::smp_message_queue::work_item*)> > /root/ceph/src/seastar/src/core/reactor.cc:3030
    #12 0x55be9dd7c81a in seastar::smp_message_queue::process_completions(unsigned int) /root/ceph/src/seastar/src/core/reactor.cc:3038
    #13 0x55be9dd8f4d2 in seastar::smp::poll_queues() /root/ceph/src/seastar/src/core/reactor.cc:3867
    #14 0x55be9de84385 in seastar::reactor::smp_pollfn::poll() (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x49d0385)
    #15 0x55be9dd763e0 in seastar::reactor::poll_once() /root/ceph/src/seastar/src/core/reactor.cc:2757
    #16 0x55be9dd71a01 in operator() /root/ceph/src/seastar/src/core/reactor.cc:2655
    #17 0x55be9dddf2de in _M_invoke /usr/include/c++/9/bits/std_function.h:285
    #18 0x55be9decdd34 in std::function<bool ()>::operator()() const /usr/include/c++/9/bits/std_function.h:688
    #19 0x55be9dd7419e in seastar::reactor::run() /root/ceph/src/seastar/src/core/reactor.cc:2681
    #20 0x55be9dd8868a in operator() /root/ceph/src/seastar/src/core/reactor.cc:3806
    #21 0x55be9dde0f1c in _M_invoke /usr/include/c++/9/bits/std_function.h:300
    #22 0x55be9dbdb838 in std::function<void ()>::operator()() const /usr/include/c++/9/bits/std_function.h:688
    #23 0x55be9e7a0974 in seastar::posix_thread::start_routine(void*) /root/ceph/src/seastar/src/core/posix.cc:52
    #24 0x7f9d1d9446da in start_thread /build/glibc-OTsEL5/glibc-2.27/nptl/pthread_create.c:463

Indirect leak of 288 byte(s) in 1 object(s) allocated from:
    #0 0x7f9d1e86bc7f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dc7f)
    #1 0x55be9e05a788 in __gnu_cxx::new_allocator<unsigned int>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
    #2 0x55be9e01a111 in std::allocator_traits<std::allocator<unsigned int> >::allocate(std::allocator<unsigned int>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
    #3 0x55be9dfc24e7 in std::_Vector_base<unsigned int, std::allocator<unsigned int> >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
    #4 0x55be9df6f79a in std::_Vector_base<unsigned int, std::allocator<unsigned int> >::_M_create_storage(unsigned long) /usr/include/c++/9/bits/stl_vector.h:358
    #5 0x55be9def1b65 in std::_Vector_base<unsigned int, std::allocator<unsigned int> >::_Vector_base(unsigned long, std::allocator<unsigned int> const&) /usr/include/c++/9/bits/stl_vector.h:302
    #6 0x55be9e488da8 in std::vector<unsigned int, std::allocator<unsigned int> >::vector(unsigned long, unsigned int const&, std::allocator<unsigned int> const&) /usr/include/c++/9/bits/stl_vector.h:521
    #7 0x55be9e471ecc in seastar::net::conntrack::load_balancer::load_balancer() /root/ceph/src/seastar/include/seastar/net/posix-stack.hh:46
    #8 0x55be9e49a129 in seastar::shared_ptr_no_esft<seastar::net::conntrack::load_balancer>::shared_ptr_no_esft() /root/ceph/src/seastar/include/seastar/core/shared_ptr.hh:160
    #9 0x55be9e49a201 in seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>::make<>() /root/ceph/src/seastar/include/seastar/core/shared_ptr.hh:266
    #10 0x55be9e489e54 in seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> seastar::make_lw_shared<seastar::net::conntrack::load_balancer>() /root/ceph/src/seastar/include/seastar/core/shared_ptr.hh:416
    #11 0x55be9e473481 in seastar::net::conntrack::conntrack() /root/ceph/src/seastar/include/seastar/net/posix-stack.hh:98
    #12 0x55be9e476130 in seastar::net::posix_server_socket_impl::posix_server_socket_impl(int, seastar::socket_address, seastar::pollable_fd, seastar::api_v2::server_socket::load_balancing_algorithm, unsigned int, std::pmr::polymorphic_allocator<char>*) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4fc2130)
    #13 0x55be9e4938ea in std::_MakeUniq<seastar::net::posix_server_socket_impl>::__single_object std::make_unique<seastar::net::posix_server_socket_impl, int&, seastar::socket_address&, seastar::pollable_fd, seastar::api_v2::server_socket::load_balancing_algorithm&, unsigned int&, std::pmr::polymorphic_allocator<char>*&>(int&, seastar::socket_address&, seastar::pollable_fd&&, seastar::api_v2::server_socket::load_balancing_algorithm&, unsigned int&, std::pmr::polymorphic_allocator<char>*&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4fdf8ea)
    #14 0x55be9e447c89 in seastar::net::posix_network_stack::listen(seastar::socket_address, seastar::listen_options) /root/ceph/src/seastar/src/net/posix-stack.cc:614
    #15 0x55be9dd47e70 in seastar::reactor::listen(seastar::socket_address, seastar::listen_options) /root/ceph/src/seastar/src/core/reactor.cc:1445
    #16 0x55be9dd979c0 in seastar::listen(seastar::socket_address, seastar::listen_options) /root/ceph/src/seastar/src/core/reactor.cc:4063
    #17 0x55be9d9a8454 in auto crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}::operator()<crimson::net::FixedCPUServerSocket>(crimson::net::FixedCPUServerSocket&) const (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x44f4454)
    #18 0x55be9da3017c in seastar::future<> seastar::do_void_futurize_helper<void>::apply<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&>(crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x457c17c)
    #19 0x55be9da05fa2 in seastar::future<> seastar::futurize<void>::apply<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&>(crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4551fa2)
    #20 0x55be9d9d8bd4 in seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}>(seastar::smp_submit_to_options, crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&&)::{lambda(crimson::net::FixedCPUServerSocket&)#1}::operator()(crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4524bd4)
    #21 0x55be9da3049c in std::_Function_handler<seastar::future<> (crimson::net::FixedCPUServerSocket&), seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}>(seastar::smp_submit_to_options, crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&&)::{lambda(crimson::net::FixedCPUServerSocket&)#1}>::_M_invoke(std::_Any_data const&, crimson::net::FixedCPUServerSocket&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x457c49c)
    #22 0x55be9da2ee9b in std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>::operator()(crimson::net::FixedCPUServerSocket&) const (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x457ae9b)
    #23 0x55be9da04e17 in seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}::operator()() const (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4550e17)
    #24 0x55be9da502fe in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&>(seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x459c2fe)
    #25 0x55be9da8f438 in seastar::smp_message_queue::async_work_item<seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::run_and_dispose() (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x45db438)
    #26 0x55be9dd6a232 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /root/ceph/src/seastar/src/core/reactor.cc:2091
    #27 0x55be9dd6e6ec in seastar::reactor::run_some_tasks() /root/ceph/src/seastar/src/core/reactor.cc:2506
    #28 0x55be9dd73c25 in seastar::reactor::run() /root/ceph/src/seastar/src/core/reactor.cc:2661
    #29 0x55be9dba8da4 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /root/ceph/src/seastar/src/core/app-template.cc:199

Indirect leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7f9d1e86bc7f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dc7f)
    #1 0x55be9e49a14a in seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer>::make<>() /root/ceph/src/seastar/include/seastar/core/shared_ptr.hh:266
    #2 0x55be9e489e54 in seastar::lw_shared_ptr<seastar::net::conntrack::load_balancer> seastar::make_lw_shared<seastar::net::conntrack::load_balancer>() /root/ceph/src/seastar/include/seastar/core/shared_ptr.hh:416
    #3 0x55be9e473481 in seastar::net::conntrack::conntrack() /root/ceph/src/seastar/include/seastar/net/posix-stack.hh:98
    #4 0x55be9e476130 in seastar::net::posix_server_socket_impl::posix_server_socket_impl(int, seastar::socket_address, seastar::pollable_fd, seastar::api_v2::server_socket::load_balancing_algorithm, unsigned int, std::pmr::polymorphic_allocator<char>*) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4fc2130)
    #5 0x55be9e4938ea in std::_MakeUniq<seastar::net::posix_server_socket_impl>::__single_object std::make_unique<seastar::net::posix_server_socket_impl, int&, seastar::socket_address&, seastar::pollable_fd, seastar::api_v2::server_socket::load_balancing_algorithm&, unsigned int&, std::pmr::polymorphic_allocator<char>*&>(int&, seastar::socket_address&, seastar::pollable_fd&&, seastar::api_v2::server_socket::load_balancing_algorithm&, unsigned int&, std::pmr::polymorphic_allocator<char>*&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4fdf8ea)
    #6 0x55be9e447c89 in seastar::net::posix_network_stack::listen(seastar::socket_address, seastar::listen_options) /root/ceph/src/seastar/src/net/posix-stack.cc:614
    #7 0x55be9dd47e70 in seastar::reactor::listen(seastar::socket_address, seastar::listen_options) /root/ceph/src/seastar/src/core/reactor.cc:1445
    #8 0x55be9dd979c0 in seastar::listen(seastar::socket_address, seastar::listen_options) /root/ceph/src/seastar/src/core/reactor.cc:4063
    #9 0x55be9d9a8454 in auto crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}::operator()<crimson::net::FixedCPUServerSocket>(crimson::net::FixedCPUServerSocket&) const (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x44f4454)
    #10 0x55be9da3017c in seastar::future<> seastar::do_void_futurize_helper<void>::apply<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&>(crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x457c17c)
    #11 0x55be9da05fa2 in seastar::future<> seastar::futurize<void>::apply<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&>(crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&, crimson::net::FixedCPUServerSocket&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4551fa2)
    #12 0x55be9d9d8bd4 in seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}>(seastar::smp_submit_to_options, crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&&)::{lambda(crimson::net::FixedCPUServerSocket&)#1}::operator()(crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4524bd4)
    #13 0x55be9da3049c in std::_Function_handler<seastar::future<> (crimson::net::FixedCPUServerSocket&), seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all<crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}>(seastar::smp_submit_to_options, crimson::net::FixedCPUServerSocket::listen(entity_addr_t)::{lambda(auto:1&)#1}&&)::{lambda(crimson::net::FixedCPUServerSocket&)#1}>::_M_invoke(std::_Any_data const&, crimson::net::FixedCPUServerSocket&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x457c49c)
    #14 0x55be9da2ee9b in std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>::operator()(crimson::net::FixedCPUServerSocket&) const (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x457ae9b)
    #15 0x55be9da04e17 in seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}::operator()() const (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x4550e17)
    #16 0x55be9da502fe in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&>(seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&) (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x459c2fe)
    #17 0x55be9da8f438 in seastar::smp_message_queue::async_work_item<seastar::sharded<crimson::net::FixedCPUServerSocket>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<> (crimson::net::FixedCPUServerSocket&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::run_and_dispose() (/root/ceph/obj-x86_64-linux-gnu-debug/bin/unittest_seastar_socket+0x45db438)
    #18 0x55be9dd6a232 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /root/ceph/src/seastar/src/core/reactor.cc:2091
    #19 0x55be9dd6e6ec in seastar::reactor::run_some_tasks() /root/ceph/src/seastar/src/core/reactor.cc:2506
    #20 0x55be9dd73c25 in seastar::reactor::run() /root/ceph/src/seastar/src/core/reactor.cc:2661
    #21 0x55be9dba8da4 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /root/ceph/src/seastar/src/core/app-template.cc:199
    #22 0x55be9dba6c55 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /root/ceph/src/seastar/src/core/app-template.cc:115
    #23 0x55be9dba71f5 in seastar::app_template::run(int, char**, std::function<seastar::future<> ()>&&) /root/ceph/src/seastar/src/core/app-template.cc:130
    #24 0x55be9d7ccaa1 in main /root/ceph/src/test/crimson/test_socket.cc:427
    #25 0x7f9d1c071b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: 408 byte(s) leaked in 3 allocation(s).

[bhalevy]

@gleb-cloudius can you please look into this issue?

[dotnwat]

@bhalevy we also see this occasionally. here is a more recent trace from a few days ago. it sort of looks like when conn_q is being destroyed via thread local storage clean-up, that a new continuation is being placed onto a reactor task queue which doesn't get cleaned-up before process exits.

Direct leak of 88 byte(s) in 1 object(s) allocated from:

    #0 0x557c91ca5b7d in operator new(unsigned long) /v/llvm/llvm/src/compiler-rt/lib/asan/asan_new_delete.cpp:95:3

    #1 0x557ca3e3cc08 in void seastar::future<void>::schedule<seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> >, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&)>(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&&, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> >&&, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> >&&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&, seastar::future_state<seastar::internal::monostate>&&)&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1395:20

    #2 0x557ca3e3ca72 in seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1585:9

    #3 0x557ca3e3c7a3 in seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > seastar::future<void>::then_impl<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > >(seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1619:16

    #4 0x557ca3e3aaed in seastar::internal::future_result<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()> >(seastar::future<void>&, seastar::noncopyable_function<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > ()>&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1248:20

    #5 0x557ca3e3a2fa in seastar::lowres_clock seastar::future<void>::then<seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > seastar::get_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, seastar::lowres_clock>&, unsigned long, seastar::basic_semaphore<seastar::named_semaphore_exception_factory, seastar::lowres_clock>::time_point)::'lambda'(), seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > >(seastar::named_semaphore_exception_factory&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1544:16

    #6 0x557ca3c8e150 in seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock> > seastar::get_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, seastar::lowres_clock>&, unsigned long, seastar::basic_semaphore<seastar::named_semaphore_exception_factory, seastar::lowres_clock>::time_point) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/semaphore.hh:558:37

    #7 0x557ca3acf373 in seastar::smp_message_queue::submit_item(unsigned int, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item> >) /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3202:9

    #8 0x557ca4715cde in seastar::futurize<std::__1::invoke_result<seastar::net::conntrack::handle::~handle()::'lambda'()>::type>::type seastar::smp_message_queue::submit<seastar::net::conntrack::handle::~handle()::'lambda'()>(unsigned int, seastar::smp_submit_to_options, seastar::net::conntrack::handle::~handle()::'lambda'()&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:269:9

    #9 0x557ca471571e in seastar::futurize<std::__1::invoke_result<seastar::net::conntrack::handle::~handle()::'lambda'()>::type>::type seastar::smp::submit_to<seastar::net::conntrack::handle::~handle()::'lambda'()>(unsigned int, seastar::smp_submit_to_options, seastar::net::conntrack::handle::~handle()::'lambda'()&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:357:44

    #10 0x557ca4714a6d in seastar::futurize<std::__1::invoke_result<seastar::net::conntrack::handle::~handle()::'lambda'()>::type>::type seastar::smp::submit_to<seastar::net::conntrack::handle::~handle()::'lambda'()>(unsigned int, seastar::net::conntrack::handle::~handle()::'lambda'()&&) /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:376:16

    #11 0x557ca4714613 in seastar::net::conntrack::handle::~handle() /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/net/posix-stack.hh:88:19

    #12 0x557ca4704ebf in seastar::net::posix_ap_server_socket_impl::connection::~connection() /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/net/posix-stack.hh:135:12

    #13 0x557ca473298f in std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>::~pair() /vectorized/llvm/bin/../include/c++/v1/__utility/pair.h:40:29

    #14 0x557ca4732930 in void std::__1::__destroy_at<std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>, 0>(std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>*) /vectorized/llvm/bin/../include/c++/v1/__memory/construct_at.h:56:13

    #15 0x557ca47328cc in void std::__1::destroy_at<std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>, 0>(std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>*) /vectorized/llvm/bin/../include/c++/v1/__memory/construct_at.h:81:5

    #16 0x557ca4732630 in void std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, void*> > >::destroy<std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>, void, void>(std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, void*> >&, std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection>*) /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:317:9

    #17 0x557ca473233b in std::__1::__hash_table<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, std::__1::__unordered_map_hasher<std::__1::tuple<int, seastar::socket_address>, std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, std::__1::hash<std::__1::tuple<int, seastar::socket_address> >, std::__1::equal_to<std::__1::tuple<int, seastar::socket_address> >, true>, std::__1::__unordered_map_equal<std::__1::tuple<int, seastar::socket_address>, std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, std::__1::equal_to<std::__1::tuple<int, seastar::socket_address> >, std::__1::hash<std::__1::tuple<int, seastar::socket_address> >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection> > >::__deallocate_node(std::__1::__hash_node_base<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, void*>*>*) /vectorized/llvm/bin/../include/c++/v1/__hash_table:1572:9

    #18 0x557ca4730831 in std::__1::__hash_table<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, std::__1::__unordered_map_hasher<std::__1::tuple<int, seastar::socket_address>, std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, std::__1::hash<std::__1::tuple<int, seastar::socket_address> >, std::__1::equal_to<std::__1::tuple<int, seastar::socket_address> >, true>, std::__1::__unordered_map_equal<std::__1::tuple<int, seastar::socket_address>, std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection>, std::__1::equal_to<std::__1::tuple<int, seastar::socket_address> >, std::__1::hash<std::__1::tuple<int, seastar::socket_address> >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection> > >::~__hash_table() /vectorized/llvm/bin/../include/c++/v1/__hash_table:1511:5

    #19 0x557ca47034d8 in std::__1::unordered_multimap<std::__1::tuple<int, seastar::socket_address>, seastar::net::posix_ap_server_socket_impl::connection, std::__1::hash<std::__1::tuple<int, seastar::socket_address> >, std::__1::equal_to<std::__1::tuple<int, seastar::socket_address> >, std::__1::allocator<std::__1::pair<std::__1::tuple<int, seastar::socket_address> const, seastar::net::posix_ap_server_socket_impl::connection> > >::~unordered_multimap() /vectorized/llvm/bin/../include/c++/v1/unordered_map:2026:5

    #20 0x7f98dcaf238e in __call_tls_dtors (/lib64/libc.so.6+0x4038e) (BuildId: 6e3c087aca9b39549e4ba92c451f1e399b586e28)

[dotnwat]

here is a reproducer. basically if a socket is to be handled on some non-main thread then conn_q has a connected added to it from the main thread. if the target shard doesn't call accept or abort_accept for some reason then entry is never removed from conn_q.

then when reactor threads are exiting conn_q thread local is destroyed and a background future is scheduled late in the shutdown sequence. appears that this happens before the thread local reactor destructor. seems it either isn't cleaning up at this level of detail or lsan is having trouble tracking something.

+ss::logger lg("ok");
+
+struct server {
+    ss::future<> start() {
+        ss::listen_options lo;
+        lo.reuse_address = true;
+        lo.set_fixed_cpu(ss::smp::count - 1);
+        s = ss::engine().listen(
+          ss::socket_address(ss::net::inet_address("127.0.0.1"), 9092), lo);
+        if (ss::this_shard_id() == 0) {
+            (void)ss::with_gate(g, [this] {
+                return s.accept().then_wrapped([](auto far) {
+                    try {
+                        auto ar = far.get();
+                        lg.info("accepted");
+                    } catch (...) {
+                        lg.info(
+                          "accepted (error): {}", std::current_exception());
+                    }
+                });
+            });
+        }
+        return ss::now();
+    }
+
+    ss::future<> stop() {
+        if (ss::this_shard_id() == 0) {
+            s.abort_accept();
+        }
+        return g.close();
+    }
+
+    ss::server_socket s;
+    ss::gate g;
+};
+
+SEASTAR_THREAD_TEST_CASE(memleak) {
+    ss::sharded<server> service;
+    service.start().get();
+    service.invoke_on_all([](server& s) { return s.start(); }).get();
+    auto c = ss::make_socket();
+    c.connect(ss::socket_address(ss::net::inet_address("127.0.0.1"), 9092))
+      .then_wrapped([](auto f) {
+          try {
+              f.get();
+          } catch (...) {
+          }
+      })
+      .get();
+    c.shutdown();
+    service.stop().get();
+}

[dotnwat]

it also appears this race could occur even if a core calls abort_accept on its socket. say this happens in some ss::sharded<>::stop() sequence. on some non-main core if abort_accept is called, then the main core could still race with an accepted connection and add an entry in the other cores conn_q thread_local container. this would lead to the same leak.

[bhalevy]

@gleb-cloudius please look into this issue

Cc @xemul

[bhalevy]

@dotnwat if you're able to reproduce the issue, it would be ideal if you could send a fix for it that you've validated.

[dotnwat]

@bhalevy i had several solutions that seemed to work, but they were all rather ugly. it wasn't until after i hacked these together that i had a full understanding of the race condition that is happening.

i'll post a message to seastar-dev for further discussion.

[niekbouman]

Any progress on this issue? I ran into this issue today.

[dotnwat]

@niekbouman there is a discussion going on here https://github.com/scylladb/seastar/pull/1265