eBay / NuRaft

C++ implementation of Raft core logic as a replication library
Apache License 2.0
1.02k stars 241 forks source link

Fix data race in `asio_rpc_listener` #392

Closed antonio2368 closed 1 year ago

antonio2368 commented 1 year ago

TSAN in our sanitizer often caught a data race in asio_rpc_listener.

E.g. in https://s3.amazonaws.com/clickhouse-test-reports/45430/7bb65cc002d27c325393f777773de6a1e1de9fdd/integration_tests__tsan__%5B2/6%5D.html there is a log with

E           Exception: Sanitizer assert found in /ClickHouse/tests/integration/test_keeper_force_recovery_single_node/_instances_0/docker.log for instance ==================

Example of sanitizer warning:

WARNING: ThreadSanitizer: data race (pid=402)
  Write of size 8 at 0x7b4400017b08 by main thread (mutexes: write M0, write M1):
    #0 std::__1::enable_if<is_move_constructible<nuraft::raft_server*>::value && is_move_assignable<nuraft::raft_server*>::value, void>::type std::__1::swap[abi:v15000]<nuraft::raft_server*>(nuraft::raft_server*&, nuraft::raft_server*&) build_docker/../contrib/llvm-project/libcxx/include/__utility/swap.h:37:7 (clickhouse+0x22d909ef) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #1 std::__1::shared_ptr<nuraft::raft_server>::swap[abi:v15000](std::__1::shared_ptr<nuraft::raft_server>&) build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:761:9 (clickhouse+0x22d909ef)
    #2 std::__1::shared_ptr<nuraft::raft_server>::reset[abi:v15000]() build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:768:22 (clickhouse+0x22d909ef)
    #3 nuraft::asio_rpc_listener::shutdown() build_docker/../contrib/NuRaft/src/asio_service.cxx:834:18 (clickhouse+0x22d909ef)
    #4 DB::KeeperServer::shutdownRaftServer() build_docker/../src/Coordination/KeeperServer.cpp:430:28 (clickhouse+0x1fc02dc7) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #5 DB::KeeperServer::shutdown() build_docker/../src/Coordination/KeeperServer.cpp:452:5 (clickhouse+0x1fc03459) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #6 DB::KeeperDispatcher::shutdown() build_docker/../src/Coordination/KeeperDispatcher.cpp:439:21 (clickhouse+0x1fbeb5b9) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #7 DB::Context::shutdownKeeperDispatcher() const build_docker/../src/Interpreters/Context.cpp:2423:36 (clickhouse+0x1cec495c) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #8 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_10::operator()() const build_docker/../programs/server/Server.cpp:1547:5 (clickhouse+0x1460aaad) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #9 BasicScopeGuard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_10>::invoke() build_docker/../base/base/scope_guard.h:99:9 (clickhouse+0x1460aaad)
    #10 BasicScopeGuard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_10>::~BasicScopeGuard() build_docker/../base/base/scope_guard.h:48:26 (clickhouse+0x1460aaad)
    #11 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/../programs/server/Server.cpp:1915:1 (clickhouse+0x145fed62) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #12 Poco::Util::Application::run() build_docker/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse+0x22ee0f1d) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #13 DB::Server::run() build_docker/../programs/server/Server.cpp:474:25 (clickhouse+0x145eb65a) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #14 Poco::Util::ServerApplication::run(int, char**) build_docker/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse+0x22f00dcf) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #15 mainEntryClickHouseServer(int, char**) build_docker/../programs/server/Server.cpp:195:20 (clickhouse+0x145e81b4) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #16 main build_docker/../programs/main.cpp:482:12 (clickhouse+0xc2843d1) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)

  Previous read of size 8 at 0x7b4400017b08 by thread T19:
    #0 std::__1::shared_ptr<nuraft::raft_server>::shared_ptr[abi:v15000](std::__1::shared_ptr<nuraft::raft_server> const&) build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:593:22 (clickhouse+0x22d922ee) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #1 nuraft::rpc_session::rpc_session(unsigned long, nuraft::asio_service_impl*, boost::asio::io_context&, boost::asio::ssl::context&, bool, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/NuRaft/src/asio_service.cxx:232:11 (clickhouse+0x22d922ee)
    #2 nuraft::rpc_session* std::__1::construct_at[abi:v15000]<nuraft::rpc_session, unsigned long, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&, nuraft::rpc_session*>(nuraft::rpc_session*, unsigned long&&, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x22d9200b) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #3 void std::__1::allocator_traits<std::__1::allocator<nuraft::rpc_session>>::construct[abi:v15000]<nuraft::rpc_session, unsigned long, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&, void, void>(std::__1::allocator<nuraft::rpc_session>&, nuraft::rpc_session*, unsigned long&&, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x22d9200b)
    #4 std::__1::__shared_ptr_emplace<nuraft::rpc_session, std::__1::allocator<nuraft::rpc_session>>::__shared_ptr_emplace[abi:v15000]<unsigned long, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&>(std::__1::allocator<nuraft::rpc_session>, unsigned long&&, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x22d9200b)
    #5 std::__1::shared_ptr<nuraft::rpc_session> std::__1::allocate_shared[abi:v15000]<nuraft::rpc_session, std::__1::allocator<nuraft::rpc_session>, unsigned long, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&, void>(std::__1::allocator<nuraft::rpc_session> const&, unsigned long&&, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x22d9200b)
    #6 std::__1::shared_ptr<nuraft::rpc_session> std::__1::make_shared[abi:v15000]<nuraft::rpc_session, unsigned long, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&, void>(unsigned long&&, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x22d9134a) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #7 std::__1::shared_ptr<nuraft::rpc_session> nuraft::cs_new<nuraft::rpc_session, unsigned long, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&>(unsigned long&&, nuraft::asio_service_impl*&, boost::asio::io_context&, boost::asio::ssl::context&, bool&, std::__1::shared_ptr<nuraft::raft_server>&, std::__1::shared_ptr<nuraft::logger>&, std::__1::function<void (std::__1::shared_ptr<nuraft::rpc_session> const&)>&) build_docker/../contrib/NuRaft/include/libnuraft/ptr.hxx:36:12 (clickhouse+0x22d9134a)
    #8 nuraft::asio_rpc_listener::start() build_docker/../contrib/NuRaft/src/asio_service.cxx:850:13 (clickhouse+0x22d9134a)
    #9 nuraft::asio_rpc_listener::handle_accept(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&) build_docker/../contrib/NuRaft/src/asio_service.cxx:880:19 (clickhouse+0x22d91b01) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #10 decltype(*std::declval<nuraft::asio_rpc_listener*&>().*std::declval<void (nuraft::asio_rpc_listener::*&)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&)>()(std::declval<std::__1::shared_ptr<nuraft::asio_rpc_listener>&>(), std::declval<std::__1::shared_ptr<nuraft::rpc_session>&>(), std::declval<boost::system::error_code const&>())) std::__1::__invoke[abi:v15000]<void (nuraft::asio_rpc_listener::*&)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*&, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, boost::system::error_code const&, void>(void (nuraft::asio_rpc_listener::*&)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*&, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, boost::system::error_code const&) build_docker/../contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0x22d9343f) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #11 std::__1::__bind_return<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), std::__1::tuple<nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, std::__1::placeholders::__ph<1>>, std::__1::tuple<boost::system::error_code const&>, __is_valid_bind_return<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), std::__1::tuple<nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, std::__1::placeholders::__ph<1>>, std::__1::tuple<boost::system::error_code const&>>::value>::type std::__1::__apply_functor[abi:v15000]<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), std::__1::tuple<nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, std::__1::placeholders::__ph<1>>, 0ul, 1ul, 2ul, 3ul, std::__1::tuple<boost::system::error_code const&>>(void (nuraft::asio_rpc_listener::*&)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), std::__1::tuple<nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, std::__1::placeholders::__ph<1>>&, std::__1::__tuple_indices<0ul, 1ul, 2ul, 3ul>, std::__1::tuple<boost::system::error_code const&>&&) build_docker/../contrib/llvm-project/libcxx/include/__functional/bind.h:263:12 (clickhouse+0x22d9343f)
    #12 std::__1::__bind_return<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), std::__1::tuple<nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, std::__1::placeholders::__ph<1>>, std::__1::tuple<boost::system::error_code const&>, __is_valid_bind_return<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), std::__1::tuple<nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, std::__1::placeholders::__ph<1>>, std::__1::tuple<boost::system::error_code const&>>::value>::type std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>::operator()[abi:v15000]<boost::system::error_code const&>(boost::system::error_code const&) build_docker/../contrib/llvm-project/libcxx/include/__functional/bind.h:295:20 (clickhouse+0x22d9343f)
    #13 boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>::operator()() build_docker/../contrib/boost/boost/asio/detail/bind_handler.hpp:171:5 (clickhouse+0x22d9343f)
    #14 void boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>>(boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>&, ...) build_docker/../contrib/boost/boost/asio/handler_invoke_hook.hpp:88:3 (clickhouse+0x22d9343f)
    #15 void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>, std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>>(boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>&, std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>&) build_docker/../contrib/boost/boost/asio/detail/handler_invoke_helpers.hpp:54:3 (clickhouse+0x22d9343f)
    #16 void boost::asio::detail::handler_work<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::asio::any_io_executor, void>::complete<boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>>(boost::asio::detail::binder1<std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::system::error_code>&, std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>&) build_docker/../contrib/boost/boost/asio/detail/handler_work.hpp:512:7 (clickhouse+0x22d9343f)
    #17 boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::any_io_executor>, boost::asio::ip::tcp, std::__1::__bind<void (nuraft::asio_rpc_listener::*)(std::__1::shared_ptr<nuraft::asio_rpc_listener>, std::__1::shared_ptr<nuraft::rpc_session>, boost::system::error_code const&), nuraft::asio_rpc_listener*, std::__1::shared_ptr<nuraft::asio_rpc_listener>&, std::__1::shared_ptr<nuraft::rpc_session>&, std::__1::placeholders::__ph<1> const&>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/../contrib/boost/boost/asio/detail/reactive_socket_accept_op.hpp:148:9 (clickhouse+0x22d9343f)
    #18 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/../contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x22d5e2d1) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #19 boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) build_docker/../contrib/boost/boost/asio/detail/impl/epoll_reactor.ipp:817:11 (clickhouse+0x22d5e2d1)
    #20 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) build_docker/../contrib/boost/boost/asio/detail/scheduler_operation.hpp:40:5 (clickhouse+0x22d41c3b) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #21 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) build_docker/../contrib/boost/boost/asio/detail/impl/scheduler.ipp:492:12 (clickhouse+0x22d41c3b)
    #22 boost::asio::detail::scheduler::run(boost::system::error_code&) build_docker/../contrib/boost/boost/asio/detail/impl/scheduler.ipp:210:10 (clickhouse+0x22d41424) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #23 boost::asio::io_context::run() build_docker/../contrib/boost/boost/asio/impl/io_context.ipp:63:24 (clickhouse+0x22d370bb) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #24 nuraft::asio_service_impl::worker_entry() build_docker/../contrib/NuRaft/src/asio_service.cxx:1779:21 (clickhouse+0x22d370bb)
    #25 decltype(*std::declval<nuraft::asio_service_impl*&>().*std::declval<void (nuraft::asio_service_impl::*&)()>()()) std::__1::__invoke[abi:v15000]<void (nuraft::asio_service_impl::*&)(), nuraft::asio_service_impl*&, void>(void (nuraft::asio_service_impl::*&)(), nuraft::asio_service_impl*&) build_docker/../contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0x22d4bf5b) (BuildId: a5753bde1234dfdac844bc70385bf039f8f9f009)
    #26 std::__1::__bind_return<void (nuraft::asio_service_impl::*)(), std::__1::tuple<nuraft::asio_service_impl*>, std::__1::tuple<>, __is_valid_bind_return<void (nuraft::asio_service_impl::*)(), std::__1::tuple<nuraft::asio_service_impl*>, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:v15000]<void (nuraft::asio_service_impl::*)(), std::__1::tuple<nuraft::asio_service_impl*>, 0ul, std::__1::tuple<>>(void (nuraft::asio_service_impl::*&)(), std::__1::tuple<nuraft::asio_service_impl*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) build_docker/../contrib/llvm-project/libcxx/include/__functional/bind.h:263:12 (clickhouse+0x22d4bf5b)
    #27 std::__1::__bind_return<void (nuraft::asio_service_impl::*)(), std::__1::tuple<nuraft::asio_service_impl*>, std::__1::tuple<>, __is_valid_bind_return<void (nuraft::asio_service_impl::*)(), std::__1::tuple<nuraft::asio_service_impl*>, std::__1::tuple<>>::value>::type std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>::operator()[abi:v15000]<>() build_docker/../contrib/llvm-project/libcxx/include/__functional/bind.h:295:20 (clickhouse+0x22d4bf5b)
    #28 decltype(std::declval<std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>>()()) std::__1::__invoke[abi:v15000]<std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>>(std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>&&) build_docker/../contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x22d4bf5b)
    #29 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>>&, std::__1::__tuple_indices<>) build_docker/../contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0x22d4bf5b)
    #30 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, std::__1::__bind<void (nuraft::asio_service_impl::*)(), nuraft::asio_service_impl*>>>(void*) build_docker/../contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0x22d4bf5b)

where write is resetting handler_ inside shutdown() while read is creating a session using handler_ inside start(). Race also happens with acceptor_ (is_open is read inside start() while stop() is closing it).