search

ClickHouse / ClickHouse

ClickHouse® is a real-time analytics DBMS
https://clickhouse.com
Apache License 2.0
36.6k stars 6.77k forks source link

Integration test `test_zookeeper_fallback_session` is flaky #66966

Closed tavplubix closed 1 month ago

tavplubix commented 1 month ago

https://s3.amazonaws.com/clickhouse-test-reports/0/f1296e5113653cf548f441e85f2e4cea5f7606fc/integration_tests__tsan__[1_6].html

tavplubix commented 1 month ago 
E           Exception: Sanitizer assert found for instance ==================
E           WARNING: ThreadSanitizer: data race (pid=1)
E             Write of size 1 at 0x726c0004fb88 by thread T664:
E               #0 Coordination::ZooKeeper::finalize(bool, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1018:20 (clickhouse+0x1a84d59c) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #1 Coordination::ZooKeeper::receiveThread() build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:822:9 (clickhouse+0x1a85829a) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #2 Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1::operator()() const build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:402:56 (clickhouse+0x1a85fef2) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #3 decltype(std::declval<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&>()()) std::__1::__invoke[abi:v15007]<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1a85fef2)
E               #4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15007]<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::__1::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::__1::tuple<>&, std::__1::__tuple_indices<...>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0x1a85fef2)
E               #5 decltype(auto) std::__1::apply[abi:v15007]<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::__1::tuple<>&>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0x1a85fef2)
E               #6 ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:251:13 (clickhouse+0x1a85fef2)
E               #7 decltype(std::declval<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>()()) std::__1::__invoke[abi:v15007]<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()&>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1a85fef2)
E               #8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0x1a85fef2)
E               #9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'(), void ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0x1a85fef2)
E               #10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::__1::vector<zkutil::ShuffleHost, std::__1::allocator<zkutil::ShuffleHost>> const&, zkutil::ZooKeeperArgs const&, std::__1::shared_ptr<DB::ZooKeeperLog>)::$_1&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0x1a85fef2)
E               #11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf8fc623) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0xf8fc623)
E               #13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:462:13 (clickhouse+0xf8fc623)
E               #14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:219:73 (clickhouse+0xf902cf1) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15007]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf902cf1)
E               #16 void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<...>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf902cf1)
E               #17 void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf902cf1)
E           
E             Previous read of size 1 at 0x726c0004fb88 by thread T2:
E               #0 Coordination::ZooKeeper::getConnectedNodeIdx() const build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.h:121:56 (clickhouse+0x1a86275d) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #1 zkutil::ZooKeeper::getConnectedHostAvailabilityZone() const build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:1547:22 (clickhouse+0x1a7eba5b) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #2 DB::StorageSystemZooKeeperConnection::fillData(std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, std::__1::shared_ptr<DB::Context const>, DB::ActionsDAG::Node const*, std::__1::vector<char8_t, std::__1::allocator<char8_t>>) const::$_1::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<zkutil::ZooKeeper>, std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&) const build_docker/./src/Storages/System/StorageSystemZooKeeperConnection.cpp:89:44 (clickhouse+0x15fbcb43) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #3 DB::StorageSystemZooKeeperConnection::fillData(std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, std::__1::shared_ptr<DB::Context const>, DB::ActionsDAG::Node const*, std::__1::vector<char8_t, std::__1::allocator<char8_t>>) const build_docker/./src/Storages/System/StorageSystemZooKeeperConnection.cpp:94:5 (clickhouse+0x15fbbf8e) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #4 DB::ReadFromSystemOneBlock::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Storages/System/IStorageSystemOneBlock.cpp:84:14 (clickhouse+0x15d4fd2e) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #5 DB::ISourceStep::updatePipeline(std::__1::vector<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder>>, std::__1::allocator<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:20:5 (clickhouse+0x1a327901) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #6 DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188:47 (clickhouse+0x1a34b1b8) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #7 DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252:34 (clickhouse+0x17a4bbc8) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #8 DB::InterpreterSelectQueryAnalyzer::execute() build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221:29 (clickhouse+0x17a4b86f) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #9 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1224:40 (clickhouse+0x17eb3086) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #10 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1390:26 (clickhouse+0x17eae757) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #11 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:521:54 (clickhouse+0x19c44798) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #12 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2366:9 (clickhouse+0x19c621a7) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #13 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1e08aaa2) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #14 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1e08b311) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #15 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205:14 (clickhouse+0x1e004682) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #16 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1e00294f) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)
E               #17 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1e000e29) (BuildId: 8bd4c7460fcb06dfb34140c2e542f7bb2ea4e8cf)