search

ClickHouse / ClickHouse

ClickHouse® is a real-time analytics DBMS
https://clickhouse.com
Apache License 2.0
37.4k stars 6.88k forks source link

Race condition in test `01171_mv_select_insert_isolation_long.sh` and system.parts #69519

Open alesapin opened 1 month ago

alesapin commented 1 month ago

Tsan report:

Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
==================
WARNING: ThreadSanitizer: data race (pid=611)
  Write of size 8 at 0x72200004e9c8 by thread T654:
    #0 __tsan_memcpy <null> (clickhouse+0x76a1feb) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__move_assign[abi:v15007](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::integral_constant<bool, true>) build_docker/./contrib/llvm-project/libcxx/include/string:2547:16 (clickhouse+0x19dda2e0) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #2 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::operator=[abi:v15007](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&) build_docker/./contrib/llvm-project/libcxx/include/string:2562:5 (clickhouse+0x19dda2e0)
    #3 DB::DataPartStorageOnDiskBase::remove(std::__1::function<DB::CanRemoveDescription ()>&&, DB::MergeTreeDataPartChecksums const&, std::__1::list<DB::IDataPartStorage::ProjectionChecksums, std::__1::allocator<DB::IDataPartStorage::ProjectionChecksums>>, bool, std::__1::shared_ptr<Poco::Logger>) build_docker/./src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:776:22 (clickhouse+0x19dda2e0)
    #4 DB::IMergeTreeDataPart::remove() build_docker/./src/Storages/MergeTree/IMergeTreeDataPart.cpp:2057:26 (clickhouse+0x19e07c3a) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #5 DB::MergeTreeData::clearPartsFromFilesystemImpl(std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeDataPart const>>> const&, std::__1::unordered_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*)::$_3::operator()() const build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2685:42 (clickhouse+0x19f2dc62) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #6 DB::MergeTreeData::clearPartsFromFilesystemImpl(std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeDataPart const>>> const&, std::__1::unordered_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*) build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2693:9 (clickhouse+0x19f2c335) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #7 DB::MergeTreeData::clearPartsFromFilesystem(std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeDataPart const>>> const&, bool, std::__1::unordered_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*) build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2658:9 (clickhouse+0x19f29d06) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #8 DB::MergeTreeData::clearOldPartsFromFilesystem(bool) build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2627:5 (clickhouse+0x19f29adc) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #9 DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1::operator()() const build_docker/./src/Storages/StorageMergeTree.cpp:1465:34 (clickhouse+0x1a3e1b36) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #10 decltype(std::declval<DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1&>()()) std::__1::__invoke[abi:v15007]<DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1&>(DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1a3e1b36)
    #11 bool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1&>(DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 (clickhouse+0x1a3e1b36)
    #12 std::__1::__function::__default_alloc_func<DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1, bool ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0x1a3e1b36)
    #13 bool std::__1::__function::__policy_invoker<bool ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&)::$_1, bool ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0x1a3e1b36)
    #14 std::__1::__function::__policy_func<bool ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19897d6d) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #15 std::__1::function<bool ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0x19897d6d)
    #16 DB::ExecutableLambdaAdapter::executeStep() build_docker/./src/Storages/MergeTree/IExecutableTask.h:70:15 (clickhouse+0x19897d6d)
    #17 DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:286:42 (clickhouse+0x19ee56eb) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #18 DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::threadFunction() build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:352:13 (clickhouse+0x19ee61e1) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #19 DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'()::operator()() const build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57:47 (clickhouse+0x19eeb8c1) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #20 decltype(std::declval<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'()&>()()) std::__1::__invoke[abi:v15007]<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'()&>(DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x19eeb8c1)
    #21 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'()&>(DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0x19eeb8c1)
    #22 std::__1::__function::__default_alloc_func<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'(), void ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0x19eeb8c1)
    #23 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0x19eeb8c1)
    #24 std::__1::__function::__policy_func<void ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xfce6edf) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #25 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0xfce6edf)
    #26 ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::worker(std::__1::__list_iterator<ThreadFromGlobalPoolImpl<false, true>, void*>) build_docker/./src/Common/ThreadPool.cpp:530:17 (clickhouse+0xfce6edf)
    #27 void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:252:73 (clickhouse+0xfcec842) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #28 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15007]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xfcec842)
    #29 decltype(auto) std::__1::__apply_tuple_impl[abi:v15007]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<...>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xfcec842)
    #30 decltype(auto) std::__1::apply[abi:v15007]<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xfcec842)
    #31 ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:251:13 (clickhouse+0xfcec842)
    #32 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15007]<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xfcec781) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #33 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xfcec781)
    #34 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xfcec781)
    #35 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xfcec781)
    #36 std::__1::__function::__policy_func<void ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xfce2825) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #37 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0xfce2825)
    #38 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:540:17 (clickhouse+0xfce2825)
    #39 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:252:73 (clickhouse+0xfce96d1) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #40 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15007]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xfce96d1)
    #41 void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<...>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xfce96d1)
    #42 void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xfce96d1)

  Previous read of size 1 at 0x72200004e9ce by thread T5711:
    #0 std::__1::enable_if<__is_cpp17_forward_iterator<char const*>::value, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&>::type std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::append[abi:v15007]<char const*>(char const*, char const*) FunctionFile.cpp (clickhouse+0x7756549) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #1 std::__1::enable_if<__is_cpp17_forward_iterator<char const*>::value, void>::type std::__1::__fs::filesystem::_PathCVT<char>::__append_range[abi:v15007]<char const*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, char const*, char const*) build_docker/./contrib/llvm-project/libcxx/include/__filesystem/path.h:316:12 (clickhouse+0x19dc5afe) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #2 void std::__1::__fs::filesystem::_PathCVT<char>::__append_source[abi:v15007]<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__filesystem/path.h:331:5 (clickhouse+0x19dc5afe)
    #3 std::__1::__fs::filesystem::path::path[abi:v15007]<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::__fs::filesystem::path::format) build_docker/./contrib/llvm-project/libcxx/include/__filesystem/path.h:483:5 (clickhouse+0x19dc5afe)
    #4 DB::DataPartStorageOnDiskBase::getFullPath() const build_docker/./src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:54:65 (clickhouse+0x19dc5afe)
    #5 DB::StorageSystemParts::processNextStorage(std::__1::shared_ptr<DB::Context const>, std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, std::__1::vector<char8_t, std::__1::allocator<char8_t>>&, DB::StoragesInfo const&, bool) build_docker/./src/Storages/System/StorageSystemParts.cpp:246:73 (clickhouse+0x162c2a45) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #6 DB::ReadFromSystemPartsBase::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Storages/System/StorageSystemPartsBase.cpp:328:18 (clickhouse+0x162ccbe4) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #7 DB::ISourceStep::updatePipeline(std::__1::vector<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder>>, std::__1::allocator<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:20:5 (clickhouse+0x1afd5882) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #8 DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188:47 (clickhouse+0x1affe8d8) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #9 DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252:34 (clickhouse+0x1815d5c8) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #10 DB::InterpreterSelectQueryAnalyzer::execute() build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:221:29 (clickhouse+0x1815d26f) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #11 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1241:40 (clickhouse+0x185d8b66) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #12 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1407:26 (clickhouse+0x185d4257) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #13 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:564:54 (clickhouse+0x1a8b868b) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #14 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2494:9 (clickhouse+0x1a8d7247) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #15 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1ed63d02) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #16 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:38 (clickhouse+0x1ed64571) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #17 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205:14 (clickhouse+0x1ecdd1e2) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #18 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1ecdb4af) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #19 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1ecd9989) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)

  Location is heap block of size 120 at 0x72200004e980 allocated by thread T5696:
    #0 operator new(unsigned long) <null> (clickhouse+0x77228a7) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #1 void* std::__1::__libcpp_operator_new[abi:v15007]<unsigned long>(unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:246:10 (clickhouse+0x19ff3f81) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #2 std::__1::__libcpp_allocate[abi:v15007](unsigned long, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:272:10 (clickhouse+0x19ff3f81)
    #3 std::__1::allocator<std::__1::__shared_ptr_emplace<DB::DataPartStorageOnDiskFull, std::__1::allocator<DB::DataPartStorageOnDiskFull>>>::allocate[abi:v15007](unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator.h:112:38 (clickhouse+0x19ff3f81)
    #4 std::__1::allocator_traits<std::__1::allocator<std::__1::__shared_ptr_emplace<DB::DataPartStorageOnDiskFull, std::__1::allocator<DB::DataPartStorageOnDiskFull>>>>::allocate[abi:v15007](std::__1::allocator<std::__1::__shared_ptr_emplace<DB::DataPartStorageOnDiskFull, std::__1::allocator<DB::DataPartStorageOnDiskFull>>>&, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:262:20 (clickhouse+0x19ff3f81)
    #5 std::__1::__allocation_guard<std::__1::allocator<std::__1::__shared_ptr_emplace<DB::DataPartStorageOnDiskFull, std::__1::allocator<DB::DataPartStorageOnDiskFull>>>>::__allocation_guard[abi:v15007]<std::__1::allocator<DB::DataPartStorageOnDiskFull>>(std::__1::allocator<DB::DataPartStorageOnDiskFull>, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocation_guard.h:53:18 (clickhouse+0x19ff3f81)
    #6 std::__1::shared_ptr<DB::DataPartStorageOnDiskFull> std::__1::allocate_shared[abi:v15007]<DB::DataPartStorageOnDiskFull, std::__1::allocator<DB::DataPartStorageOnDiskFull>, std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, void>(std::__1::allocator<DB::DataPartStorageOnDiskFull> const&, std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:952:48 (clickhouse+0x19ff3f81)
    #7 std::__1::shared_ptr<DB::DataPartStorageOnDiskFull> std::__1::make_shared[abi:v15007]<DB::DataPartStorageOnDiskFull, std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, void>(std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x19ff3f81)
    #8 DB::MergeTreeDataPartBuilder::getPartStorageByType(DB::MergeTreeDataPartStorageType, std::__1::shared_ptr<DB::IVolume> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool) build_docker/./src/Storages/MergeTree/MergeTreeDataPartBuilder.cpp:89:20 (clickhouse+0x19ff3f81)
    #9 DB::MergeTreeDataPartBuilder::withPartStorageType(DB::MergeTreeDataPartStorageType) build_docker/./src/Storages/MergeTree/MergeTreeDataPartBuilder.cpp:121:20 (clickhouse+0x19ff470a) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #10 DB::MergeTreeDataPartBuilder::withPartFormat(DB::MergeTreeDataPartFormat) build_docker/./src/Storages/MergeTree/MergeTreeDataPartBuilder.cpp:128:35 (clickhouse+0x19ff470a)
    #11 DB::MergeTreeDataWriter::writeTempPartImpl(DB::BlockWithPartition&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<DB::Context const>, long, bool) build_docker/./src/Storages/MergeTree/MergeTreeDataWriter.cpp:548:10 (clickhouse+0x1a07a90b) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #12 DB::MergeTreeDataWriter::writeTempPart(DB::BlockWithPartition&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<DB::Context const>, bool) build_docker/./src/Storages/MergeTree/MergeTreeDataWriter.cpp:426:12 (clickhouse+0x1a078b56) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #13 DB::MergeTreeSink::consume(DB::Chunk&) build_docker/./src/Storages/MergeTree/MergeTreeSink.cpp:118:40 (clickhouse+0x1a3f5f22) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #14 DB::SinkToStorage::onConsume(DB::Chunk) build_docker/./src/Processors/Sinks/SinkToStorage.cpp:18:5 (clickhouse+0x1aec04f2) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #15 DB::ExceptionKeepingTransform::work()::$_1::operator()() const build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:150:51 (clickhouse+0x1ad87dc5) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #16 decltype(std::declval<DB::ExceptionKeepingTransform::work()::$_1&>()()) std::__1::__invoke[abi:v15007]<DB::ExceptionKeepingTransform::work()::$_1&>(DB::ExceptionKeepingTransform::work()::$_1&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1ad87dc5)
    #17 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ExceptionKeepingTransform::work()::$_1&>(DB::ExceptionKeepingTransform::work()::$_1&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0x1ad87dc5)
    #18 std::__1::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0x1ad87dc5)
    #19 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0x1ad87dc5)
    #20 std::__1::__function::__policy_func<void ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x1ad87b37) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #21 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0x1ad87b37)
    #22 DB::runStep(std::__1::function<void ()>, DB::ThreadStatus*, std::__1::atomic<unsigned long>*) build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:114:9 (clickhouse+0x1ad87b37)
    #23 DB::ExceptionKeepingTransform::work() build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:150:34 (clickhouse+0x1ad87434) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #24 DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:47:26 (clickhouse+0x1a9d0b69) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #25 DB::ExecutionThreadContext::executeTask() build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:96:9 (clickhouse+0x1a9d0b69)
    #26 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:283:26 (clickhouse+0x1a9c1394) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #27 DB::PipelineExecutor::executeStep(std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:157:5 (clickhouse+0x1a9c0d05) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #28 DB::PushingPipelineExecutor::finish() build_docker/./src/Processors/Executors/PushingPipelineExecutor.cpp:130:19 (clickhouse+0x1a9db72b) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #29 _ZZN2DB10TCPHandler18processInsertQueryEvENK3$_0clINS_23PushingPipelineExecutorEEEDaRT_NS_5BlockE build_docker/./src/Server/TCPHandler.cpp:988:22 (clickhouse+0x1a8c4e7d) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #30 DB::TCPHandler::processInsertQuery() build_docker/./src/Server/TCPHandler.cpp:1056:9 (clickhouse+0x1a8c4e7d)
    #31 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:581:17 (clickhouse+0x1a8b8815) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #32 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2494:9 (clickhouse+0x1a8d7247) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #33 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1ed63d02) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #34 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:38 (clickhouse+0x1ed64571) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #35 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205:14 (clickhouse+0x1ecdd1e2) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #36 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1ecdb4af) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #37 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1ecd9989) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)

  Thread T654 'Common' (tid=1276, running) created by main thread at:
    #0 pthread_create <null> (clickhouse+0x76a5fcb) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #1 std::__1::__libcpp_thread_create[abi:v15007](unsigned long*, void* (*)(void*), void*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:376:10 (clickhouse+0xfce891e) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #2 std::__1::thread::thread<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) build_docker/./contrib/llvm-project/libcxx/include/thread:311:16 (clickhouse+0xfce891e)
    #3 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool) build_docker/./src/Common/ThreadPool.cpp:252:35 (clickhouse+0xfce13b4) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #4 ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, Priority, unsigned long, bool) build_docker/./src/Common/ThreadPool.cpp:337:5 (clickhouse+0xfcec2a7) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #5 ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./src/Common/ThreadPool.h:218:38 (clickhouse+0xfcec2a7)
    #6 void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool) build_docker/./src/Common/ThreadPool.cpp:252:35 (clickhouse+0xfce4427) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #7 ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleOrThrowOnError(std::__1::function<void ()>, Priority) build_docker/./src/Common/ThreadPool.cpp:325:5 (clickhouse+0xfce3eea) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #8 DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>::MergeTreeBackgroundExecutor(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, unsigned long, unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, std::__1::basic_string_view<char, std::__1::char_traits<char>>) build_docker/./src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:57:15 (clickhouse+0x19ee2da9) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #9 DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>* std::__1::construct_at[abi:v15007]<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>*>(DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>*, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35:48 (clickhouse+0x179836de) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #10 void std::__1::allocator_traits<std::__1::allocator<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>>>::construct[abi:v15007]<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, void, void>(std::__1::allocator<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>>&, DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>*, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x17950e91) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #11 std::__1::__shared_ptr_emplace<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>, std::__1::allocator<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>>>::__shared_ptr_emplace[abi:v15007]<char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&>(std::__1::allocator<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>>, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292:9 (clickhouse+0x17950e91)
    #12 std::__1::shared_ptr<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>> std::__1::allocate_shared[abi:v15007]<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>, std::__1::allocator<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>>, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, void>(std::__1::allocator<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>> const&, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953:55 (clickhouse+0x17950e91)
    #13 std::__1::shared_ptr<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>> std::__1::make_shared[abi:v15007]<DB::MergeTreeBackgroundExecutor<DB::RoundRobinRuntimeQueue>, char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, void>(char const (&) [7], unsigned long&, unsigned long&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&, StrongTypedef<unsigned long, CurrentMetrics::MetricTag> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962:12 (clickhouse+0x17950e91)
    #14 DB::Context::initializeBackgroundExecutorsIfNeeded() build_docker/./src/Interpreters/Context.cpp:5485:31 (clickhouse+0x17950e91)
    #15 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:2092:21 (clickhouse+0xfe7c2f1) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #16 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1ed9c7be) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #17 DB::Server::run() build_docker/./programs/server/Server.cpp:407:25 (clickhouse+0xfe65925) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #18 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1edbafe0) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #19 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:213:20 (clickhouse+0xfe62755) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #20 main build_docker/./programs/main.cpp:256:21 (clickhouse+0x7724918) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)

  Thread T5711 'TCPHandler' (tid=135758, running) created by thread T723 at:
    #0 pthread_create <null> (clickhouse+0x76a5fcb) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:198:6 (clickhouse+0x1ecd933f) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #2 Poco::Thread::start(Poco::Runnable&) build_docker/./base/poco/Foundation/src/Thread.cpp:118:2 (clickhouse+0x1ecdaeaf) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #3 Poco::PooledThread::start() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:91:10 (clickhouse+0x1ecdf306) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #4 Poco::ThreadPool::getThread() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:471:14 (clickhouse+0x1ecdf306)
    #5 Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./base/poco/Foundation/src/ThreadPool.cpp:375:2 (clickhouse+0x1ecdf707) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #6 Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:168:29 (clickhouse+0x1ed64dd7) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #7 Poco::Net::TCPServer::run() build_docker/./base/poco/Net/src/TCPServer.cpp:148:39 (clickhouse+0x1ed634aa) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #8 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1ecdb4af) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #9 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1ecd9989) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)

  Thread T5696 'TCPHandler' (tid=135093, running) created by thread T723 at:
    #0 pthread_create <null> (clickhouse+0x76a5fcb) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:198:6 (clickhouse+0x1ecd933f) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #2 Poco::Thread::start(Poco::Runnable&) build_docker/./base/poco/Foundation/src/Thread.cpp:118:2 (clickhouse+0x1ecdaeaf) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #3 Poco::PooledThread::start() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:91:10 (clickhouse+0x1ecdf306) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #4 Poco::ThreadPool::getThread() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:471:14 (clickhouse+0x1ecdf306)
    #5 Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./base/poco/Foundation/src/ThreadPool.cpp:375:2 (clickhouse+0x1ecdf707) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #6 Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:168:29 (clickhouse+0x1ed64dd7) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #7 Poco::Net::TCPServer::run() build_docker/./base/poco/Net/src/TCPServer.cpp:148:39 (clickhouse+0x1ed634aa) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #8 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1ecdb4af) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)
    #9 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1ecd9989) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a)

SUMMARY: ThreadSanitizer: data race (/usr/bin/clickhouse+0x76a1feb) (BuildId: a1dea124d1ab5c5740733a9984a3158bd462180a) in __tsan_memcpy
==================

Thread which deleted part:

2024.09.10 21:16:11.924525 [ 1276 ] {} <Trace> test_jhx7iz5b.dst (0faa9e5b-46be-4ae0-90f9-ae3e70d901ee): Found 1 old parts to remove. Parts: [all_3_3_0]
2024.09.10 21:16:11.924660 [ 1276 ] {} <Debug> test_jhx7iz5b.dst (0faa9e5b-46be-4ae0-90f9-ae3e70d901ee): Removing 1 parts from filesystem (serially): Parts: [all_3_3_0]

Unforutnately cannot find thread which read this system.parts. But it was inserted properly:

grep 'c88965c0-55fb-4d55-84a1-21eb3e934269' clickhouse-server.log                                                                                                                                                                                                                                                              [16:47:42]
2024.09.10 21:16:09.846332 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Debug> executeQuery: (from [::1]:35536) (TID: (80, 35, efddd055-cefa-4cf0-97b8-2cf6c10e3a86), TIDH: 3120300155748237000) (comment: 01171_mv_select_insert_isolation_long.sh) INSERT INTO src VALUES /* (42, 3) */  (stage: Complete)
2024.09.10 21:16:09.860914 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> Planner: Query to stage Complete only analyze
2024.09.10 21:16:09.861597 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> Planner: Query from stage FetchColumns to stage Complete only analyze
2024.09.10 21:16:10.066640 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> test_jhx7iz5b.src (8894e53f-058f-4158-9d9d-dbbb0f0045f6): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2024.09.10 21:16:10.157387 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> MergedBlockOutputStream: filled checksums 1_3_3_0 (state Temporary)
2024.09.10 21:16:10.586896 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> test_jhx7iz5b.src (8894e53f-058f-4158-9d9d-dbbb0f0045f6): Renaming temporary part tmp_insert_1_3_3_0 to 1_3_3_0 with tid (80, 35, efddd055-cefa-4cf0-97b8-2cf6c10e3a86).
2024.09.10 21:16:10.603065 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> Planner: Query to stage Complete
2024.09.10 21:16:10.603871 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> Planner: Query from stage FetchColumns to stage Complete
2024.09.10 21:16:10.625301 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> test_jhx7iz5b.dst (0faa9e5b-46be-4ae0-90f9-ae3e70d901ee): Trying to reserve 1.00 MiB using storage policy from min volume index 0
2024.09.10 21:16:10.757150 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> MergedBlockOutputStream: filled checksums all_3_3_0 (state Temporary)
2024.09.10 21:16:11.177913 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> test_jhx7iz5b.dst (0faa9e5b-46be-4ae0-90f9-ae3e70d901ee): Renaming temporary part tmp_insert_all_3_3_0 to all_3_3_0 with tid (80, 35, efddd055-cefa-4cf0-97b8-2cf6c10e3a86).
2024.09.10 21:16:11.208939 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Trace> PushingToViews: Pushing (sequentially) from test_jhx7iz5b.src (8894e53f-058f-4158-9d9d-dbbb0f0045f6) to test_jhx7iz5b.mv (d4f374fe-af79-47d3-b094-642ca209d2c2) took 605 ms.
2024.09.10 21:16:11.222968 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Debug> executeQuery: Read 2 rows, 16.00 B in 1.373795 sec., 1.4558212833792523 rows/sec., 11.65 B/sec.
2024.09.10 21:16:11.227095 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Debug> MemoryTracker: Peak memory usage (for query): 3.16 MiB.
2024.09.10 21:16:11.227247 [ 135093 ] {c88965c0-55fb-4d55-84a1-21eb3e934269} <Debug> TCPHandler: Processed in 1.386293264 sec.

I didn't understand how is it possible, we have proper check in system.parts: https://github.com/ClickHouse/clickhouse/blob/master/src/Storages/System/StorageSystemParts.cpp#L239-L241 and when we grap parts to remove we check shared pointer uniqueness: https://github.com/ClickHouse/clickhouse/blob/master/src/Storages/MergeTree/MergeTreeData.cpp#L2472-L2476. And all this happen with data parts lock (exclusively).

vitlibar commented 1 week ago

It seems other race conditions with system.parts also happen (see https://s3.amazonaws.com/clickhouse-test-reports/70053/57db54239f1e668b9bdb2050ae5ea34df4a94fc0/stateless_tests__tsan__s3_storage__[3_3].html):

WARNING: ThreadSanitizer: data race (pid=610)
  Write of size 8 at 0x7220004f17d0 by thread T1475:
    #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::operator=(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) <null> (clickhouse+0x7751a8b) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #1 DB::DataPartStorageOnDiskBase::rename(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<Poco::Logger>, bool, bool) build_docker/./src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:682:14 (clickhouse+0x19d01c14) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #2 DB::IMergeTreeDataPart::renameTo(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool) build_docker/./src/Storages/MergeTree/IMergeTreeDataPart.cpp:1915:26 (clickhouse+0x19d41772) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #3 DB::MergeTreeData::Transaction::renameParts() build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:6856:27 (clickhouse+0x19ea7a88) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #4 DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda1'()::operator()() const build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:941:21 (clickhouse+0x1a29d0e3) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #5 DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda2'()::operator()() const build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1088:43 (clickhouse+0x1a29c9b8) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #6 DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()::operator()() const build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1121:13 (clickhouse+0x1a29c7d0) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #7 void DB::ZooKeeperRetriesControl::retryLoop<DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()&, void DB::ZooKeeperRetriesControl::retryLoop<DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()&&)::'lambda'()>(DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()&&, void DB::ZooKeeperRetriesControl::retryLoop<DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()&&)::'lambda'()&&) build_docker/./src/Common/ZooKeeper/ZooKeeperRetries.h:74:17 (clickhouse+0x1a29c7d0)
    #8 void DB::ZooKeeperRetriesControl::retryLoop<DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long)::'lambda3'()&&) build_docker/./src/Common/ZooKeeper/ZooKeeperRetries.h:48:9 (clickhouse+0x1a287454) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #9 DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::__1::shared_ptr<DB::IMergeTreeDataPart>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long) build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1113:17 (clickhouse+0x1a287454)
    #10 DB::ReplicatedMergeTreeSinkImpl<false>::finishDelayedChunk(std::__1::shared_ptr<DB::ZooKeeperWithFaultInjection> const&) build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:480:33 (clickhouse+0x1a275983) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #11 DB::ReplicatedMergeTreeSinkImpl<false>::onFinish() build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1199:5 (clickhouse+0x1a2867f1) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #12 DB::ExceptionKeepingTransform::work()::$_3::operator()() const build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:186:47 (clickhouse+0x1ab4b610) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #13 decltype(std::declval<DB::ExceptionKeepingTransform::work()::$_3&>()()) std::__1::__invoke[abi:v15007]<DB::ExceptionKeepingTransform::work()::$_3&>(DB::ExceptionKeepingTransform::work()::$_3&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1ab4b610)
    #14 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ExceptionKeepingTransform::work()::$_3&>(DB::ExceptionKeepingTransform::work()::$_3&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0x1ab4b610)
    #15 std::__1::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_3, void ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0x1ab4b610)
    #16 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_3, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0x1ab4b610)
    #17 std::__1::__function::__policy_func<void ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x1ab4b037) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #18 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0x1ab4b037)
    #19 DB::runStep(std::__1::function<void ()>, DB::ThreadStatus*, std::__1::atomic<unsigned long>*) build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:114:9 (clickhouse+0x1ab4b037)
    #20 DB::ExceptionKeepingTransform::work() build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:186:30 (clickhouse+0x1ab4aa40) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #21 DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:47:26 (clickhouse+0x1a7a7149) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #22 DB::ExecutionThreadContext::executeTask() build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:96:9 (clickhouse+0x1a7a7149)
    #23 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:289:26 (clickhouse+0x1a797254) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #24 DB::PipelineExecutor::executeSingleThread(unsigned long) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:255:5 (clickhouse+0x1a7965ab) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #25 DB::PipelineExecutor::executeImpl(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:441:9 (clickhouse+0x1a7965ab)
    #26 DB::PipelineExecutor::execute(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:126:9 (clickhouse+0x1a796289) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #27 DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::__1::shared_ptr<DB::ThreadGroup>, unsigned long, bool) build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:49:24 (clickhouse+0x1a795049) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #28 DB::CompletedPipelineExecutor::execute()::$_0::operator()() const build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:89:13 (clickhouse+0x1a795049)
    #29 decltype(std::declval<DB::CompletedPipelineExecutor::execute()::$_0&>()()) std::__1::__invoke[abi:v15007]<DB::CompletedPipelineExecutor::execute()::$_0&>(DB::CompletedPipelineExecutor::execute()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1a795049)
    #30 decltype(auto) std::__1::__apply_tuple_impl[abi:v15007]<DB::CompletedPipelineExecutor::execute()::$_0&, std::__1::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<...>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0x1a795049)
    #31 decltype(auto) std::__1::apply[abi:v15007]<DB::CompletedPipelineExecutor::execute()::$_0&, std::__1::tuple<>&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0x1a795049)
    #32 ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:311:13 (clickhouse+0x1a795049)
    #33 decltype(std::declval<DB::CompletedPipelineExecutor::execute()::$_0>()()) std::__1::__invoke[abi:v15007]<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()&>(DB::CompletedPipelineExecutor::execute()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0x1a795049)
    #34 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0x1a795049)
    #35 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15007]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0x1a795049)
    #36 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::CompletedPipelineExecutor::execute()::$_0>(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0x1a795049)
    #37 std::__1::__function::__policy_func<void ()>::operator()[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf6dafdd) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #38 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1197:12 (clickhouse+0xf6dafdd)
    #39 ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::worker() build_docker/./src/Common/ThreadPool.cpp:785:17 (clickhouse+0xf6dafdd)
    #40 decltype(*std::declval<ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool*>().*std::declval<void (ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::*)()>()()) std::__1::__invoke[abi:v15007]<void (ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool*, void>(void (ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool*&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:359:23 (clickhouse+0xf6e2f7b) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #41 void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool*>&, std::__1::__tuple_indices<2ul>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf6e2f7b)
    #42 void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::__1::thread>::ThreadFromThreadPool*>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf6e2f7b)

  Previous read of size 8 at 0x7220004f17d0 by thread T831:
    #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__get_long_size[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/string:1584:34 (clickhouse+0x19cefaeb) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::size[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/string:968:31 (clickhouse+0x19cefaeb)
    #2 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::length[abi:v15007]() const build_docker/./contrib/llvm-project/libcxx/include/string:969:100 (clickhouse+0x19cefaeb)
    #3 std::__1::__fs::filesystem::__is_pathable_string<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void>::__range_end[abi:v15007](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__filesystem/path.h:111:29 (clickhouse+0x19cefaeb)
    #4 void std::__1::__fs::filesystem::_PathCVT<char>::__append_source[abi:v15007]<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__filesystem/path.h:332:20 (clickhouse+0x19cefaeb)
    #5 std::__1::__fs::filesystem::path::path[abi:v15007]<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::__fs::filesystem::path::format) build_docker/./contrib/llvm-project/libcxx/include/__filesystem/path.h:483:5 (clickhouse+0x19cefaeb)
    #6 DB::DataPartStorageOnDiskBase::getFullPath() const build_docker/./src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp:54:65 (clickhouse+0x19cefaeb)
    #7 DB::StorageSystemParts::processNextStorage(std::__1::shared_ptr<DB::Context const>, std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, std::__1::vector<char8_t, std::__1::allocator<char8_t>>&, DB::StoragesInfo const&, bool) build_docker/./src/Storages/System/StorageSystemParts.cpp:243:73 (clickhouse+0x15a9dee5) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #8 DB::ReadFromSystemPartsBase::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Storages/System/StorageSystemPartsBase.cpp:337:18 (clickhouse+0x15aa82e4) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #9 DB::ISourceStep::updatePipeline(std::__1::vector<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder>>, std::__1::allocator<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:20:5 (clickhouse+0x1adbca62) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #10 DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188:47 (clickhouse+0x1ade79db) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #11 DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:261:34 (clickhouse+0x182027d6) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #12 DB::InterpreterSelectQueryAnalyzer::execute() build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:228:29 (clickhouse+0x1820242f) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #13 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1346:40 (clickhouse+0x1865e76d) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #14 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1513:26 (clickhouse+0x18659917) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #15 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:593:54 (clickhouse+0x1a6846b6) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #16 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2526:9 (clickhouse+0x1a6a2be7) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #17 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1ec39a62) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #18 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:38 (clickhouse+0x1ec3a2d1) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #19 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205:14 (clickhouse+0x1ebb2f42) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #20 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1ebb120f) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)
    #21 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1ebaf6e9) (BuildId: e775ea996e9cd3fb7a932f266b992ad5f20c5e3d)