search

Altinity / clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
https://altinity.com
Other
1.26k stars 225 forks source link

Directory for table data store/d51/d51243d3-2961-40bd-8f99-d0338ad96769/ already exists after 44 times, please check your schema dependencies #890

Closed dxygit1 closed 6 months ago

dxygit1 commented 6 months ago

When I deleted the database and restarted ClickHouse, I encountered an error during rollback, indicating that the path exists.

 curl "chi-clickhouse-cluster-dtstack-0-0:7171/backup/restore/shard0-increment-20240408044511?rm" -X POST
{"status":"acknowledged","operation":"restore","backup_name":"shard0-increment-20240408044511"}

 curl "chi-clickhouse-cluster-dtstack-0-0:7171/backup/status"
{"command":"restore --rm shard0-increment-20240408044511","status":"error","start":"2024-04-09 05:38:56","finish":"2024-04-09 05:38:57","error":"can't create table `distributed_1`.`easydo_agent_monitor`: code: 57, message: Directory for table data store/d51/d51243d3-2961-40bd-8f99-d0338ad96769/ already exists after 44 times, please check your schema dependencies"}

image image

dxygit1 commented 6 months ago

I deleted the storage directory and restarted ClickHouse, but I still encountered an error.

curl "chi-clickhouse-cluster-dtstack-1-0:7171/backup/status"
{"command":"restore --rm shard1-increment-20240408050855","status":"error","start":"2024-04-09 06:36:04","finish":"2024-04-09 06:36:10","error":"can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies"}
2024/04/09 06:40:20.991523  info CREATE TABLE local_1.proc_stat_metrics UUID '72b5f6fe-0721-4c58-aac0-e5598e194e56' (`_tenant_id_` Float64, `pid` Float64, `__instance_id__` LowCardinality(String), `_mode_` String, `mem_percent` Float64, `_at_date_` Date, `cmdline` String, `iowrite_bytes` Float64, `_receive_timestamp_` DateTime64(3, 'UTC'), `_custom_tags_` String, `page_faults_children_minor_faults` Float64, `ioread_bytes` Float64, `cpu_percent` Float64, `page_faults_children_major_faults` Float64, `open_file_descriptors` Float64, `run_time` Float64, `__appname__` LowCardinality(String), `mem_rss` Float64, `__source__` LowCardinality(String), `cpu_normalized_pct` Float64, `threads` Float64, `involuntary_ctx_switches` Float64, `__ci_resource__` LowCardinality(String), `mem_real` Float64, `__process_name__` LowCardinality(String), `__ip__` LowCardinality(String), `page_faults_major_faults` Float64, `ioread_count` Float64, `iowrite_count` Float64, `__instance_name__` LowCardinality(String), `page_faults_minor_faults` Float64, `_alias_` String, `name` String, `_at_timestamp_` DateTime64(3, 'UTC'), `voluntary_ctx_switches` Float64, `_tag_set_` Array(String), `__catalog__` LowCardinality(String), `_field_set_` Array(String), `status` String, `username` String, `mem_vms` Float64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/proc_stat_metrics/2024-04-08/{shard}/hits', '{replica}') PARTITION BY toString(_at_date_) ORDER BY (_at_date_, _at_timestamp_, intHash64(toInt64(toDateTime(_at_timestamp_)))) SAMPLE BY intHash64(toInt64(toDateTime(_at_timestamp_))) SETTINGS index_granularity = 8192 logger=clickhouse
2024/04/09 06:40:20.993446  info clickhouse connection closed logger=clickhouse
2024/04/09 06:40:20.993472 error metrics.ExecuteWithMetrics(restore) return error: can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies logger=metrics
2024/04/09 06:40:20.993508 debug api.status.stop -> status.commands[5] == {ActionRowStatus:{Command:restore --rm shard1-increment-20240408050855 Status:error Start:2024-04-09 06:40:20 Finish:2024-04-09 06:40:20 Error:can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies} Ctx:<nil> Cancel:<nil>} logger=status
2024/04/09 06:40:20.993540 error API /backup/restore error: can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies logger=server

image

image

grep -m 1 -C 50 -i "code: 57" /var/log/clickhouse-server/clickhouse-server.err.log


clickhouse@chi-clickhouse-cluster-dtstack-1-0-0:/$ grep -m 1 -C 50 -i "code: 57" /var/log/clickhouse-server/clickhouse-server.err.log
11. DB::create(DB::StorageFactory::Arguments const&) @ 0x0000000012f55958 in /usr/bin/clickhouse
12. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x000000001268513b in /usr/bin/clickhouse
13. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x0000000011d1b91e in /usr/bin/clickhouse
14. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
15. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
16. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
17. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
18. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
19. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
20. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
21. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
22. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
23. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
24. ? @ 0x00007f41f0328609 in ?
25. ? @ 0x00007f41f024d353 in ?

2024.04.09 06:36:09.239587 [ 304 ] {d244a429-507e-4c0b-bc14-e7224485c684} StorageDistributed (metrics_summary): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:09.307556 [ 280 ] {} void DB::SystemLog::flushImpl(const std::vector &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: std::exception. Code: 1001, type: std::1::fs::filesystem::filesystem_error, e.what() = filesystem error: in create_directories: Permission denied ["/var/lib/clickhouse/cold/store"], Stack trace (when copying this message, always include the lines below):

  1. std::system_error::system_error(std::error_code, String const&) @ 0x0000000018069955 in /usr/bin/clickhouse
  2. std::fs::filesystem::filesystem_error::filesystem_error[abi:v15000](String const&, std::fs::filesystem::path const&, std::error_code) @ 0x000000000c68d121 in /usr/bin/clickhouse
  3. void std::fs::filesystem::throw_filesystem_error[abi:v15000]<String&, std::__fs::filesystem::path const&, std::error_code const&>(String&, std::__fs::filesystem::path const&, std::error_code const&) @ 0x000000001801f17a in /usr/bin/clickhouse
  4. std::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler::report(std::error_code const&) const (.llvm.17456556289392299875) @ 0x0000000018022636 in /usr/bin/clickhouse
  5. std::fs::filesystem::create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022e39 in /usr/bin/clickhouse
  6. std::fs::filesystem::create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022ed5 in /usr/bin/clickhouse
  7. std::fs::filesystem::create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022ed5 in /usr/bin/clickhouse
  8. std::fs::filesystem::create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022ed5 in /usr/bin/clickhouse
  9. DB::DiskLocal::createDirectories(String const&) @ 0x00000000114e1d19 in /usr/bin/clickhouse
  10. DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x0000000012bece0d in /usr/bin/clickhouse
  11. DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, bool, std::shared_ptr, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete>, bool) @ 0x0000000012f58fd6 in /usr/bin/clickhouse
  12. DB::create(DB::StorageFactory::Arguments const&) @ 0x0000000012f557cb in /usr/bin/clickhouse
  13. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr, std::shared_ptr, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x000000001268513b in /usr/bin/clickhouse
  14. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete>&) @ 0x0000000011d1b91e in /usr/bin/clickhouse
  15. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
  16. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
  17. DB::SystemLog::prepareTable() @ 0x00000000121005a0 in /usr/bin/clickhouse
  18. DB::SystemLog::savingThreadFunction() @ 0x0000000012101b8a in /usr/bin/clickhouse
  19. void std::function::policy_invoker<void ()>::__call_impl<std::function::default_alloc_func<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::SystemLogBase::startup()::'lambda'()>(DB::SystemLogBase::startup()::'lambda'()&&)::'lambda'(), void ()>>(std::function::policy_storage const*) @ 0x000000000c7609b2 in /usr/bin/clickhouse
  20. void std::thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::thread_struct, std::default_delete>, void ThreadPoolImpl::scheduleImpl(std::function<void ()>, Priority, std::optional, bool)::'lambda0'()>>(void) @ 0x000000000c725fa4 in /usr/bin/clickhouse
  21. ? @ 0x00007f41f0328609 in ?
  22. ? @ 0x00007f41f024d353 in ?

Cannot print extra info for Poco::Exception (version 23.8.8.21.altinitystable (altinity build)) 2024.04.09 06:36:09.307656 [ 304 ] {6f12932c-35cf-470b-9e0a-328b42d7c91a} StorageDistributed (system_metrics): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:09.409480 [ 304 ] {ded26fd4-38f6-4ef9-8120-b04c28e0928b} StorageDistributed (arms_frontend_log): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:09.428884 [ 304 ] {55dc6cf7-b37f-4530-adb0-efe818032324} StorageDistributed (easydo_agent_monitor): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:09.728726 [ 304 ] {db11e494-903f-4f17-bd11-7d95d2048793} StorageDistributed (easydo_collect_task_monitor): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:09.853968 [ 304 ] {a5ed609b-f284-4987-a6a0-df96c6f943d9} StorageDistributed (proc_stat_metrics): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:09.865944 [ 304 ] {0d1cacc1-0939-4aa8-b5b0-b3e9e0e0f9d2} StorageDistributed (test): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:10.165324 [ 304 ] {8b79bb80-4a1d-412e-8ae5-de47eff7c39b} StorageDistributed (system_attribute_meta): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored. 2024.04.09 06:36:10.176574 [ 304 ] {c27d069e-e20f-46e7-ac79-c1a386f9fa68} executeQuery: Code: 57. DB::Exception: Directory for table data store/287/2875f3f9-466f-43ac-938a-62fe69210777/ already exists. (TABLE_ALREADY_EXISTS) (version 23.8.8.21.altinitystable (altinity build)) (from [::1]:54944) (in query: CREATE TABLE local_1.test UUID '2875f3f9-466f-43ac-938a-62fe69210777' (__appname__ LowCardinality(String), __source__ LowCardinality(String), _tenant_id_ Float64, _message_ String, _mode_ String, __path__ LowCardinality(String), __ip__ LowCardinality(String), _collect_time_ Float64, _at_date_ Date, __hostname__ LowCardinality(String), _receive_timestamp_ DateTime64(3, 'UTC'), _custom_tags_ String, _at_timestamp_ DateTime64(3, 'UTC'), _tag_set_ Array(String), _context_seq_ Float64, _field_set_ Array(String)) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/test/2024-04-08/{shard}/hits', '{replica}') PARTITION BY toString(_atdate) ORDER BY (_atdate, _attimestamp, intHash64(toInt64(toDateTime(_attimestamp)))) SAMPLE BY intHash64(toInt64(toDateTime(_attimestamp))) SETTINGS index_granularity = 8192), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c644ed7 in /usr/bin/clickhouse
  2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity::type, std::type_identity::type>, String&&, String&&) @ 0x0000000007157e87 in /usr/bin/clickhouse
  3. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete>&) @ 0x0000000011d1e14c in /usr/bin/clickhouse
  4. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
  5. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
  6. DB::executeQueryImpl(char const, char const, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
  7. DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
  8. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
  9. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
  10. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
  11. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
  12. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
  13. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
  14. ? @ 0x00007f41f0328609 in ?
  15. ? @ 0x00007f41f024d353 in ?

2024.04.09 06:36:10.176780 [ 304 ] {c27d069e-e20f-46e7-ac79-c1a386f9fa68} TCPHandler: Code: 57. DB::Exception: Directory for table data store/287/2875f3f9-466f-43ac-938a-62fe69210777/ already exists. (TABLE_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c644ed7 in /usr/bin/clickhouse
  2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity::type, std::type_identity::type>, String&&, String&&) @ 0x0000000007157e87 in /usr/bin/clickhouse
  3. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete>&) @ 0x0000000011d1e14c in /usr/bin/clickhouse
  4. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
  5. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
  6. DB::executeQueryImpl(char const, char const, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
  7. DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
  8. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
  9. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
  10. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
  11. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
  12. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
  13. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
  14. ? @ 0x00007f41f0328609 in ?
  15. ? @ 0x00007f41f024d353 in ?

2024.04.09 06:36:10.182710 [ 304 ] {8e74c6fa-8e93-4c7b-921c-2b6d485f09d7} executeQuery: Code: 57. DB::Exception: Directory for table data store/0ad/0add270e-2cdc-4e78-aadb-11c0bf77df7b/ already exists. (TABLE_ALREADY_EXISTS) (version 23.8.8.21.altinitystable (altinity build)) (from [::1]:54968) (in query: CREATE TABLE local_1.system_metrics UUID '0add270e-2cdc-4e78-aadb-11c0bf77df7b' (kernel_vmstat__thp_zero_page_alloc Float64, netstat__tcp_closing Float64, disk__used_percent Float64, net__interface String, processes__paging Float64, processes__total Float64, kernel_vmstat__zone_reclaim_failed Float64, swap__in Float64, procstat__mem_vms Float64, kernel_vmstat__numa_foreign Float64, kernel_vmstat__numa_huge_pte_updates Float64, kernel_vmstat__pgpgin Float64, net__bytes_sent Float64, procstat__pid Float64, cpu__usage_guest Float64, cpu__time_guest_nice Float64, mem__cached Float64, system__n_users Float64, cpu__usage_guest_nice Float64, diskio__name String, linux_sysctl_fs__aio_nr Float64, net__bytes_recv_per_sec Float64, kernel_vmstat__nr_page_table_pages Float64, kernel_vmstat__unevictable_pgs_stranded Float64, system__n_cpus Float64, __hostname__ String, linux_sysctl_fs__inode_nr Float64, kernel_vmstat__compact_fail Float64, cpu__time_iowait Float64, kernel_vmstat__thp_split Float64, processes__unknown Float64, kernel_vmstat__balloon_deflate Float64, system__n_net_cards Float64, cpu__usage_iowait Float64, diskio__read_await Float64, processes__zombies Float64, system__load5 Float64, _tenant_id_ Float64, _mode_ String, kernel_vmstat__compact_stall Float64, kernel_vmstat__nr_dirty Float64, kernel_vmstat__nr_kernel_stack Float64, kernel_vmstat__nr_anon_pages Float64, kernel_vmstat__nr_free_pages Float64, disk__total Float64, kernel_vmstat__allocstall Float64, diskio__iops_in_progress Float64, kernel_vmstat__nr_vmscan_write Float64, procstat__mem_stack Float64, procstat__cpu_usage Float64, kernel_vmstat__pgrefill_normal Float64, kernel_vmstat__unevictable_pgs_mlocked Float64, timestamp String, kernel_vmstat__nr_file_pages Float64, netstat__tcp_fin_wait1 Float64, procstat__mem_rss Float64, netstat__tcp_fin_wait2 Float64, kernel_vmstat__pgsteal_direct_dma32 Float64, kernel_vmstat__nr_inactive_file Float64, kernel__processes_forked Float64, kernel_vmstat__swap_ra_hit Float64, kernel_vmstat__nr_anon_transparent_hugepages Float64, netstat__tcp_listen Float64, kernel_vmstat__htlb_buddy_alloc_fail Float64, kernel_vmstat__pgrefill_movable Float64, system__load1 Float64, processes__running Float64, kernel_vmstat__pgscan_kswapd_dma32 Float64, mem__buffered Float64, kernel_vmstat__compact_isolated Float64, disk__free Float64, kernel_vmstat__pageoutrun Float64, kernel_vmstat__pgalloc_movable Float64, mem__available_percent Float64, kernel_vmstat__pgscan_direct_normal Float64, procstat__mem_swap Float64, kernel_vmstat__nr_shmem Float64, kernel_vmstat__pgfault Float64, __instance_id__ LowCardinality(String), kernel_vmstat__pgscan_direct_dma Float64, _custom_tags String, kernel_vmstat__kswapd_inodesteal Float64, cpu__time_softirq Float64, disk__inodes_total Float64, kernel_vmstat__pgscan_kswapd_movable Float64, kernel_vmstat__workingset_refault Float64, procstat__mem_data Float64, kernel_vmstat__nr_isolated_file Float64, linux_sysctl_fs__aio_max_nr Float64, diskio__write_time Float64, linux_sysctl_fs__inode_free_nr Float64, swap__used_percent Float64, cpu__time_steal Float64, netstat__tcp_syn_sent Float64, kernel_vmstat__kswapd_high_wmark_hit_quickly Float64, kernel_vmstat__pgmigrate_success Float64, mem__total Float64, kernel_vmstat__pglazyfreed Float64, linux_sysctl_fs__dentry_want_pages Float64, cpu__usage_idle Float64, cpu__usage_user Float64, __ip__ String, kernel_vmstat__pgscan_direct_dma32 Float64, kernel_vmstat__unevictable_pgs_culled Float64, kernel_vmstat__thp_fault_fallback Float64, kernel_vmstat__pgsteal_kswapd_dma32 Float64, kernel_vmstat__workingset_activate Float64, kernel_vmstat__thp_collapse_alloc Float64, cpu__time_irq Float64, kernel_vmstat__drop_slab Float64, kernel_vmstat__balloon_migrate Float64, system__n_disks Float64, kernel_vmstat__nr_written Float64, kernel_vmstat__nr_dirty_threshold Float64, kernel_vmstat__pgsteal_kswapd_dma Float64, kernel_vmstat__pgsteal_kswapd_normal Float64, kernel_vmstat__slabs_scanned Float64, procstat__mem_locked Float64, diskio__writes Float64, diskio__write_bytes_per_sec Float64, netstat__tcp_time_wait Float64, cpu__usage_irq Float64, kernel_vmstat__nr_slab_reclaimable Float64, disk__inodes_used Float64, kernel_vmstat__numa_interleave Float64, _receive_timestamp_ DateTime64(3, 'UTC'), kernel_vmstat__pgmigrate_fail Float64, _custom_tags_ String, kernel_vmstat__unevictable_pgs_scanned Float64, diskio__await Float64, net__drop_out Float64, kernel_vmstat__nr_unstable Float64, kernel_vmstat__nr_vmscan_immediate_reclaim Float64, kernel__boot_time Float64, kernel_vmstat__pgrotated Float64, mem__free Float64, netstat__tcp_last_ack Float64, kernel_vmstat__pgmajfault Float64, diskio__io_time Float64, processes__blocked Float64, cpu__time_guest Float64, procstat__write_count Float64, kernel_vmstat__nr_dirty_background_threshold Float64, kernel_vmstat__htlb_buddy_alloc_success Float64, kernel_vmstat__pgalloc_normal Float64, procstat__process_name String, kernel_vmstat__swap_ra Float64, _tag_set_ Array(String), swap__free Float64, kernel_vmstat__pgfree Float64, swap__total Float64, kernel_vmstat__numa_pte_updates Float64, kernel_vmstat__nr_writeback Float64, net__err_out Float64, net__drop_in Float64, kernel_vmstat__compact_success Float64, kernel_vmstat__pgdeactivate Float64, _at_date_ Date, procstat__status Float64, kernel_vmstat__nr_unevictable Float64, cpu__usage_cpu String, netstat__tcp_established Float64, kernel_vmstat__pgscan_kswapd_normal Float64, kernel_vmstat__nr_active_file Float64, mem__active Float64, kernel_vmstat__pgscan_direct_throttle Float64, kernel_vmstat__pgsteal_direct_movable Float64, _metrics_type_ String, netstat__udp_socket Float64, diskio__merged_read_count_per_sec Float64, diskio__reads_per_sec Float64, kernel__context_switches Float64, kernel_vmstat__nr_inactive_anon Float64, net__packets_sent Float64, netstat__tcp_close_wait Float64, system__load15 Float64, kernel_vmstat__numa_miss Float64, kernel__interrupts Float64, diskio__read_bytes_per_sec Float64, kernel_vmstat__nr_alloc_batch Float64, kernel_vmstat__pgalloc_dma32 Float64, mem__used Float64, _field_set_ Array(String), kernel_vmstat__pgrefill_dma Float64, diskio__read_bytes Float64, kernel_vmstat__kswapd_low_wmark_hit_quickly Float64, procstat__write_bytes Float64, mem__used_percent Float64, disk__path String, linux_sysctl_fs__dentry_age_limit Float64, netstat__tcp_syn_recv Float64, kernel_vmstat__compact_migrate_scanned Float64, kernel_vmstat__pgsteal_kswapd_movable Float64, kernel_vmstat__nr_bounce Float64, cpu__usage_nice Float64, kernel_vmstat__nr_free_cma Float64, kernel_vmstat__thp_zero_page_alloc_failed Float64, disk__inodes_fress Float64, kernel_vmstat__pgalloc_dma Float64, disk__fstype String, kernel_vmstat__pgscan_kswapd_dma Float64, linux_sysctl_fs__file_max Float64, kernel_vmstat__pgscan_direct_movable Float64, diskio__read_time Float64, net__packets_recv Float64, diskio__writes_per_sec Float64, processes__total_threads Float64, cpu__usage_system Float64, procstat__read_bytes Float64, kernel_vmstat__pgrefill_dma32 Float64, kernel_vmstat__thp_fault_alloc Float64, cpu__time_cpu String, cpu__usage_steal Float64, kernel_vmstat__nr_mapped Float64, kernel_vmstat__unevictable_pgs_rescued Float64, linux_sysctl_fs__file_nr Float64, kernel_vmstat__nr_mlock Float64, diskio__write_bytes Float64, net__hw_addr String, kernel_vmstat__nr_dirtied Float64, kernel_vmstat__unevictable_pgs_munlocked Float64, diskio__write_await Float64, cpu__time_nice Float64, swap__out Float64, swap__used Float64, diskio__merged_write_count_per_sec Float64, netstat__tcp_close Float64, linux_sysctl_fs__dentry_unused_nr Float64, system__n_services Float64, kernel_vmstat__numa_hint_faults_local Float64, __appname__ String, diskio__merged_read_count Float64, __source__ String, processes__stopped Float64, linux_sysctl_fs__dentry_nr Float64, net__err_in Float64, kernel_vmstat__pswpin Float64, kernel_vmstat__drop_pagecache Float64, __instance_name__ LowCardinality(String), diskio__merged_write_count Float64, _at_timestamp_ DateTime64(3, 'UTC'), mem__inactive Float64, net__bytes_sent_per_sec Float64, kernel_vmstat__pgsteal_direct_dma Float64, kernel_vmstat__pswpout Float64, net__bytes_recv Float64, kernel_vmstat__numa_pages_migrated Float64, disk__used Float64, netstat__tcp_none Float64, kernel_vmstat__workingset_nodereclaim Float64, kernel_vmstat__balloon_inflate Float64, cpu__time_system Float64, procstat__read_count Float64, mem__available Float64, kernel_vmstat__numa_other Float64, diskio__io_util Float64, kernel_vmstat__pgpgout Float64, system__uptime Float64, processes__sleeping Float64, kernel_vmstat__nr_active_anon Float64, kernel_vmstat__numa_hint_faults Float64, cpu__time_user Float64, kernel_vmstat__pginodesteal Float64, cpu__usage_softirq Float64, processes__dead Float64, disk__device String, kernel_vmstat__nr_writeback_temp Float64, kernel_vmstat__nr_isolated_anon Float64, system__uptime_format String, diskio__reads Float64, kernel_vmstat__nr_slab_unreclaimable Float64, kernel_vmstat__pgsteal_direct_normal Float64, kernel_vmstat__numa_local Float64, cpu__time_idle Float64, kernel_vmstat__numa_hit Float64, kernel_vmstat__compact_free_scanned Float64, net__packets_sent_per_sec Float64, kernel_vmstat__pgactivate Float64, kernel_vmstat__thp_collapse_alloc_failed Float64, linux_sysctl_fs__inode_preshrink_nr Float64, kernel_vmstat__unevictable_pgs_cleared Float64, net__packets_recv_per_sec Float64, __path__ LowCardinality(String), __device__ LowCardinality(String), __fstype__ LowCardinality(String), __name__ LowCardinality(String), __interface__ LowCardinality(String), system__platform String, system__platform_family String, system__os String, system__platform_version String, system__kernel_arch String, system__kernel_version String, cpu__usage_busy Float64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/system_metrics/2024-04-08/{shard}/hits', '{replica}') PARTITION BY toString(_atdate) ORDER BY (_atdate, _attimestamp, intHash64(toInt64(toDateTime(_attimestamp)))) SAMPLE BY intHash64(toInt64(toDateTime(_attimestamp))) SETTINGS index_granularity = 8192), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c644ed7 in /usr/bin/clickhouse
  2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity::type, std::type_identity::type>, String&&, String&&) @ 0x0000000007157e87 in /usr/bin/clickhouse
  3. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete>&) @ 0x0000000011d1e14c in /usr/bin/clickhouse
  4. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
  5. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
  6. DB::executeQueryImpl(char const, char const, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
  7. DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
  8. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
  9. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
  10. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
  11. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
  12. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
  13. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
dxygit1 commented 6 months ago

Could it be because I restored a distributed table referencing a local table first, causing the local table to be affected during the restoration?

Slach commented 6 months ago

could you share grep "72b5f6fe-0721-4c58-aac0-e5598e194e56" -r /var/lib/clickhouse/backup/shard1-increment-20240408050855/metadata/

Slach commented 6 months ago

Could it be because I restored a distributed table referencing a local table first, causing the local table to be affected during the restoration?

no,

it looks like data was not dropped as expected.

but you shared screenshot where executed DROP DATABASE local_1 SYNC this operation shall delete physical all /var/lib/clikhouse/store/XXX/XXXXX directories which relaed to tables which present in local_1 database

Slach commented 6 months ago

according to shared screenshots did you deleted database on -0-0-0 but try to restore on -0-1-0 node?

dxygit1 commented 6 months ago

Yesterday, I tried restoring local_1 first, and then restoring others had no issues. I'll retest it again. Thanks.

Slach commented 6 months ago

ok. close issue, feel free reopen if reproduce again