search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.24k stars 499 forks source link

TiDB operator cannot scale up the tiflash after scaling down to 0 #5834

Open kos-team opened 3 weeks ago

kos-team commented 3 weeks ago

Bug Report

What version of Kubernetes are you using? Client Version: v1.31.1 Kustomize Version: v5.4.2

What version of TiDB Operator are you using? v1.6.0

What's the status of the TiDB cluster pods? TiFlash pods are in CrashBackOffLoop State.

What did you do? We scaled in tiflash from 3 to 0 and then scaled it out from 0 to 3.

How to reproduce

  1. Deploy a TiDB cluster with tiflash enabled, for example: 
    apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
name: test-cluster
spec:
configUpdateStrategy: RollingUpdate
enableDynamicConfiguration: true
helper:
image: alpine:3.16.0
pd:
baseImage: pingcap/pd
# config: "[dashboard]\n  internal-proxy = true\n"
maxFailoverCount: 0
mountClusterClientSecret: true
replicas: 3
requests:
  storage: 10Gi
pvReclaimPolicy: Retain
tidb:
baseImage: pingcap/tidb
config: "[performance]\n  tcp-keep-alive = true\ngraceful-wait-before-shutdown\
  \ = 30\n"
maxFailoverCount: 0
replicas: 3
service:
  externalTrafficPolicy: Local
  type: NodePort
tikv:
baseImage: pingcap/tikv
config: |
  [raftdb]
    max-open-files = 256
  [rocksdb]
    max-open-files = 128
maxFailoverCount: 0
mountClusterClientSecret: true
replicas: 3
requests:
  storage: 100Gi
tiflash:
baseImage: pingcap/tiflash
version: "v8.1.0"
replicas: 3
config:
  # configure the configuration file for TiFlash process
  config: |
    [logger]
      log = "/data0/logs/somelog"
  proxy: |
    [security]
      cert-allowed-cn = ["CNNAME"]
storageClaims:
  - resources:
      requests:
        storage: 10Gi
timezone: UTC
version: v8.1.0
  2. Change the spec.tiflash.replicas from 3 to 0:
  3. After the operator finishes the reconciliation, change the spec.tiflash.replicas back to 3.

What did you expect to see? We expected that TiFlash pods are running and be in Healthy state

What did you see instead? The Tiflash pods kept crashing and be in CrashBackOffLoop state.

Root Cause We think the root cause of this problem is that when scaling in the TiFlash, the stores will be in Tombstone state. After we change the spec.tiflash.replicas from 0 to 3, the operator will delete the original statefulset and create a new one with replicas set to 3 instead of changing the original statefulset. This behaviour bypasses the ScaleOut function at this line https://github.com/pingcap/tidb-operator/blob/master/pkg/manager/member/tiflash_scaler.go#L52. After encounter this issue, the user cannot simply delete the CR and apply it again to make the TiFlash run correctly as the operator will not delete pvcs after user deletes CR causing the new cluster reusing the stores that in Tombstone state.

csuzhangxc commented 3 weeks ago

can you show the log of TiFlash? and what's the TidbCluster CR .status after scaled in to 0?

kos-team commented 3 weeks ago

Here is the TiFlash log, the key error message is [\"failed to start node: StoreTombstone(\\"store is tombstone\\")\"]:

TiFlash log ``` "Logging information to /data0/logs/server.log", "Logging errors to /data0/logs/error.log", "arg matches is ArgMatches { args: {\"engine-addr\": MatchedArg { occurs: 1, indices: [2], vals: [\"0.0.0.0:3930\"] }, \"advertise-addr\": MatchedArg { occurs: 1, indices: [12], vals: [\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20170\"] }, \"engine-git-hash\": MatchedArg { occurs: 1, indices: [8], vals: [\"c1838001167c8ba83af759085a71ad61e6c2a5af\"] }, \"config\": MatchedArg { occurs: 1, indices: [6], vals: [\"/data0/proxy.toml\"] }, \"pd-endpoints\": MatchedArg { occurs: 1, indices: [16], vals: [\"test-cluster-pd.acto-namespace.svc:2379\"] }, \"engine-label\": MatchedArg { occurs: 1, indices: [18], vals: [\"tiflash\"] }, \"engine-version\": MatchedArg { occurs: 1, indices: [10], vals: [\"v8.1.0\"] }, \"data-dir\": MatchedArg { occurs: 1, indices: [4], vals: [\"/data0/proxy\"] }, \"addr\": MatchedArg { occurs: 1, indices: [14], vals: [\"0.0.0.0:20170\"] }}, subcommand: None, usage: Some(\"USAGE:\\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version \") }", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:56] [\"Welcome To RaftStore Proxy\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"Git Commit Hash: 7dc50b4eb06124e31f03adb06c20ff7ab61c5f79\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"Git Commit Branch: HEAD\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"UTC Build Time: 2024-05-24 06:42:38\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"Storage Engine: tiflash\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"Prometheus Prefix: tiflash_proxy_\"] [thread_id=0x4]", "[2024/11/01 01:20:56.222 +00:00] [INFO] [lib.rs:58] [\"Profile: release\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [INFO] [lib.rs:58] [\"Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure openssl-vendored\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [INFO] [mod.rs:80] [\"cgroup quota: memory=None, cpu=None, cores={}\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [INFO] [mod.rs:87] [\"memory limit in bytes: 270365872128, cpu cores quota: 40\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [WARN] [run.rs:1664] [\"check: kernel\"] [err=\"kernel parameters net.core.somaxconn got 4096, expect 32768\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [WARN] [run.rs:1664] [\"check: kernel\"] [err=\"kernel parameters net.ipv4.tcp_syncookies got 1, expect 0\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [WARN] [run.rs:1664] [\"check: kernel\"] [err=\"kernel parameters vm.swappiness got 60, expect 0\"] [thread_id=0x4]", "[2024/11/01 01:20:56.223 +00:00] [INFO] [run.rs:389] [\"bootstrap with tikv-raft-engine\"] [thread_id=0x4]", "[2024/11/01 01:20:56.226 +00:00] [INFO] [util.rs:638] [\"connecting to PD endpoint\"] [endpoints=test-cluster-pd.acto-namespace.svc:2379] [thread_id=0x4]", "[2024/11/01 01:20:56.232 +00:00] [INFO] [] [\"TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter\"] [thread_id=0x4]", "[2024/11/01 01:20:56.235 +00:00] [INFO] [util.rs:638] [\"connecting to PD endpoint\"] [endpoints=http://test-cluster-pd-1.test-cluster-pd-peer.acto-namespace.svc:2379] [thread_id=0x4]", "[2024/11/01 01:20:56.243 +00:00] [INFO] [util.rs:638] [\"connecting to PD endpoint\"] [endpoints=http://test-cluster-pd-0.test-cluster-pd-peer.acto-namespace.svc:2379] [thread_id=0x4]", "[2024/11/01 01:20:56.252 +00:00] [INFO] [util.rs:807] [\"connected to PD member\"] [endpoints=http://test-cluster-pd-0.test-cluster-pd-peer.acto-namespace.svc:2379] [thread_id=0x4]", "[2024/11/01 01:20:56.252 +00:00] [INFO] [util.rs:630] [\"all PD endpoints are consistent\"] [endpoints=\"[\\\"test-cluster-pd.acto-namespace.svc:2379\\\"]\"] [thread_id=0x4]", "[2024/11/01 01:20:56.253 +00:00] [INFO] [common.rs:329] [\"connect to PD cluster\"] [cluster_id=7432111146104685610] [thread_id=0x4]", "[2024/11/01 01:20:56.253 +00:00] [INFO] [run.rs:622] [\"linked with external jemalloc\"] [thread_id=0x4]", "[2024/11/01 01:20:56.253 +00:00] [INFO] [run.rs:627] [\"using proxy config\"] [config=\"ProxyConfig { server: ServerConfig { engine_addr: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:3930\\\", engine_store_version: \\\"v8.1.0\\\", engine_store_git_hash: \\\"c1838001167c8ba83af759085a71ad61e6c2a5af\\\", addr: \\\"0.0.0.0:20170\\\", status_addr: \\\"0.0.0.0:20292\\\", advertise_status_addr: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20292\\\", advertise_addr: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20170\\\", background_thread_count: 4, status_thread_pool_size: 2, reject_messages_on_memory_ratio: 0.05 }, raft_store: RaftstoreConfig { snap_handle_pool_size: 16, region_worker_tick_interval: ReadableDuration(500ms), apply_low_priority_pool_size: 40, evict_cache_on_memory_ratio: 0.1, enable_v2_compatible_learner: true }, rocksdb: RocksDbConfig { lockcf: LockCfConfig { block_cache_size: Some(ReadableSize(134217728)) }, writecf: WriteCfConfig { block_cache_size: Some(ReadableSize(134217728)) }, defaultcf: DefaultCfConfig { block_cache_size: Some(ReadableSize(134217728)) } }, raftdb: RaftDbConfig { defaultcf: DefaultCfConfig { block_cache_size: Some(ReadableSize(13518241792)) } }, storage: StorageConfig { reserve_space: ReadableSize(1073741824) }, enable_io_snoop: false, memory_usage_high_water: 0.1, readpool: ReadPoolConfig { unified: UnifiedReadPoolConfig { max_thread_count: 4 } }, import: ImportConfig { num_threads: 80 }, engine_store: EngineStoreConfig { enable_fast_add_peer: false, enable_unips: false } }\"] [thread_id=0x4]", "[2024/11/01 01:20:56.253 +00:00] [INFO] [config.rs:379] [\"quota: cpu 40 mem in bytes 270365872128\"] [thread_id=0x4]", "[2024/11/01 01:20:56.254 +00:00] [INFO] [config.rs:390] [\"config.server.labels after address: {\\\"engine\\\": \\\"tiflash\\\"}\"] [thread_id=0x4]", "[2024/11/01 01:20:56.254 +00:00] [INFO] [run.rs:629] [\"after address config\"] [config=\"TikvConfig { cfg_path: \\\"/data0/proxy.toml\\\", log_level: LogLevel(Info), log_file: \\\"\\\", log_format: Text, log_rotation_timespan: ReadableDuration(0ns), log_rotation_size: ReadableSize(314572800), slow_log_file: \\\"\\\", slow_log_threshold: ReadableDuration(1s), panic_when_unexpected_key_or_data: false, enable_io_snoop: false, abort_on_panic: false, memory_usage_limit: None, memory_usage_high_water: 0.1, region_cache_memory_limit: ReadableSize(0), log: LogConfig { level: LogLevel(Info), format: Text, enable_timestamp: true, file: File { filename: \\\"\\\", max_size: 300, max_days: 0, max_backups: 0 } }, memory: MemoryConfig { enable_heap_profiling: true, profiling_sample_per_bytes: ReadableSize(524288) }, quota: QuotaConfig { foreground_cpu_time: 0, foreground_write_bandwidth: ReadableSize(0), foreground_read_bandwidth: ReadableSize(0), max_delay_duration: ReadableDuration(500ms), background_cpu_time: 0, background_write_bandwidth: ReadableSize(0), background_read_bandwidth: ReadableSize(0), enable_auto_tune: false }, readpool: ReadPoolConfig { unified: UnifiedReadPoolConfig { min_thread_count: 1, max_thread_count: 4, stack_size: ReadableSize(10485760), max_tasks_per_worker: 2000, auto_adjust_pool_size: false }, storage: StorageReadPoolConfig { use_unified_pool: None, high_concurrency: 8, normal_concurrency: 8, low_concurrency: 8, max_tasks_per_worker_high: 2000, max_tasks_per_worker_normal: 2000, max_tasks_per_worker_low: 2000, stack_size: ReadableSize(10485760) }, coprocessor: CoprReadPoolConfig { use_unified_pool: None, high_concurrency: 32, normal_concurrency: 32, low_concurrency: 32, max_tasks_per_worker_high: 2000, max_tasks_per_worker_normal: 2000, max_tasks_per_worker_low: 2000, stack_size: ReadableSize(10485760) } }, server: Config { cluster_id: 7432111146104685610, addr: \\\"0.0.0.0:20170\\\", advertise_addr: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20170\\\", status_addr: \\\"0.0.0.0:20292\\\", advertise_status_addr: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20292\\\", status_thread_pool_size: 2, max_grpc_send_msg_len: 10485760, raft_client_grpc_send_msg_buffer: 524288, raft_client_queue_size: 8192, raft_client_max_backoff: ReadableDuration(5s), raft_client_initial_reconnect_backoff: ReadableDuration(1s), raft_msg_max_batch_size: 128, grpc_compression_type: None, grpc_gzip_compression_level: 2, grpc_min_message_size_to_compress: 4096, grpc_concurrency: 5, grpc_concurrent_stream: 1024, grpc_raft_conn_num: 1, grpc_memory_pool_quota: ReadableSize(9223372036854775807), grpc_stream_initial_window_size: ReadableSize(2097152), grpc_keepalive_time: ReadableDuration(10s), grpc_keepalive_timeout: ReadableDuration(3s), concurrent_send_snap_limit: 32, concurrent_recv_snap_limit: 32, end_point_recursion_limit: 1000, end_point_stream_channel_size: 8, end_point_batch_row_limit: 64, end_point_stream_batch_row_limit: 128, end_point_enable_batch_if_possible: true, end_point_request_max_handle_duration: None, end_point_max_concurrency: 40, end_point_perf_level: Uninitialized, snap_io_max_bytes_per_sec: ReadableSize(104857600), snap_max_total_size: ReadableSize(0), stats_concurrency: 1, heavy_load_threshold: 75, heavy_load_wait_duration: None, enable_request_batch: true, background_thread_count: 4, end_point_slow_log_threshold: ReadableDuration(1s), forward_max_connections_per_address: 4, reject_messages_on_memory_ratio: 0.05, simplify_metrics: false, labels: {\\\"engine\\\": \\\"tiflash\\\"}, end_point_concurrency: None, end_point_stack_size: None, end_point_max_tasks: None }, storage: Config { data_dir: \\\"/data0/proxy\\\", engine: RaftKv, gc_ratio_threshold: 1.1, max_key_size: 8192, scheduler_concurrency: 524288, scheduler_worker_pool_size: 8, scheduler_pending_write_threshold: ReadableSize(104857600), reserve_space: ReadableSize(1073741824), reserve_raft_space: ReadableSize(1073741824), enable_async_apply_prewrite: false, api_version: 1, enable_ttl: false, background_error_recovery_window: ReadableDuration(3600s), ttl_check_poll_interval: ReadableDuration(43200s), txn_status_cache_capacity: 5120000, flow_control: FlowControlConfig { enable: true, soft_pending_compaction_bytes_limit: ReadableSize(206158430208), hard_pending_compaction_bytes_limit: ReadableSize(1099511627776), memtables_threshold: 5, l0_files_threshold: 20 }, block_cache: BlockCacheConfig { shared: None, capacity: None, num_shard_bits: 6, strict_capacity_limit: false, high_pri_pool_ratio: 0.8, memory_allocator: Some(\\\"nodump\\\") }, io_rate_limit: IoRateLimitConfig { max_bytes_per_sec: ReadableSize(0), mode: WriteOnly, strict: false, foreground_read_priority: High, foreground_write_priority: High, flush_priority: High, level_zero_compaction_priority: Medium, compaction_priority: Low, replication_priority: High, load_balance_priority: High, gc_priority: High, import_priority: Medium, export_priority: Medium, other_priority: High } }, pd: Config { endpoints: [\\\"test-cluster-pd.acto-namespace.svc:2379\\\"], retry_interval: ReadableDuration(300ms), retry_max_count: 9223372036854775807, retry_log_every: 10, update_interval: ReadableDuration(600s), enable_forwarding: false }, metric: MetricConfig { job: \\\"tikv\\\", interval: ReadableDuration(15s), address: \\\"\\\" }, raft_store: Config { prevote: true, raftdb_path: \\\"\\\", capacity: ReadableSize(0), raft_base_tick_interval: ReadableDuration(1s), raft_heartbeat_ticks: 2, raft_election_timeout_ticks: 10, raft_min_election_timeout_ticks: 0, raft_max_election_timeout_ticks: 0, raft_max_size_per_msg: ReadableSize(1048576), raft_max_inflight_msgs: 256, raft_entry_max_size: ReadableSize(8388608), raft_log_compact_sync_interval: ReadableDuration(2s), raft_log_gc_tick_interval: ReadableDuration(3s), request_voter_replicated_index_interval: ReadableDuration(300s), raft_log_gc_threshold: 50, raft_log_gc_count_limit: None, raft_log_gc_size_limit: None, follower_read_max_log_gap: 100, raft_log_reserve_max_ticks: 6, raft_engine_purge_interval: ReadableDuration(10s), max_manual_flush_rate: 3.0, raft_entry_cache_life_time: ReadableDuration(30s), raft_reject_transfer_leader_duration: ReadableDuration(3s), unsafe_disable_check_quorum: false, split_region_check_tick_interval: ReadableDuration(10s), region_split_check_diff: None, region_compact_check_interval: ReadableDuration(300s), region_compact_check_step: None, region_compact_min_tombstones: 10000, region_compact_tombstones_percent: 30, region_compact_min_redundant_rows: 50000, region_compact_redundant_rows_percent: Some(20), pd_heartbeat_tick_interval: ReadableDuration(60s), pd_store_heartbeat_tick_interval: ReadableDuration(10s), pd_report_min_resolved_ts_interval: ReadableDuration(1s), snap_mgr_gc_tick_interval: ReadableDuration(60s), snap_gc_timeout: ReadableDuration(14400s), snap_wait_split_duration: ReadableDuration(34s), lock_cf_compact_interval: ReadableDuration(600s), lock_cf_compact_bytes_threshold: ReadableSize(268435456), periodic_full_compact_start_times: ReadableSchedule([]), periodic_full_compact_start_max_cpu: 0.1, notify_capacity: 40960, messages_per_tick: 4096, max_peer_down_duration: ReadableDuration(600s), max_leader_missing_duration: ReadableDuration(7200s), abnormal_leader_missing_duration: ReadableDuration(600s), peer_stale_state_check_interval: ReadableDuration(300s), gc_peer_check_interval: ReadableDuration(60s), leader_transfer_max_log_lag: 128, snap_apply_batch_size: ReadableSize(10485760), snap_apply_copy_symlink: false, region_worker_tick_interval: ReadableDuration(500ms), clean_stale_ranges_tick: 20, consistency_check_interval: ReadableDuration(0ns), report_region_flow_interval: ReadableDuration(60s), raft_store_max_leader_lease: ReadableDuration(9s), check_leader_lease_interval: ReadableDuration(0ns), renew_leader_lease_advance_duration: ReadableDuration(0ns), allow_unsafe_vote_after_start: false, right_derive_when_split: true, merge_max_log_gap: 10, merge_check_tick_interval: ReadableDuration(2s), use_delete_range: false, snap_generator_pool_size: 2, cleanup_import_sst_interval: ReadableDuration(600s), local_read_batch_size: 1024, apply_batch_system: Config { max_batch_size: None, pool_size: 2, reschedule_duration: ReadableDuration(5s), low_priority_pool_size: 40 }, store_batch_system: Config { max_batch_size: None, pool_size: 2, reschedule_duration: ReadableDuration(5s), low_priority_pool_size: 1 }, store_io_pool_size: 0, store_io_notify_capacity: 40960, future_poll_size: 1, hibernate_regions: true, dev_assert: false, apply_yield_duration: ReadableDuration(500ms), apply_yield_write_size: ReadableSize(32768), perf_level: Uninitialized, evict_cache_on_memory_ratio: 0.1, cmd_batch: true, cmd_batch_concurrent_ready_max_count: 1, raft_write_size_limit: ReadableSize(1048576), waterfall_metrics: true, io_reschedule_concurrent_max_count: 4, io_reschedule_hotpot_duration: ReadableDuration(5s), raft_msg_flush_interval: ReadableDuration(250\u00b5s), region_max_size: ReadableSize(0), region_split_size: ReadableSize(0), clean_stale_peer_delay: ReadableDuration(0ns), inspect_interval: ReadableDuration(100ms), inspect_cpu_util_thd: 0.4, slow_trend_unsensitive_cause: 10.0, slow_trend_unsensitive_result: 0.5, slow_trend_network_io_factor: 0.0, reactive_memory_lock_tick_interval: ReadableDuration(2s), reactive_memory_lock_timeout_tick: 5, report_region_buckets_tick_interval: ReadableDuration(10s), check_long_uncommitted_interval: ReadableDuration(10s), long_uncommitted_base_threshold: ReadableDuration(20s), max_entry_cache_warmup_duration: ReadableDuration(1s), max_snapshot_file_raw_size: ReadableSize(104857600), unreachable_backoff: ReadableDuration(10s), check_peers_availability_interval: ReadableDuration(30s), check_request_snapshot_interval: ReadableDuration(60s), enable_v2_compatible_learner: true }, coprocessor: Config { split_region_on_table: false, batch_split_limit: 10, region_max_size: None, region_split_size: None, region_max_keys: None, region_split_keys: None, consistency_check_method: Mvcc, perf_level: Uninitialized, enable_region_bucket: None, region_bucket_size: ReadableSize(52428800), region_size_threshold_for_approximate: ReadableSize(786432000), prefer_approximate_bucket: true, region_bucket_merge_size_ratio: 0.33 }, coprocessor_v2: Config { coprocessor_plugin_directory: None }, rocksdb: DbConfig { info_log_level: Info, wal_recovery_mode: PointInTime, wal_dir: \\\"\\\", wal_ttl_seconds: 0, wal_size_limit: ReadableSize(0), max_total_wal_size: None, max_background_jobs: 0, max_background_flushes: 0, max_manifest_file_size: ReadableSize(134217728), create_if_missing: true, max_open_files: 40960, enable_statistics: true, stats_dump_period: None, compaction_readahead_size: ReadableSize(0), info_log_max_size: ReadableSize(1073741824), info_log_roll_time: ReadableDuration(0ns), info_log_keep_log_file_num: 10, info_log_dir: \\\"\\\", rate_bytes_per_sec: ReadableSize(10737418240), rate_limiter_refill_period: ReadableDuration(100ms), rate_limiter_mode: WriteOnly, auto_tuned: None, rate_limiter_auto_tuned: true, bytes_per_sync: ReadableSize(1048576), wal_bytes_per_sync: ReadableSize(524288), max_sub_compactions: 0, writable_file_max_buffer_size: ReadableSize(1048576), use_direct_io_for_flush_and_compaction: false, enable_pipelined_write: false, enable_multi_batch_write: None, enable_unordered_write: false, allow_concurrent_memtable_write: None, write_buffer_limit: None, write_buffer_stall_ratio: 0.0, write_buffer_flush_oldest_first: true, paranoid_checks: None, defaultcf: DefaultCfConfig { block_size: ReadableSize(32768), block_cache_size: Some(ReadableSize(134217728)), disable_block_cache: false, cache_index_and_filter_blocks: true, pin_l0_filter_and_index_blocks: true, use_bloom_filter: true, optimize_filters_for_hits: true, optimize_filters_for_memory: false, whole_key_filtering: true, bloom_filter_bits_per_key: 10, block_based_bloom_filter: false, ribbon_filter_above_level: None, read_amp_bytes_per_bit: 0, compression_per_level: [No, No, Lz4, Lz4, Lz4, Zstd, Zstd], write_buffer_size: Some(ReadableSize(134217728)), write_buffer_limit: None, max_write_buffer_number: 5, min_write_buffer_number_to_merge: 1, max_bytes_for_level_base: ReadableSize(536870912), target_file_size_base: None, level0_file_num_compaction_trigger: 4, level0_slowdown_writes_trigger: None, level0_stop_writes_trigger: None, max_compaction_bytes: ReadableSize(2147483648), compaction_pri: MinOverlappingRatio, dynamic_level_bytes: true, num_levels: 7, max_bytes_for_level_multiplier: 10, compaction_style: Level, disable_auto_compactions: false, disable_write_stall: false, soft_pending_compaction_bytes_limit: None, hard_pending_compaction_bytes_limit: None, force_consistency_checks: false, prop_size_index_distance: 4194304, prop_keys_index_distance: 40960, enable_doubly_skiplist: true, enable_compaction_guard: None, compaction_guard_min_output_file_size: ReadableSize(8388608), compaction_guard_max_output_file_size: ReadableSize(134217728), bottommost_level_compression: Zstd, bottommost_zstd_compression_dict_size: 0, bottommost_zstd_compression_sample_size: 0, prepopulate_block_cache: Disabled, format_version: None, checksum: CRC32c, max_compactions: None, ttl: None, periodic_compaction_seconds: None, titan: TitanCfConfig { min_blob_size: ReadableSize(1024), blob_file_compression: Zstd, zstd_dict_size: ReadableSize(0), blob_cache_size: ReadableSize(0), min_gc_batch_size: ReadableSize(16777216), max_gc_batch_size: ReadableSize(67108864), discardable_ratio: 0.5, sample_ratio: None, merge_small_file_threshold: ReadableSize(8388608), blob_run_mode: Normal, level_merge: false, range_merge: true, max_sorted_runs: 20, gc_merge_rewrite: false } }, writecf: WriteCfConfig { block_size: ReadableSize(32768), block_cache_size: Some(ReadableSize(134217728)), disable_block_cache: false, cache_index_and_filter_blocks: true, pin_l0_filter_and_index_blocks: true, use_bloom_filter: true, optimize_filters_for_hits: false, optimize_filters_for_memory: false, whole_key_filtering: false, bloom_filter_bits_per_key: 10, block_based_bloom_filter: false, ribbon_filter_above_level: None, read_amp_bytes_per_bit: 0, compression_per_level: [No, No, Lz4, Lz4, Lz4, Zstd, Zstd], write_buffer_size: Some(ReadableSize(134217728)), write_buffer_limit: None, max_write_buffer_number: 5, min_write_buffer_number_to_merge: 1, max_bytes_for_level_base: ReadableSize(536870912), target_file_size_base: None, level0_file_num_compaction_trigger: 4, level0_slowdown_writes_trigger: None, level0_stop_writes_trigger: None, max_compaction_bytes: ReadableSize(2147483648), compaction_pri: MinOverlappingRatio, dynamic_level_bytes: true, num_levels: 7, max_bytes_for_level_multiplier: 10, compaction_style: Level, disable_auto_compactions: false, disable_write_stall: false, soft_pending_compaction_bytes_limit: None, hard_pending_compaction_bytes_limit: None, force_consistency_checks: false, prop_size_index_distance: 4194304, prop_keys_index_distance: 40960, enable_doubly_skiplist: true, enable_compaction_guard: None, compaction_guard_min_output_file_size: ReadableSize(8388608), compaction_guard_max_output_file_size: ReadableSize(134217728), bottommost_level_compression: Zstd, bottommost_zstd_compression_dict_size: 0, bottommost_zstd_compression_sample_size: 0, prepopulate_block_cache: Disabled, format_version: None, checksum: CRC32c, max_compactions: None, ttl: None, periodic_compaction_seconds: None, titan: TitanCfConfig { min_blob_size: ReadableSize(1024), blob_file_compression: Zstd, zstd_dict_size: ReadableSize(0), blob_cache_size: ReadableSize(0), min_gc_batch_size: ReadableSize(16777216), max_gc_batch_size: ReadableSize(67108864), discardable_ratio: 0.5, sample_ratio: None, merge_small_file_threshold: ReadableSize(8388608), blob_run_mode: ReadOnly, level_merge: false, range_merge: true, max_sorted_runs: 20, gc_merge_rewrite: false } }, lockcf: LockCfConfig { block_size: ReadableSize(16384), block_cache_size: Some(ReadableSize(134217728)), disable_block_cache: false, cache_index_and_filter_blocks: true, pin_l0_filter_and_index_blocks: true, use_bloom_filter: true, optimize_filters_for_hits: false, optimize_filters_for_memory: false, whole_key_filtering: true, bloom_filter_bits_per_key: 10, block_based_bloom_filter: false, ribbon_filter_above_level: None, read_amp_bytes_per_bit: 0, compression_per_level: [No, No, No, No, No, No, No], write_buffer_size: None, write_buffer_limit: None, max_write_buffer_number: 5, min_write_buffer_number_to_merge: 1, max_bytes_for_level_base: ReadableSize(134217728), target_file_size_base: None, level0_file_num_compaction_trigger: 1, level0_slowdown_writes_trigger: None, level0_stop_writes_trigger: None, max_compaction_bytes: ReadableSize(2147483648), compaction_pri: ByCompensatedSize, dynamic_level_bytes: true, num_levels: 7, max_bytes_for_level_multiplier: 10, compaction_style: Level, disable_auto_compactions: false, disable_write_stall: false, soft_pending_compaction_bytes_limit: None, hard_pending_compaction_bytes_limit: None, force_consistency_checks: false, prop_size_index_distance: 4194304, prop_keys_index_distance: 40960, enable_doubly_skiplist: true, enable_compaction_guard: None, compaction_guard_min_output_file_size: ReadableSize(8388608), compaction_guard_max_output_file_size: ReadableSize(134217728), bottommost_level_compression: Disable, bottommost_zstd_compression_dict_size: 0, bottommost_zstd_compression_sample_size: 0, prepopulate_block_cache: Disabled, format_version: None, checksum: CRC32c, max_compactions: None, ttl: None, periodic_compaction_seconds: None, titan: TitanCfConfig { min_blob_size: ReadableSize(1024), blob_file_compression: Zstd, zstd_dict_size: ReadableSize(0), blob_cache_size: ReadableSize(0), min_gc_batch_size: ReadableSize(16777216), max_gc_batch_size: ReadableSize(67108864), discardable_ratio: 0.5, sample_ratio: None, merge_small_file_threshold: ReadableSize(8388608), blob_run_mode: ReadOnly, level_merge: false, range_merge: true, max_sorted_runs: 20, gc_merge_rewrite: false } }, raftcf: RaftCfConfig { block_size: ReadableSize(16384), block_cache_size: None, disable_block_cache: false, cache_index_and_filter_blocks: true, pin_l0_filter_and_index_blocks: true, use_bloom_filter: true, optimize_filters_for_hits: true, optimize_filters_for_memory: false, whole_key_filtering: true, bloom_filter_bits_per_key: 10, block_based_bloom_filter: false, ribbon_filter_above_level: None, read_amp_bytes_per_bit: 0, compression_per_level: [No, No, No, No, No, No, No], write_buffer_size: Some(ReadableSize(134217728)), write_buffer_limit: None, max_write_buffer_number: 5, min_write_buffer_number_to_merge: 1, max_bytes_for_level_base: ReadableSize(134217728), target_file_size_base: None, level0_file_num_compaction_trigger: 1, level0_slowdown_writes_trigger: None, level0_stop_writes_trigger: None, max_compaction_bytes: ReadableSize(2147483648), compaction_pri: ByCompensatedSize, dynamic_level_bytes: true, num_levels: 7, max_bytes_for_level_multiplier: 10, compaction_style: Level, disable_auto_compactions: false, disable_write_stall: false, soft_pending_compaction_bytes_limit: None, hard_pending_compaction_bytes_limit: None, force_consistency_checks: false, prop_size_index_distance: 4194304, prop_keys_index_distance: 40960, enable_doubly_skiplist: true, enable_compaction_guard: None, compaction_guard_min_output_file_size: ReadableSize(8388608), compaction_guard_max_output_file_size: ReadableSize(134217728), bottommost_level_compression: Disable, bottommost_zstd_compression_dict_size: 0, bottommost_zstd_compression_sample_size: 0, prepopulate_block_cache: Disabled, format_version: None, checksum: CRC32c, max_compactions: None, ttl: None, periodic_compaction_seconds: None, titan: TitanCfConfig { min_blob_size: ReadableSize(1024), blob_file_compression: Zstd, zstd_dict_size: ReadableSize(0), blob_cache_size: ReadableSize(0), min_gc_batch_size: ReadableSize(16777216), max_gc_batch_size: ReadableSize(67108864), discardable_ratio: 0.5, sample_ratio: None, merge_small_file_threshold: ReadableSize(8388608), blob_run_mode: ReadOnly, level_merge: false, range_merge: true, max_sorted_runs: 20, gc_merge_rewrite: false } }, titan: TitanDbConfig { enabled: false, dirname: \\\"\\\", disable_gc: false, max_background_gc: 1, purge_obsolete_files_period: ReadableDuration(10s) } }, raftdb: RaftDbConfig { wal_recovery_mode: PointInTime, wal_dir: \\\"\\\", wal_ttl_seconds: 0, wal_size_limit: ReadableSize(0), max_total_wal_size: ReadableSize(4294967296), max_background_jobs: 4, max_background_flushes: 1, max_manifest_file_size: ReadableSize(20971520), create_if_missing: true, max_open_files: 40960, enable_statistics: true, stats_dump_period: ReadableDuration(600s), compaction_readahead_size: ReadableSize(0), info_log_max_size: ReadableSize(1073741824), info_log_roll_time: ReadableDuration(0ns), info_log_keep_log_file_num: 10, info_log_dir: \\\"\\\", info_log_level: Info, max_sub_compactions: 2, writable_file_max_buffer_size: ReadableSize(1048576), use_direct_io_for_flush_and_compaction: false, enable_pipelined_write: true, enable_unordered_write: false, allow_concurrent_memtable_write: true, bytes_per_sync: ReadableSize(1048576), wal_bytes_per_sync: ReadableSize(524288), defaultcf: RaftDefaultCfConfig { block_size: ReadableSize(65536), block_cache_size: Some(ReadableSize(13518241792)), disable_block_cache: false, cache_index_and_filter_blocks: true, pin_l0_filter_and_index_blocks: true, use_bloom_filter: false, optimize_filters_for_hits: true, optimize_filters_for_memory: false, whole_key_filtering: true, bloom_filter_bits_per_key: 10, block_based_bloom_filter: false, ribbon_filter_above_level: None, read_amp_bytes_per_bit: 0, compression_per_level: [No, No, Lz4, Lz4, Lz4, Zstd, Zstd], write_buffer_size: Some(ReadableSize(134217728)), write_buffer_limit: None, max_write_buffer_number: 5, min_write_buffer_number_to_merge: 1, max_bytes_for_level_base: ReadableSize(536870912), target_file_size_base: None, level0_file_num_compaction_trigger: 4, level0_slowdown_writes_trigger: None, level0_stop_writes_trigger: None, max_compaction_bytes: ReadableSize(2147483648), compaction_pri: ByCompensatedSize, dynamic_level_bytes: true, num_levels: 7, max_bytes_for_level_multiplier: 10, compaction_style: Level, disable_auto_compactions: false, disable_write_stall: false, soft_pending_compaction_bytes_limit: None, hard_pending_compaction_bytes_limit: None, force_consistency_checks: false, prop_size_index_distance: 4194304, prop_keys_index_distance: 40960, enable_doubly_skiplist: true, enable_compaction_guard: None, compaction_guard_min_output_file_size: ReadableSize(8388608), compaction_guard_max_output_file_size: ReadableSize(134217728), bottommost_level_compression: Disable, bottommost_zstd_compression_dict_size: 0, bottommost_zstd_compression_sample_size: 0, prepopulate_block_cache: Disabled, format_version: Some(2), checksum: CRC32c, max_compactions: None, ttl: None, periodic_compaction_seconds: None, titan: TitanCfConfig { min_blob_size: ReadableSize(1024), blob_file_compression: Zstd, zstd_dict_size: ReadableSize(0), blob_cache_size: ReadableSize(0), min_gc_batch_size: ReadableSize(16777216), max_gc_batch_size: ReadableSize(67108864), discardable_ratio: 0.5, sample_ratio: None, merge_small_file_threshold: ReadableSize(8388608), blob_run_mode: Normal, level_merge: false, range_merge: true, max_sorted_runs: 20, gc_merge_rewrite: false } }, titan: TitanDbConfig { enabled: false, dirname: \\\"\\\", disable_gc: false, max_background_gc: 4, purge_obsolete_files_period: ReadableDuration(10s) } }, raft_engine: RaftEngineConfig { enable: true, config: Config { dir: \\\"\\\", spill_dir: None, recovery_mode: TolerateTailCorruption, recovery_read_block_size: ReadableSize(16384), recovery_threads: 4, batch_compression_threshold: ReadableSize(8192), compression_level: None, bytes_per_sync: None, format_version: V2, target_file_size: ReadableSize(134217728), purge_threshold: ReadableSize(10737418240), purge_rewrite_threshold: None, purge_rewrite_garbage_ratio: 0.6, memory_limit: None, enable_log_recycle: true, prefill_for_recycle: false, prefill_limit: None } }, security: SecurityConfig { ca_path: \\\"\\\", cert_path: \\\"\\\", key_path: \\\"\\\", override_ssl_target: \\\"\\\", cert_allowed_cn: {}, redact_info_log: None, encryption: EncryptionConfig { data_encryption_method: Plaintext, data_key_rotation_period: ReadableDuration(604800s), enable_file_dictionary_log: true, file_dictionary_rewrite_threshold: 1000000, master_key: Plaintext, previous_master_key: Plaintext } }, import: Config { num_threads: 80, stream_channel_window: 128, import_mode_timeout: ReadableDuration(600s), memory_use_ratio: 0.3 }, backup: BackupConfig { num_threads: 8, batch_size: 8, sst_max_size: ReadableSize(150994944), enable_auto_tune: true, auto_tune_remain_threads: 8, auto_tune_refresh_interval: ReadableDuration(60s), io_thread_size: 2, s3_multi_part_size: ReadableSize(5242880), hadoop: HadoopConfig { home: \\\"\\\", linux_user: \\\"\\\" } }, log_backup: BackupStreamConfig { min_ts_interval: ReadableDuration(10s), max_flush_interval: ReadableDuration(180s), num_threads: 12, enable: true, temp_path: \\\"\\\", file_size_limit: ReadableSize(268435456), temp_file_memory_quota: ReadableSize(536870912), initial_scan_pending_memory_quota: ReadableSize(536870912), initial_scan_rate_limit: ReadableSize(62914560), initial_scan_concurrency: 6 }, pessimistic_txn: Config { wait_for_lock_timeout: ReadableDuration(1s), wake_up_delay_duration: ReadableDuration(20ms), pipelined: true, in_memory: true }, gc: GcConfig { ratio_threshold: 1.1, batch_keys: 512, max_write_bytes_per_sec: ReadableSize(0), enable_compaction_filter: true, compaction_filter_skip_version_check: false, num_threads: 1 }, split: SplitConfig { qps_threshold: None, split_balance_score: 0.25, split_contained_score: 0.5, detect_times: 10, sample_num: 20, sample_threshold: 100, byte_threshold: None, grpc_thread_cpu_overload_threshold_ratio: 0.5, unified_read_pool_thread_cpu_overload_threshold_ratio: 0.8, region_cpu_overload_threshold_ratio: None, size_threshold: None, key_threshold: None }, cdc: CdcConfig { min_ts_interval: ReadableDuration(1s), hibernate_regions_compatible: true, incremental_scan_threads: 4, incremental_scan_concurrency: 6, incremental_scan_concurrency_limit: 10000, incremental_scan_speed_limit: ReadableSize(134217728), incremental_fetch_speed_limit: ReadableSize(536870912), incremental_scan_ts_filter_ratio: 0.2, tso_worker_threads: 1, sink_memory_quota: ReadableSize(536870912), old_value_cache_memory_quota: ReadableSize(536870912), old_value_cache_size: 0 }, resolved_ts: ResolvedTsConfig { enable: true, advance_ts_interval: ReadableDuration(20s), scan_lock_pool_size: 2, memory_quota: ReadableSize(268435456), incremental_scan_concurrency: 6 }, resource_metering: Config { receiver_address: \\\"\\\", report_receiver_interval: ReadableDuration(60s), max_resource_groups: 100, precision: ReadableDuration(1s) }, causal_ts: Config { renew_interval: ReadableDuration(100ms), renew_batch_min_size: 100, renew_batch_max_size: 8192, alloc_ahead_buffer: ReadableDuration(3s) }, resource_control: Config { enabled: true } }\"] [thread_id=0x4]", "[2024/11/01 01:20:56.254 +00:00] [INFO] [mod.rs:2553] [\"readpool.storage.use-unified-pool is not set, set to true by default\"] [thread_id=0x4]", "[2024/11/01 01:20:56.254 +00:00] [INFO] [mod.rs:2576] [\"readpool.coprocessor.use-unified-pool is not set, set to true by default\"] [thread_id=0x4]", "[2024/11/01 01:20:56.260 +00:00] [INFO] [config.rs:464] [\"check_critical_config finished compatible_adjust\"] [thread_id=0x4]", "[2024/11/01 01:20:56.262 +00:00] [INFO] [config.rs:469] [\"check_critical_config finished validate\"] [thread_id=0x4]", "[2024/11/01 01:20:56.262 +00:00] [INFO] [config.rs:471] [\"check_critical_config finished check_critical_cfg_with\"] [thread_id=0x4]", "[2024/11/01 01:20:56.262 +00:00] [INFO] [common.rs:449] [\"beginning system configuration check\"] [thread_id=0x4]", "[2024/11/01 01:20:56.263 +00:00] [INFO] [config.rs:1077] [\"data dir\"] [mount_fs=\"FsInfo { tp: \\\"tmpfs\\\", opts: \\\"rw,relatime,size=367001600k,inode64\\\", mnt_dir: \\\"/data0\\\", fsname: \\\"tmpfs\\\" }\"] [data_path=/data0/proxy] [thread_id=0x4]", "[2024/11/01 01:20:56.263 +00:00] [WARN] [common.rs:465] [\"check: rocksdb-data-dir\"] [err=\"config fs: data-dir.rotation.get: \\\"tmpfs\\\" no device find in block\"] [path=/data0/proxy] [thread_id=0x4]", "[2024/11/01 01:20:56.263 +00:00] [INFO] [config.rs:1077] [\"data dir\"] [mount_fs=\"FsInfo { tp: \\\"tmpfs\\\", opts: \\\"rw,relatime,size=367001600k,inode64\\\", mnt_dir: \\\"/data0\\\", fsname: \\\"tmpfs\\\" }\"] [data_path=/data0/proxy/raft] [thread_id=0x4]", "[2024/11/01 01:20:56.263 +00:00] [WARN] [common.rs:473] [\"check: raftdb-path\"] [err=\"config fs: data-dir.rotation.get: \\\"tmpfs\\\" no device find in block\"] [path=/data0/proxy/raft] [thread_id=0x4]", "[2024/11/01 01:20:56.264 +00:00] [INFO] [run.rs:766] [\"using config\"] [config=\"{\\\"slow-log-file\\\":\\\"\\\",\\\"slow-log-threshold\\\":\\\"1s\\\",\\\"panic-when-unexpected-key-or-data\\\":false,\\\"abort-on-panic\\\":false,\\\"memory-usage-limit\\\":\\\"23201491626B\\\",\\\"memory-usage-high-water\\\":0.1,\\\"region-cache-memory-limit\\\":\\\"0KiB\\\",\\\"log\\\":{\\\"level\\\":\\\"info\\\",\\\"format\\\":\\\"text\\\",\\\"enable-timestamp\\\":true,\\\"file\\\":{\\\"filename\\\":\\\"\\\",\\\"max-size\\\":300,\\\"max-days\\\":0,\\\"max-backups\\\":0}},\\\"memory\\\":{\\\"enable-heap-profiling\\\":true,\\\"profiling-sample-per-bytes\\\":\\\"512KiB\\\"},\\\"quota\\\":{\\\"foreground-cpu-time\\\":0,\\\"foreground-write-bandwidth\\\":\\\"0KiB\\\",\\\"foreground-read-bandwidth\\\":\\\"0KiB\\\",\\\"max-delay-duration\\\":\\\"500ms\\\",\\\"background-cpu-time\\\":0,\\\"background-write-bandwidth\\\":\\\"0KiB\\\",\\\"background-read-bandwidth\\\":\\\"0KiB\\\",\\\"enable-auto-tune\\\":false},\\\"readpool\\\":{\\\"unified\\\":{\\\"min-thread-count\\\":1,\\\"max-thread-count\\\":4,\\\"stack-size\\\":\\\"10MiB\\\",\\\"max-tasks-per-worker\\\":2000,\\\"auto-adjust-pool-size\\\":false},\\\"storage\\\":{\\\"use-unified-pool\\\":true,\\\"high-concurrency\\\":8,\\\"normal-concurrency\\\":8,\\\"low-concurrency\\\":8,\\\"max-tasks-per-worker-high\\\":2000,\\\"max-tasks-per-worker-normal\\\":2000,\\\"max-tasks-per-worker-low\\\":2000,\\\"stack-size\\\":\\\"10MiB\\\"},\\\"coprocessor\\\":{\\\"use-unified-pool\\\":true,\\\"high-concurrency\\\":32,\\\"normal-concurrency\\\":32,\\\"low-concurrency\\\":32,\\\"max-tasks-per-worker-high\\\":2000,\\\"max-tasks-per-worker-normal\\\":2000,\\\"max-tasks-per-worker-low\\\":2000,\\\"stack-size\\\":\\\"10MiB\\\"}},\\\"server\\\":{\\\"addr\\\":\\\"0.0.0.0:20170\\\",\\\"advertise-addr\\\":\\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20170\\\",\\\"status-addr\\\":\\\"0.0.0.0:20292\\\",\\\"advertise-status-addr\\\":\\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20292\\\",\\\"status-thread-pool-size\\\":2,\\\"max-grpc-send-msg-len\\\":10485760,\\\"raft-client-grpc-send-msg-buffer\\\":524288,\\\"raft-client-queue-size\\\":8192,\\\"raft-msg-max-batch-size\\\":128,\\\"grpc-compression-type\\\":\\\"none\\\",\\\"grpc-gzip-compression-level\\\":2,\\\"grpc-min-message-size-to-compress\\\":4096,\\\"grpc-concurrency\\\":5,\\\"grpc-concurrent-stream\\\":1024,\\\"grpc-raft-conn-num\\\":1,\\\"grpc-memory-pool-quota\\\":\\\"9223372036854775807B\\\",\\\"grpc-stream-initial-window-size\\\":\\\"2MiB\\\",\\\"grpc-keepalive-time\\\":\\\"10s\\\",\\\"grpc-keepalive-timeout\\\":\\\"3s\\\",\\\"concurrent-send-snap-limit\\\":32,\\\"concurrent-recv-snap-limit\\\":32,\\\"end-point-recursion-limit\\\":1000,\\\"end-point-stream-channel-size\\\":8,\\\"end-point-batch-row-limit\\\":64,\\\"end-point-stream-batch-row-limit\\\":128,\\\"end-point-enable-batch-if-possible\\\":true,\\\"end-point-request-max-handle-duration\\\":\\\"1m\\\",\\\"end-point-max-concurrency\\\":40,\\\"end-point-perf-level\\\":0,\\\"snap-io-max-bytes-per-sec\\\":\\\"100MiB\\\",\\\"snap-max-total-size\\\":\\\"0KiB\\\",\\\"stats-concurrency\\\":1,\\\"heavy-load-threshold\\\":75,\\\"heavy-load-wait-duration\\\":null,\\\"enable-request-batch\\\":true,\\\"background-thread-count\\\":4,\\\"end-point-slow-log-threshold\\\":\\\"1s\\\",\\\"forward-max-connections-per-address\\\":4,\\\"reject-messages-on-memory-ratio\\\":0.05,\\\"simplify-metrics\\\":false,\\\"labels\\\":{\\\"engine\\\":\\\"tiflash\\\"}},\\\"storage\\\":{\\\"data-dir\\\":\\\"/data0/proxy\\\",\\\"engine\\\":\\\"raft-kv\\\",\\\"gc-ratio-threshold\\\":1.1,\\\"max-key-size\\\":8192,\\\"scheduler-concurrency\\\":524288,\\\"scheduler-worker-pool-size\\\":8,\\\"scheduler-pending-write-threshold\\\":\\\"100MiB\\\",\\\"reserve-space\\\":\\\"1GiB\\\",\\\"reserve-raft-space\\\":\\\"1GiB\\\",\\\"enable-async-apply-prewrite\\\":false,\\\"api-version\\\":1,\\\"enable-ttl\\\":false,\\\"background-error-recovery-window\\\":\\\"1h\\\",\\\"ttl-check-poll-interval\\\":\\\"12h\\\",\\\"txn-status-cache-capacity\\\":5120000,\\\"flow-control\\\":{\\\"enable\\\":true,\\\"soft-pending-compaction-bytes-limit\\\":\\\"192GiB\\\",\\\"hard-pending-compaction-bytes-limit\\\":\\\"1TiB\\\",\\\"memtables-threshold\\\":5,\\\"l0-files-threshold\\\":20},\\\"block-cache\\\":{\\\"shared\\\":null,\\\"capacity\\\":\\\"13276MiB\\\",\\\"num-shard-bits\\\":6,\\\"strict-capacity-limit\\\":false,\\\"high-pri-pool-ratio\\\":0.8,\\\"memory-allocator\\\":\\\"nodump\\\"},\\\"io-rate-limit\\\":{\\\"max-bytes-per-sec\\\":\\\"0KiB\\\",\\\"mode\\\":\\\"write-only\\\",\\\"strict\\\":false,\\\"foreground-read-priority\\\":\\\"high\\\",\\\"foreground-write-priority\\\":\\\"high\\\",\\\"flush-priority\\\":\\\"high\\\",\\\"level-zero-compaction-priority\\\":\\\"medium\\\",\\\"compaction-priority\\\":\\\"low\\\",\\\"replication-priority\\\":\\\"high\\\",\\\"load-balance-priority\\\":\\\"high\\\",\\\"gc-priority\\\":\\\"high\\\",\\\"import-priority\\\":\\\"medium\\\",\\\"export-priority\\\":\\\"medium\\\",\\\"other-priority\\\":\\\"high\\\"}},\\\"pd\\\":{\\\"endpoints\\\":[\\\"test-cluster-pd.acto-namespace.svc:2379\\\"],\\\"retry-interval\\\":\\\"300ms\\\",\\\"retry-max-count\\\":9223372036854775807,\\\"retry-log-every\\\":10,\\\"update-interval\\\":\\\"10m\\\",\\\"enable-forwarding\\\":false},\\\"metric\\\":{\\\"job\\\":\\\"tikv\\\"},\\\"raftstore\\\":{\\\"prevote\\\":true,\\\"raftdb-path\\\":\\\"/data0/proxy/raft\\\",\\\"capacity\\\":\\\"0KiB\\\",\\\"raft-base-tick-interval\\\":\\\"1s\\\",\\\"raft-heartbeat-ticks\\\":2,\\\"raft-election-timeout-ticks\\\":10,\\\"raft-min-election-timeout-ticks\\\":10,\\\"raft-max-election-timeout-ticks\\\":20,\\\"raft-max-size-per-msg\\\":\\\"1MiB\\\",\\\"raft-max-inflight-msgs\\\":256,\\\"raft-entry-max-size\\\":\\\"8MiB\\\",\\\"raft-log-compact-sync-interval\\\":\\\"2s\\\",\\\"raft-log-gc-tick-interval\\\":\\\"3s\\\",\\\"request-voter-replicated-index-interval\\\":\\\"5m\\\",\\\"raft-log-gc-threshold\\\":50,\\\"raft-log-gc-count-limit\\\":73728,\\\"raft-log-gc-size-limit\\\":\\\"72MiB\\\",\\\"follower-read-max-log-gap\\\":100,\\\"raft-log-reserve-max-ticks\\\":6,\\\"raft-engine-purge-interval\\\":\\\"10s\\\",\\\"max-manual-flush-rate\\\":3.0,\\\"raft-entry-cache-life-time\\\":\\\"30s\\\",\\\"split-region-check-tick-interval\\\":\\\"10s\\\",\\\"region-split-check-diff\\\":\\\"6MiB\\\",\\\"region-compact-check-interval\\\":\\\"5m\\\",\\\"region-compact-check-step\\\":100,\\\"region-compact-min-tombstones\\\":10000,\\\"region-compact-tombstones-percent\\\":30,\\\"region-compact-min-redundant-rows\\\":50000,\\\"region-compact-redundant-rows-percent\\\":20,\\\"pd-heartbeat-tick-interval\\\":\\\"1m\\\",\\\"pd-store-heartbeat-tick-interval\\\":\\\"10s\\\",\\\"pd-report-min-resolved-ts-interval\\\":\\\"1s\\\",\\\"snap-mgr-gc-tick-interval\\\":\\\"1m\\\",\\\"snap-gc-timeout\\\":\\\"4h\\\",\\\"snap-wait-split-duration\\\":\\\"34s\\\",\\\"lock-cf-compact-interval\\\":\\\"10m\\\",\\\"lock-cf-compact-bytes-threshold\\\":\\\"256MiB\\\",\\\"periodic-full-compact-start-times\\\":[],\\\"periodic-full-compact-start-max-cpu\\\":0.1,\\\"notify-capacity\\\":40960,\\\"messages-per-tick\\\":4096,\\\"max-peer-down-duration\\\":\\\"10m\\\",\\\"max-leader-missing-duration\\\":\\\"2h\\\",\\\"abnormal-leader-missing-duration\\\":\\\"10m\\\",\\\"peer-stale-state-check-interval\\\":\\\"5m\\\",\\\"gc-peer-check-interval\\\":\\\"1m\\\",\\\"leader-transfer-max-log-lag\\\":128,\\\"snap-apply-batch-size\\\":\\\"10MiB\\\",\\\"snap-apply-copy-symlink\\\":false,\\\"region-worker-tick-interval\\\":\\\"500ms\\\",\\\"clean-stale-ranges-tick\\\":20,\\\"consistency-check-interval\\\":\\\"0s\\\",\\\"report-region-flow-interval\\\":\\\"1m\\\",\\\"raft-store-max-leader-lease\\\":\\\"9s\\\",\\\"check-leader-lease-interval\\\":\\\"2s250ms\\\",\\\"renew-leader-lease-advance-duration\\\":\\\"2s250ms\\\",\\\"right-derive-when-split\\\":true,\\\"merge-max-log-gap\\\":10,\\\"merge-check-tick-interval\\\":\\\"2s\\\",\\\"use-delete-range\\\":false,\\\"snap-generator-pool-size\\\":2,\\\"cleanup-import-sst-interval\\\":\\\"10m\\\",\\\"local-read-batch-size\\\":1024,\\\"apply-max-batch-size\\\":256,\\\"apply-pool-size\\\":2,\\\"apply-reschedule-duration\\\":\\\"5s\\\",\\\"apply-low-priority-pool-size\\\":40,\\\"store-max-batch-size\\\":256,\\\"store-pool-size\\\":2,\\\"store-reschedule-duration\\\":\\\"5s\\\",\\\"store-low-priority-pool-size\\\":0,\\\"store-io-pool-size\\\":0,\\\"store-io-notify-capacity\\\":40960,\\\"future-poll-size\\\":1,\\\"hibernate-regions\\\":true,\\\"dev-assert\\\":false,\\\"apply-yield-duration\\\":\\\"500ms\\\",\\\"apply-yield-write-size\\\":\\\"32KiB\\\",\\\"perf-level\\\":0,\\\"evict-cache-on-memory-ratio\\\":0.1,\\\"cmd-batch\\\":true,\\\"cmd-batch-concurrent-ready-max-count\\\":1,\\\"raft-write-size-limit\\\":\\\"1MiB\\\",\\\"waterfall-metrics\\\":true,\\\"io-reschedule-concurrent-max-count\\\":4,\\\"io-reschedule-hotpot-duration\\\":\\\"5s\\\",\\\"inspect-interval\\\":\\\"100ms\\\",\\\"inspect-cpu-util-thd\\\":0.4,\\\"slow-trend-unsensitive-cause\\\":10.0,\\\"slow-trend-unsensitive-result\\\":0.5,\\\"slow-trend-network-io-factor\\\":0.0,\\\"reactive-memory-lock-tick-interval\\\":\\\"2s\\\",\\\"reactive-memory-lock-timeout-tick\\\":5,\\\"report-region-buckets-tick-interval\\\":\\\"10s\\\",\\\"check-long-uncommitted-interval\\\":\\\"10s\\\",\\\"long-uncommitted-base-threshold\\\":\\\"20s\\\",\\\"max-entry-cache-warmup-duration\\\":\\\"1s\\\",\\\"max-snapshot-file-raw-size\\\":\\\"100MiB\\\",\\\"unreachable-backoff\\\":\\\"10s\\\",\\\"enable-v2-compatible-learner\\\":true},\\\"coprocessor\\\":{\\\"split-region-on-table\\\":false,\\\"batch-split-limit\\\":10,\\\"region-max-size\\\":\\\"144MiB\\\",\\\"region-split-size\\\":\\\"96MiB\\\",\\\"region-max-keys\\\":1440000,\\\"region-split-keys\\\":960000,\\\"consistency-check-method\\\":\\\"mvcc\\\",\\\"enable-region-bucket\\\":null,\\\"region-bucket-size\\\":\\\"50MiB\\\",\\\"region-size-threshold-for-approximate\\\":\\\"750MiB\\\",\\\"prefer-approximate-bucket\\\":true,\\\"region-bucket-merge-size-ratio\\\":0.33},\\\"coprocessor-v2\\\":{\\\"coprocessor-plugin-directory\\\":null},\\\"rocksdb\\\":{\\\"info-log-level\\\":\\\"info\\\",\\\"wal-recovery-mode\\\":2,\\\"wal-dir\\\":\\\"\\\",\\\"wal-ttl-seconds\\\":0,\\\"wal-size-limit\\\":\\\"0KiB\\\",\\\"max-total-wal-size\\\":\\\"4GiB\\\",\\\"max-background-jobs\\\":9,\\\"max-background-flushes\\\":3,\\\"max-manifest-file-size\\\":\\\"128MiB\\\",\\\"create-if-missing\\\":true,\\\"max-open-files\\\":40960,\\\"stats-dump-period\\\":\\\"10m\\\",\\\"compaction-readahead-size\\\":\\\"0KiB\\\",\\\"info-log-max-size\\\":\\\"1GiB\\\",\\\"info-log-roll-time\\\":\\\"0s\\\",\\\"info-log-keep-log-file-num\\\":10,\\\"info-log-dir\\\":\\\"\\\",\\\"rate-bytes-per-sec\\\":\\\"10GiB\\\",\\\"rate-limiter-refill-period\\\":\\\"100ms\\\",\\\"rate-limiter-mode\\\":2,\\\"rate-limiter-auto-tuned\\\":true,\\\"bytes-per-sync\\\":\\\"1MiB\\\",\\\"wal-bytes-per-sync\\\":\\\"512KiB\\\",\\\"max-sub-compactions\\\":3,\\\"writable-file-max-buffer-size\\\":\\\"1MiB\\\",\\\"use-direct-io-for-flush-and-compaction\\\":false,\\\"enable-pipelined-write\\\":false,\\\"enable-multi-batch-write\\\":null,\\\"enable-unordered-write\\\":false,\\\"allow-concurrent-memtable-write\\\":true,\\\"write-buffer-limit\\\":null,\\\"defaultcf\\\":{\\\"block-size\\\":\\\"32KiB\\\",\\\"block-cache-size\\\":\\\"128MiB\\\",\\\"disable-block-cache\\\":false,\\\"cache-index-and-filter-blocks\\\":true,\\\"pin-l0-filter-and-index-blocks\\\":true,\\\"use-bloom-filter\\\":true,\\\"optimize-filters-for-hits\\\":true,\\\"optimize-filters-for-memory\\\":false,\\\"whole-key-filtering\\\":true,\\\"bloom-filter-bits-per-key\\\":10,\\\"block-based-bloom-filter\\\":false,\\\"ribbon-filter-above-level\\\":null,\\\"read-amp-bytes-per-bit\\\":0,\\\"compression-per-level\\\":[\\\"no\\\",\\\"no\\\",\\\"lz4\\\",\\\"lz4\\\",\\\"lz4\\\",\\\"zstd\\\",\\\"zstd\\\"],\\\"write-buffer-size\\\":\\\"128MiB\\\",\\\"write-buffer-limit\\\":null,\\\"max-write-buffer-number\\\":5,\\\"min-write-buffer-number-to-merge\\\":1,\\\"max-bytes-for-level-base\\\":\\\"512MiB\\\",\\\"target-file-size-base\\\":null,\\\"level0-file-num-compaction-trigger\\\":4,\\\"level0-slowdown-writes-trigger\\\":20,\\\"level0-stop-writes-trigger\\\":20,\\\"max-compaction-bytes\\\":\\\"2GiB\\\",\\\"compaction-pri\\\":3,\\\"dynamic-level-bytes\\\":true,\\\"num-levels\\\":7,\\\"max-bytes-for-level-multiplier\\\":10,\\\"compaction-style\\\":0,\\\"disable-auto-compactions\\\":false,\\\"disable-write-stall\\\":true,\\\"soft-pending-compaction-bytes-limit\\\":\\\"192GiB\\\",\\\"hard-pending-compaction-bytes-limit\\\":\\\"1TiB\\\",\\\"force-consistency-checks\\\":false,\\\"prop-size-index-distance\\\":4194304,\\\"prop-keys-index-distance\\\":40960,\\\"enable-doubly-skiplist\\\":true,\\\"enable-compaction-guard\\\":true,\\\"compaction-guard-min-output-file-size\\\":\\\"8MiB\\\",\\\"compaction-guard-max-output-file-size\\\":\\\"128MiB\\\",\\\"bottommost-level-compression\\\":\\\"zstd\\\",\\\"bottommost-zstd-compression-dict-size\\\":0,\\\"bottommost-zstd-compression-sample-size\\\":0,\\\"prepopulate-block-cache\\\":\\\"disabled\\\",\\\"format-version\\\":null,\\\"checksum\\\":\\\"crc32c\\\",\\\"max-compactions\\\":null,\\\"ttl\\\":null,\\\"periodic-compaction-seconds\\\":null,\\\"titan\\\":{\\\"min-blob-size\\\":\\\"1KiB\\\",\\\"blob-file-compression\\\":\\\"zstd\\\",\\\"zstd-dict-size\\\":\\\"0KiB\\\",\\\"blob-cache-size\\\":\\\"0KiB\\\",\\\"min-gc-batch-size\\\":\\\"16MiB\\\",\\\"max-gc-batch-size\\\":\\\"64MiB\\\",\\\"discardable-ratio\\\":0.5,\\\"merge-small-file-threshold\\\":\\\"8MiB\\\",\\\"blob-run-mode\\\":\\\"normal\\\",\\\"level-merge\\\":false,\\\"range-merge\\\":true,\\\"max-sorted-runs\\\":20}},\\\"writecf\\\":{\\\"block-size\\\":\\\"32KiB\\\",\\\"block-cache-size\\\":\\\"128MiB\\\",\\\"disable-block-cache\\\":false,\\\"cache-index-and-filter-blocks\\\":true,\\\"pin-l0-filter-and-index-blocks\\\":true,\\\"use-bloom-filter\\\":true,\\\"optimize-filters-for-hits\\\":false,\\\"optimize-filters-for-memory\\\":false,\\\"whole-key-filtering\\\":false,\\\"bloom-filter-bits-per-key\\\":10,\\\"block-based-bloom-filter\\\":false,\\\"ribbon-filter-above-level\\\":null,\\\"read-amp-bytes-per-bit\\\":0,\\\"compression-per-level\\\":[\\\"no\\\",\\\"no\\\",\\\"lz4\\\",\\\"lz4\\\",\\\"lz4\\\",\\\"zstd\\\",\\\"zstd\\\"],\\\"write-buffer-size\\\":\\\"128MiB\\\",\\\"write-buffer-limit\\\":null,\\\"max-write-buffer-number\\\":5,\\\"min-write-buffer-number-to-merge\\\":1,\\\"max-bytes-for-level-base\\\":\\\"512MiB\\\",\\\"target-file-size-base\\\":null,\\\"level0-file-num-compaction-trigger\\\":4,\\\"level0-slowdown-writes-trigger\\\":20,\\\"level0-stop-writes-trigger\\\":20,\\\"max-compaction-bytes\\\":\\\"2GiB\\\",\\\"compaction-pri\\\":3,\\\"dynamic-level-bytes\\\":true,\\\"num-levels\\\":7,\\\"max-bytes-for-level-multiplier\\\":10,\\\"compaction-style\\\":0,\\\"disable-auto-compactions\\\":false,\\\"disable-write-stall\\\":true,\\\"soft-pending-compaction-bytes-limit\\\":\\\"192GiB\\\",\\\"hard-pending-compaction-bytes-limit\\\":\\\"1TiB\\\",\\\"force-consistency-checks\\\":false,\\\"prop-size-index-distance\\\":4194304,\\\"prop-keys-index-distance\\\":40960,\\\"enable-doubly-skiplist\\\":true,\\\"enable-compaction-guard\\\":true,\\\"compaction-guard-min-output-file-size\\\":\\\"8MiB\\\",\\\"compaction-guard-max-output-file-size\\\":\\\"128MiB\\\",\\\"bottommost-level-compression\\\":\\\"zstd\\\",\\\"bottommost-zstd-compression-dict-size\\\":0,\\\"bottommost-zstd-compression-sample-size\\\":0,\\\"prepopulate-block-cache\\\":\\\"disabled\\\",\\\"format-version\\\":null,\\\"checksum\\\":\\\"crc32c\\\",\\\"max-compactions\\\":null,\\\"ttl\\\":null,\\\"periodic-compaction-seconds\\\":null,\\\"titan\\\":{\\\"min-blob-size\\\":\\\"1KiB\\\",\\\"blob-file-compression\\\":\\\"zstd\\\",\\\"zstd-dict-size\\\":\\\"0KiB\\\",\\\"blob-cache-size\\\":\\\"0KiB\\\",\\\"min-gc-batch-size\\\":\\\"16MiB\\\",\\\"max-gc-batch-size\\\":\\\"64MiB\\\",\\\"discardable-ratio\\\":0.5,\\\"merge-small-file-threshold\\\":\\\"8MiB\\\",\\\"blob-run-mode\\\":\\\"read-only\\\",\\\"level-merge\\\":false,\\\"range-merge\\\":true,\\\"max-sorted-runs\\\":20}},\\\"lockcf\\\":{\\\"block-size\\\":\\\"16KiB\\\",\\\"block-cache-size\\\":\\\"128MiB\\\",\\\"disable-block-cache\\\":false,\\\"cache-index-and-filter-blocks\\\":true,\\\"pin-l0-filter-and-index-blocks\\\":true,\\\"use-bloom-filter\\\":true,\\\"optimize-filters-for-hits\\\":false,\\\"optimize-filters-for-memory\\\":false,\\\"whole-key-filtering\\\":true,\\\"bloom-filter-bits-per-key\\\":10,\\\"block-based-bloom-filter\\\":false,\\\"ribbon-filter-above-level\\\":null,\\\"read-amp-bytes-per-bit\\\":0,\\\"compression-per-level\\\":[\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\"],\\\"write-buffer-size\\\":\\\"32MiB\\\",\\\"write-buffer-limit\\\":null,\\\"max-write-buffer-number\\\":5,\\\"min-write-buffer-number-to-merge\\\":1,\\\"max-bytes-for-level-base\\\":\\\"128MiB\\\",\\\"target-file-size-base\\\":null,\\\"level0-file-num-compaction-trigger\\\":1,\\\"level0-slowdown-writes-trigger\\\":20,\\\"level0-stop-writes-trigger\\\":20,\\\"max-compaction-bytes\\\":\\\"2GiB\\\",\\\"compaction-pri\\\":0,\\\"dynamic-level-bytes\\\":true,\\\"num-levels\\\":7,\\\"max-bytes-for-level-multiplier\\\":10,\\\"compaction-style\\\":0,\\\"disable-auto-compactions\\\":false,\\\"disable-write-stall\\\":true,\\\"soft-pending-compaction-bytes-limit\\\":\\\"192GiB\\\",\\\"hard-pending-compaction-bytes-limit\\\":\\\"1TiB\\\",\\\"force-consistency-checks\\\":false,\\\"prop-size-index-distance\\\":4194304,\\\"prop-keys-index-distance\\\":40960,\\\"enable-doubly-skiplist\\\":true,\\\"enable-compaction-guard\\\":null,\\\"compaction-guard-min-output-file-size\\\":\\\"8MiB\\\",\\\"compaction-guard-max-output-file-size\\\":\\\"128MiB\\\",\\\"bottommost-level-compression\\\":\\\"disable\\\",\\\"bottommost-zstd-compression-dict-size\\\":0,\\\"bottommost-zstd-compression-sample-size\\\":0,\\\"prepopulate-block-cache\\\":\\\"disabled\\\",\\\"format-version\\\":null,\\\"checksum\\\":\\\"crc32c\\\",\\\"max-compactions\\\":null,\\\"ttl\\\":null,\\\"periodic-compaction-seconds\\\":null,\\\"titan\\\":{\\\"min-blob-size\\\":\\\"1KiB\\\",\\\"blob-file-compression\\\":\\\"zstd\\\",\\\"zstd-dict-size\\\":\\\"0KiB\\\",\\\"blob-cache-size\\\":\\\"0KiB\\\",\\\"min-gc-batch-size\\\":\\\"16MiB\\\",\\\"max-gc-batch-size\\\":\\\"64MiB\\\",\\\"discardable-ratio\\\":0.5,\\\"merge-small-file-threshold\\\":\\\"8MiB\\\",\\\"blob-run-mode\\\":\\\"read-only\\\",\\\"level-merge\\\":false,\\\"range-merge\\\":true,\\\"max-sorted-runs\\\":20}},\\\"raftcf\\\":{\\\"block-size\\\":\\\"16KiB\\\",\\\"block-cache-size\\\":null,\\\"disable-block-cache\\\":false,\\\"cache-index-and-filter-blocks\\\":true,\\\"pin-l0-filter-and-index-blocks\\\":true,\\\"use-bloom-filter\\\":true,\\\"optimize-filters-for-hits\\\":true,\\\"optimize-filters-for-memory\\\":false,\\\"whole-key-filtering\\\":true,\\\"bloom-filter-bits-per-key\\\":10,\\\"block-based-bloom-filter\\\":false,\\\"ribbon-filter-above-level\\\":null,\\\"read-amp-bytes-per-bit\\\":0,\\\"compression-per-level\\\":[\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\",\\\"no\\\"],\\\"write-buffer-size\\\":\\\"128MiB\\\",\\\"write-buffer-limit\\\":null,\\\"max-write-buffer-number\\\":5,\\\"min-write-buffer-number-to-merge\\\":1,\\\"max-bytes-for-level-base\\\":\\\"128MiB\\\",\\\"target-file-size-base\\\":null,\\\"level0-file-num-compaction-trigger\\\":1,\\\"level0-slowdown-writes-trigger\\\":20,\\\"level0-stop-writes-trigger\\\":20,\\\"max-compaction-bytes\\\":\\\"2GiB\\\",\\\"compaction-pri\\\":0,\\\"dynamic-level-bytes\\\":true,\\\"num-levels\\\":7,\\\"max-bytes-for-level-multiplier\\\":10,\\\"compaction-style\\\":0,\\\"disable-auto-compactions\\\":false,\\\"disable-write-stall\\\":true,\\\"soft-pending-compaction-bytes-limit\\\":\\\"192GiB\\\",\\\"hard-pending-compaction-bytes-limit\\\":\\\"1TiB\\\",\\\"force-consistency-checks\\\":false,\\\"prop-size-index-distance\\\":4194304,\\\"prop-keys-index-distance\\\":40960,\\\"enable-doubly-skiplist\\\":true,\\\"enable-compaction-guard\\\":null,\\\"compaction-guard-min-output-file-size\\\":\\\"8MiB\\\",\\\"compaction-guard-max-output-file-size\\\":\\\"128MiB\\\",\\\"bottommost-level-compression\\\":\\\"disable\\\",\\\"bottommost-zstd-compression-dict-size\\\":0,\\\"bottommost-zstd-compression-sample-size\\\":0,\\\"prepopulate-block-cache\\\":\\\"disabled\\\",\\\"format-version\\\":null,\\\"checksum\\\":\\\"crc32c\\\",\\\"max-compactions\\\":null,\\\"ttl\\\":null,\\\"periodic-compaction-seconds\\\":null,\\\"titan\\\":{\\\"min-blob-size\\\":\\\"1KiB\\\",\\\"blob-file-compression\\\":\\\"zstd\\\",\\\"zstd-dict-size\\\":\\\"0KiB\\\",\\\"blob-cache-size\\\":\\\"0KiB\\\",\\\"min-gc-batch-size\\\":\\\"16MiB\\\",\\\"max-gc-batch-size\\\":\\\"64MiB\\\",\\\"discardable-ratio\\\":0.5,\\\"merge-small-file-threshold\\\":\\\"8MiB\\\",\\\"blob-run-mode\\\":\\\"read-only\\\",\\\"level-merge\\\":false,\\\"range-merge\\\":true,\\\"max-sorted-runs\\\":20}},\\\"titan\\\":{\\\"enabled\\\":false,\\\"dirname\\\":\\\"\\\",\\\"disable-gc\\\":false,\\\"max-background-gc\\\":1,\\\"purge-obsolete-files-period\\\":\\\"10s\\\"}},\\\"raftdb\\\":{\\\"wal-recovery-mode\\\":2,\\\"wal-dir\\\":\\\"\\\",\\\"wal-ttl-seconds\\\":0,\\\"wal-size-limit\\\":\\\"0KiB\\\",\\\"max-total-wal-size\\\":\\\"4GiB\\\",\\\"max-background-jobs\\\":4,\\\"max-background-flushes\\\":1,\\\"max-manifest-file-size\\\":\\\"20MiB\\\",\\\"create-if-missing\\\":true,\\\"max-open-files\\\":40960,\\\"stats-dump-period\\\":\\\"10m\\\",\\\"compaction-readahead-size\\\":\\\"0KiB\\\",\\\"info-log-max-size\\\":\\\"1GiB\\\",\\\"info-log-roll-time\\\":\\\"0s\\\",\\\"info-log-keep-log-file-num\\\":10,\\\"info-log-dir\\\":\\\"\\\",\\\"info-log-level\\\":\\\"info\\\",\\\"max-sub-compactions\\\":2,\\\"writable-file-max-buffer-size\\\":\\\"1MiB\\\",\\\"use-direct-io-for-flush-and-compaction\\\":false,\\\"enable-pipelined-write\\\":true,\\\"enable-unordered-write\\\":false,\\\"allow-concurrent-memtable-write\\\":true,\\\"bytes-per-sync\\\":\\\"1MiB\\\",\\\"wal-bytes-per-sync\\\":\\\"512KiB\\\",\\\"defaultcf\\\":{\\\"block-size\\\":\\\"64KiB\\\",\\\"block-cache-size\\\":\\\"12892MiB\\\",\\\"disable-block-cache\\\":false,\\\"cache-index-and-filter-blocks\\\":true,\\\"pin-l0-filter-and-index-blocks\\\":true,\\\"use-bloom-filter\\\":false,\\\"optimize-filters-for-hits\\\":true,\\\"optimize-filters-for-memory\\\":false,\\\"whole-key-filtering\\\":true,\\\"bloom-filter-bits-per-key\\\":10,\\\"block-based-bloom-filter\\\":false,\\\"ribbon-filter-above-level\\\":null,\\\"read-amp-bytes-per-bit\\\":0,\\\"compression-per-level\\\":[\\\"no\\\",\\\"no\\\",\\\"lz4\\\",\\\"lz4\\\",\\\"lz4\\\",\\\"zstd\\\",\\\"zstd\\\"],\\\"write-buffer-size\\\":\\\"128MiB\\\",\\\"write-buffer-limit\\\":null,\\\"max-write-buffer-number\\\":5,\\\"min-write-buffer-number-to-merge\\\":1,\\\"max-bytes-for-level-base\\\":\\\"512MiB\\\",\\\"target-file-size-base\\\":null,\\\"level0-file-num-compaction-trigger\\\":4,\\\"level0-slowdown-writes-trigger\\\":20,\\\"level0-stop-writes-trigger\\\":20,\\\"max-compaction-bytes\\\":\\\"2GiB\\\",\\\"compaction-pri\\\":0,\\\"dynamic-level-bytes\\\":true,\\\"num-levels\\\":7,\\\"max-bytes-for-level-multiplier\\\":10,\\\"compaction-style\\\":0,\\\"disable-auto-compactions\\\":false,\\\"disable-write-stall\\\":false,\\\"soft-pending-compaction-bytes-limit\\\":\\\"192GiB\\\",\\\"hard-pending-compaction-bytes-limit\\\":\\\"1TiB\\\",\\\"force-consistency-checks\\\":false,\\\"prop-size-index-distance\\\":4194304,\\\"prop-keys-index-distance\\\":40960,\\\"enable-doubly-skiplist\\\":true,\\\"enable-compaction-guard\\\":null,\\\"compaction-guard-min-output-file-size\\\":\\\"8MiB\\\",\\\"compaction-guard-max-output-file-size\\\":\\\"128MiB\\\",\\\"bottommost-level-compression\\\":\\\"disable\\\",\\\"bottommost-zstd-compression-dict-size\\\":0,\\\"bottommost-zstd-compression-sample-size\\\":0,\\\"prepopulate-block-cache\\\":\\\"disabled\\\",\\\"format-version\\\":2,\\\"checksum\\\":\\\"crc32c\\\",\\\"max-compactions\\\":null,\\\"ttl\\\":null,\\\"periodic-compaction-seconds\\\":null,\\\"titan\\\":{\\\"min-blob-size\\\":\\\"1KiB\\\",\\\"blob-file-compression\\\":\\\"zstd\\\",\\\"zstd-dict-size\\\":\\\"0KiB\\\",\\\"blob-cache-size\\\":\\\"0KiB\\\",\\\"min-gc-batch-size\\\":\\\"16MiB\\\",\\\"max-gc-batch-size\\\":\\\"64MiB\\\",\\\"discardable-ratio\\\":0.5,\\\"merge-small-file-threshold\\\":\\\"8MiB\\\",\\\"blob-run-mode\\\":\\\"normal\\\",\\\"level-merge\\\":false,\\\"range-merge\\\":true,\\\"max-sorted-runs\\\":20}},\\\"titan\\\":{\\\"enabled\\\":false,\\\"dirname\\\":\\\"\\\",\\\"disable-gc\\\":false,\\\"max-background-gc\\\":4,\\\"purge-obsolete-files-period\\\":\\\"10s\\\"}},\\\"raft-engine\\\":{\\\"enable\\\":true,\\\"dir\\\":\\\"/data0/proxy/raft-engine\\\",\\\"spill-dir\\\":null,\\\"recovery-mode\\\":\\\"tolerate-corrupted-tail-records\\\",\\\"recovery-read-block-size\\\":\\\"16KiB\\\",\\\"recovery-threads\\\":4,\\\"batch-compression-threshold\\\":\\\"8KiB\\\",\\\"compression-level\\\":null,\\\"bytes-per-sync\\\":null,\\\"format-version\\\":2,\\\"target-file-size\\\":\\\"128MiB\\\",\\\"purge-threshold\\\":\\\"10GiB\\\",\\\"purge-rewrite-threshold\\\":\\\"1GiB\\\",\\\"purge-rewrite-garbage-ratio\\\":0.6,\\\"memory-limit\\\":\\\"40554880819B\\\",\\\"enable-log-recycle\\\":true,\\\"prefill-for-recycle\\\":false,\\\"prefill-limit\\\":null},\\\"security\\\":{\\\"ca-path\\\":\\\"\\\",\\\"cert-path\\\":\\\"\\\",\\\"key-path\\\":\\\"\\\",\\\"cert-allowed-cn\\\":[],\\\"redact-info-log\\\":null,\\\"encryption\\\":{\\\"data-encryption-method\\\":\\\"plaintext\\\",\\\"data-key-rotation-period\\\":\\\"7d\\\",\\\"enable-file-dictionary-log\\\":true,\\\"file-dictionary-rewrite-threshold\\\":1000000,\\\"master-key\\\":{\\\"type\\\":\\\"plaintext\\\"},\\\"previous-master-key\\\":{\\\"type\\\":\\\"plaintext\\\"}}},\\\"import\\\":{\\\"num-threads\\\":80,\\\"stream-channel-window\\\":128,\\\"import-mode-timeout\\\":\\\"10m\\\",\\\"memory-use-ratio\\\":0.3},\\\"backup\\\":{\\\"num-threads\\\":8,\\\"batch-size\\\":8,\\\"sst-max-size\\\":\\\"144MiB\\\",\\\"enable-auto-tune\\\":true,\\\"auto-tune-remain-threads\\\":8,\\\"auto-tune-refresh-interval\\\":\\\"1m\\\",\\\"io-thread-size\\\":2,\\\"s3-multi-part-size\\\":\\\"5MiB\\\",\\\"hadoop\\\":{\\\"home\\\":\\\"\\\",\\\"linux-user\\\":\\\"\\\"}},\\\"log-backup\\\":{\\\"min-ts-interval\\\":\\\"10s\\\",\\\"max-flush-interval\\\":\\\"3m\\\",\\\"num-threads\\\":12,\\\"enable\\\":true,\\\"temp-path\\\":\\\"/data0/proxy/log-backup-temp\\\",\\\"file-size-limit\\\":\\\"256MiB\\\",\\\"initial-scan-pending-memory-quota\\\":\\\"512MiB\\\",\\\"initial-scan-rate-limit\\\":\\\"60MiB\\\",\\\"initial-scan-concurrency\\\":6},\\\"pessimistic-txn\\\":{\\\"wait-for-lock-timeout\\\":\\\"1s\\\",\\\"wake-up-delay-duration\\\":\\\"20ms\\\",\\\"pipelined\\\":true,\\\"in-memory\\\":true},\\\"gc\\\":{\\\"ratio-threshold\\\":1.1,\\\"batch-keys\\\":512,\\\"max-write-bytes-per-sec\\\":\\\"0KiB\\\",\\\"enable-compaction-filter\\\":true,\\\"compaction-filter-skip-version-check\\\":false,\\\"num-threads\\\":1},\\\"split\\\":{\\\"qps-threshold\\\":3000,\\\"split-balance-score\\\":0.25,\\\"split-contained-score\\\":0.5,\\\"detect-times\\\":10,\\\"sample-num\\\":20,\\\"sample-threshold\\\":100,\\\"byte-threshold\\\":31457280,\\\"grpc-thread-cpu-overload-threshold-ratio\\\":0.5,\\\"unified-read-pool-thread-cpu-overload-threshold-ratio\\\":0.8,\\\"region-cpu-overload-threshold-ratio\\\":0.25},\\\"cdc\\\":{\\\"min-ts-interval\\\":\\\"1s\\\",\\\"hibernate-regions-compatible\\\":true,\\\"incremental-scan-threads\\\":4,\\\"incremental-scan-concurrency\\\":6,\\\"incremental-scan-concurrency-limit\\\":10000,\\\"incremental-scan-speed-limit\\\":\\\"128MiB\\\",\\\"incremental-fetch-speed-limit\\\":\\\"512MiB\\\",\\\"incremental-scan-ts-filter-ratio\\\":0.2,\\\"tso-worker-threads\\\":1,\\\"sink-memory-quota\\\":\\\"512MiB\\\",\\\"old-value-cache-memory-quota\\\":\\\"512MiB\\\"},\\\"resolved-ts\\\":{\\\"enable\\\":true,\\\"advance-ts-interval\\\":\\\"20s\\\",\\\"scan-lock-pool-size\\\":2,\\\"memory-quota\\\":\\\"256MiB\\\",\\\"incremental-scan-concurrency\\\":6},\\\"resource-metering\\\":{\\\"receiver-address\\\":\\\"\\\",\\\"report-receiver-interval\\\":\\\"1m\\\",\\\"max-resource-groups\\\":100,\\\"precision\\\":\\\"1s\\\"},\\\"causal-ts\\\":{\\\"renew-interval\\\":\\\"100ms\\\",\\\"renew-batch-min-size\\\":100,\\\"renew-batch-max-size\\\":8192,\\\"alloc-ahead-buffer\\\":\\\"3s\\\"},\\\"resource-control\\\":{\\\"enabled\\\":true}}\"] [thread_id=0x4]", "[2024/11/01 01:20:56.270 +00:00] [INFO] [resource_group.rs:145] [\"add resource group\"] [ru=2147483647] [name=default] [thread_id=0x4]", "[2024/11/01 01:20:56.270 +00:00] [INFO] [resource_group.rs:145] [\"add resource group\"] [ru=2147483647] [name=default] [thread_id=0x4]", "[2024/11/01 01:20:56.270 +00:00] [INFO] [client.rs:413] [\"[global_config] start watch global config\"] [revision=1244] [path=resource_group/settings] [thread_id=0x4]", "[2024/11/01 01:20:58.701 +00:00] [INFO] [mod.rs:130] [\"encryption: none of key dictionary and file dictionary are found.\"] [thread_id=0x4]", "[2024/11/01 01:20:58.701 +00:00] [INFO] [mod.rs:549] [\"encryption is disabled.\"] [thread_id=0x4]", "[2024/11/01 01:20:58.701 +00:00] [INFO] [run.rs:157] [\"start probing cluster's raftstore version\"] [thread_id=0x4]", "[2024/11/01 01:20:58.753 +00:00] [INFO] [run.rs:160] [\"cluster's raftstore version is V1\"] [thread_id=0x4]", "[2024/11/01 01:20:58.753 +00:00] [INFO] [run.rs:172] [\"set raft-store proxy helper\"] [thread_id=0x4]", "[2024/11/01 01:20:58.753 +00:00] [INFO] [run.rs:176] [\"wait for engine-store server to start\"] [thread_id=0x4]", "[2024/11/01 01:20:59.153 +00:00] [INFO] [run.rs:190] [\"engine-store server is started\"] [thread_id=0x4]", "[2024/11/01 01:20:59.153 +00:00] [WARN] [config.rs:323] [\"Memory allocator nodump is not supported, continue with default allocator\"] [thread_id=0x4]", "[2024/11/01 01:20:59.154 +00:00] [INFO] [engine.rs:93] [\"Recovering raft logs takes 867.006\u00b5s\"] [thread_id=0x4]", "[2024/11/01 01:20:59.196 +00:00] [INFO] [engine.rs:80] [\"disabled pagestorage\"] [thread_id=0x4]", "[2024/11/01 01:20:59.343 +00:00] [INFO] [mod.rs:299] [\"Storage started.\"] [thread_id=0x4]", "[2024/11/01 01:20:59.345 +00:00] [INFO] [run.rs:1162] [\"set store 6 to engine-store\"] [thread_id=0x4]", "[2024/11/01 01:20:59.346 +00:00] [INFO] [sst_importer.rs:203] [\"sst importer memory limit when apply\"] [size=81109761638] [ratio=0.3] [thread_id=0x4]", "[2024/11/01 01:20:59.354 +00:00] [INFO] [forwarder.rs:107] [\"created tiflash forwarder\"] [store_id=6] [thread_id=0x4]", "[2024/11/01 01:20:59.355 +00:00] [INFO] [node.rs:197] [\"put store to PD\"] [store=\"id: 6 address: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:3930\\\" labels { key: \\\"engine\\\" value: \\\"tiflash\\\" } version: \\\"v8.1.0\\\" peer_address: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20170\\\" status_address: \\\"test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc:20292\\\" git_hash: \\\"c1838001167c8ba83af759085a71ad61e6c2a5af\\\" start_timestamp: 1730424059 deploy_path: \\\"/tiflash\\\"\"] [thread_id=0x4]", "[2024/11/01 01:20:59.356 +00:00] [FATAL] [run.rs:1337] [\"failed to start node: StoreTombstone(\\\"store is tombstone\\\")\"] [thread_id=0x4]" ```

Here is the dump of the TiDBCluster CR Status:

TiDBCluster CR Status ``` "custom_resource_status": { "clusterID": "7432111146104685610", "conditions": [ { "lastTransitionTime": "2024-11-01T01:12:51Z", "lastUpdateTime": "2024-11-01T01:12:51Z", "message": "TiDB cluster is fully up and running", "reason": "Ready", "status": "True", "type": "Ready" } ], "pd": { "image": "pingcap/pd:v8.1.0", "leader": { "clientURL": "http://test-cluster-pd-0.test-cluster-pd-peer.acto-namespace.svc:2379", "health": true, "id": "4767470187507236263", "lastTransitionTime": "2024-11-01T01:07:04Z", "name": "test-cluster-pd-0" }, "members": { "test-cluster-pd-0": { "clientURL": "http://test-cluster-pd-0.test-cluster-pd-peer.acto-namespace.svc:2379", "health": true, "id": "4767470187507236263", "lastTransitionTime": "2024-11-01T01:07:04Z", "name": "test-cluster-pd-0" }, "test-cluster-pd-1": { "clientURL": "http://test-cluster-pd-1.test-cluster-pd-peer.acto-namespace.svc:2379", "health": true, "id": "5912562467187867209", "lastTransitionTime": "2024-11-01T01:07:10Z", "name": "test-cluster-pd-1" }, "test-cluster-pd-2": { "clientURL": "http://test-cluster-pd-2.test-cluster-pd-peer.acto-namespace.svc:2379", "health": true, "id": "10348874404322055688", "lastTransitionTime": "2024-11-01T01:07:12Z", "name": "test-cluster-pd-2" } }, "phase": "Normal", "statefulSet": { "availableReplicas": 3, "collisionCount": 0, "currentReplicas": 3, "currentRevision": "test-cluster-pd-6d575dd9d5", "observedGeneration": 1, "readyReplicas": 3, "replicas": 3, "updateRevision": "test-cluster-pd-6d575dd9d5", "updatedReplicas": 3 }, "synced": true, "volumes": { "pd": { "boundCount": 3, "currentCapacity": "10Gi", "currentCount": 3, "currentStorageClass": "standard", "modifiedCapacity": "10Gi", "modifiedCount": 3, "modifiedStorageClass": "standard", "name": "pd", "resizedCapacity": "10Gi", "resizedCount": 3 } } }, "pump": {}, "ticdc": { "captures": { "test-cluster-ticdc-0": { "id": "57711d9e-21ee-455e-ab60-b7c4095b7d4f", "podName": "test-cluster-ticdc-0", "ready": true, "version": "v8.1.0" }, "test-cluster-ticdc-1": { "id": "0afafa3c-c438-45d0-aa56-d8d526df8720", "isOwner": true, "podName": "test-cluster-ticdc-1", "ready": true, "version": "v8.1.0" }, "test-cluster-ticdc-2": { "id": "bb7a1540-9c78-471c-8561-4262cdb53938", "podName": "test-cluster-ticdc-2", "ready": true, "version": "v8.1.0" } }, "phase": "Normal", "statefulSet": { "availableReplicas": 3, "collisionCount": 0, "currentReplicas": 3, "currentRevision": "test-cluster-ticdc-7ffddb8864", "observedGeneration": 1, "readyReplicas": 3, "replicas": 3, "updateRevision": "test-cluster-ticdc-7ffddb8864", "updatedReplicas": 3 }, "synced": true }, "tidb": { "image": "pingcap/tidb:v8.1.0", "members": { "test-cluster-tidb-0": { "health": true, "lastTransitionTime": "2024-11-01T01:07:51Z", "name": "test-cluster-tidb-0", "node": "acto-0-cluster-1-worker4" }, "test-cluster-tidb-1": { "health": true, "lastTransitionTime": "2024-11-01T01:07:51Z", "name": "test-cluster-tidb-1", "node": "acto-0-cluster-1-worker" }, "test-cluster-tidb-2": { "health": true, "lastTransitionTime": "2024-11-01T01:07:51Z", "name": "test-cluster-tidb-2", "node": "acto-0-cluster-1-worker3" } }, "phase": "Normal", "statefulSet": { "availableReplicas": 3, "collisionCount": 0, "currentReplicas": 3, "currentRevision": "test-cluster-tidb-7cb86478fc", "observedGeneration": 1, "readyReplicas": 3, "replicas": 3, "updateRevision": "test-cluster-tidb-7cb86478fc", "updatedReplicas": 3 } }, "tiflash": { "image": "pingcap/tiflash:v8.1.0", "phase": "Normal", "statefulSet": { "availableReplicas": 0, "collisionCount": 0, "currentRevision": "test-cluster-tiflash-6f88f87d7", "observedGeneration": 4, "replicas": 0, "updateRevision": "test-cluster-tiflash-6f88f87d7" }, "synced": true, "tombstoneStores": { "15": { "id": "15", "ip": "test-cluster-tiflash-1.test-cluster-tiflash-peer.acto-namespace.svc", "lastTransitionTime": "2024-11-01T01:12:40Z", "leaderCount": 0, "podName": "test-cluster-tiflash-1", "state": "Tombstone" }, "4": { "id": "4", "ip": "test-cluster-tiflash-2.test-cluster-tiflash-peer.acto-namespace.svc", "lastTransitionTime": "2024-11-01T01:12:20Z", "leaderCount": 0, "podName": "test-cluster-tiflash-2", "state": "Tombstone" }, "6": { "id": "6", "ip": "test-cluster-tiflash-0.test-cluster-tiflash-peer.acto-namespace.svc", "lastTransitionTime": "2024-11-01T01:12:50Z", "leaderCount": 0, "podName": "test-cluster-tiflash-0", "state": "Tombstone" } } }, "tikv": { "bootStrapped": true, "conditions": [ { "lastTransitionTime": "2024-11-01T01:07:47Z", "message": "Leader can be scheduled to all nodes", "reason": "NoLeaderEviction", "status": "False", "type": "LeaderEvicting" } ], "image": "pingcap/tikv:v8.1.0", "phase": "Normal", "statefulSet": { "availableReplicas": 3, "collisionCount": 0, "currentReplicas": 3, "currentRevision": "test-cluster-tikv-546854fb45", "observedGeneration": 1, "readyReplicas": 3, "replicas": 3, "updateRevision": "test-cluster-tikv-546854fb45", "updatedReplicas": 3 }, "stores": { "1": { "id": "1", "ip": "test-cluster-tikv-0.test-cluster-tikv-peer.acto-namespace.svc", "lastTransitionTime": "2024-11-01T01:07:45Z", "leaderCount": 18, "podName": "test-cluster-tikv-0", "state": "Up" }, "16": { "id": "16", "ip": "test-cluster-tikv-1.test-cluster-tikv-peer.acto-namespace.svc", "lastTransitionTime": "2024-11-01T01:07:45Z", "leaderCount": 18, "podName": "test-cluster-tikv-1", "state": "Up" }, "5": { "id": "5", "ip": "test-cluster-tikv-2.test-cluster-tikv-peer.acto-namespace.svc", "lastTransitionTime": "2024-11-01T01:07:45Z", "leaderCount": 27, "podName": "test-cluster-tikv-2", "state": "Up" } }, "synced": true, "volumes": { "tikv": { "boundCount": 3, "currentCapacity": "100Gi", "currentCount": 3, "currentStorageClass": "standard", "modifiedCapacity": "100Gi", "modifiedCount": 3, "modifiedStorageClass": "standard", "name": "tikv", "resizedCapacity": "100Gi", "resizedCount": 3 } } }, "tiproxy": {} } ```
csuzhangxc commented 3 weeks ago

it seems we need to use a new PV (delete the PVC/PV after scaled in to 0) without the data for the previous Store