Closed def- closed 4 weeks ago
This is actually much easier, doesn't even require concurrency:
drop cluster c cascade;
create cluster c REPLICAS(r1 (size = '2'), r2 (size = '2'));
drop cluster replica c.r1;
create source counter in cluster c from load generator counter;
SELECT count(*) from counter;
Hangs already. Why does r2 not work anymore after dropping r1?
materialize=> select * from mz_clusters where name = 'c';
id | name | owner_id | privileges | managed | size | replication_factor | disk | availability_zones | introspection_debugging | introspection_interval
-----+------+----------+--------------------+---------+------+--------------------+------+--------------------+-------------------------+------------------------
u10 | c | s1 | {s1=UC/s1,s2=U/s1} | f | | | | | |
(1 row)
materialize=> select * from mz_cluster_replicas where cluster_id = 'u10';
id | name | cluster_id | size | availability_zone | owner_id | disk
-----+------+------------+------+-------------------+----------+------
u18 | r2 | u10 | 2 | | s1 | f
(1 row)
materialize=> select * from mz_internal.mz_source_statuses;
id | name | type | last_status_change_at | status | error | details
-----+------------------+----------------+----------------------------+---------+-------+-----------------------------------------------------------------
u14 | counter_progress | progress | | running | |
u15 | counter | load-generator | 2024-07-11 10:33:18.778+00 | paused | | {"hints":["There is currently no replica running this source"]}
(2 rows)
Logs:
environmentd: 2024-07-11T10:33:16.996805Z INFO coord::handle_message{kind="command-execute"}:message_command:coord::handle_execute{session="68f4f7f9-c9f8-4e0b-bc6f-d821a7ca2ad5"}:coord::handle_execute_inner{stmt="CREATE CLUSTER c (REPLICAS = (r1 (SIZE = '2'), r2 (SIZE = '2')))"}:coord::catalog_transact:coord::catalog_transact_conn:coord::catalog_transact_inner:catalog::transact:catalog::transact_inner:transact_op: mz_adapter::catalog::transact: create cluster c
environmentd: 2024-07-11T10:33:17.361211Z INFO mz_compute_client::controller::replica: starting replica task replica=User(17)
environmentd: 2024-07-11T10:33:17.361272Z INFO coord::handle_message{kind="command-execute"}:message_command:coord::handle_execute{session="68f4f7f9-c9f8-4e0b-bc6f-d821a7ca2ad5"}:coord::handle_execute_inner{stmt="CREATE CLUSTER c (REPLICAS = (r1 (SIZE = '2'), r2 (SIZE = '2')))"}: mz_adapter::coord::introspection: installing introspection subscribe id=t107 replica_id=u17 type_=ComputeErrorCounts
environmentd: 2024-07-11T10:33:17.361343Z INFO mz_orchestrator_process: launching cluster-u10-replica-u17-gen-0-0 via /usr/local/bin/clusterd --storage-controller-listen-addr=/tmp/1a380ce41d0b07d4f943a66a2d61bcbe9a4711b4 --compute-controller-listen-addr=/tmp/1c79659f469530488f8703a82923f658779f9a1f --internal-http-listen-addr=/tmp/e6e2c69c0194b8eb01739bd838681f86a68d0fda --opentelemetry-resource=cluster_id=u10 --opentelemetry-resource=replica_id=u17 --persist-pubsub-url=http://localhost:6879 --persist-txn-tables=eager --environment-id=docker-container-f6159cb4-1c6c-4f25-a5e7-4c7635633d43-0 --secrets-reader=local-file --secrets-reader-local-file-dir=/mzdata/secrets --startup-log-filter=info --log-format=text --log-prefix=cluster-u10-replica-u17-gen-0...
environmentd: 2024-07-11T10:33:17.361548Z INFO mz_compute_client::controller::replica: starting replica task replica=User(18)
environmentd: 2024-07-11T10:33:17.361596Z INFO coord::handle_message{kind="command-execute"}:message_command:coord::handle_execute{session="68f4f7f9-c9f8-4e0b-bc6f-d821a7ca2ad5"}:coord::handle_execute_inner{stmt="CREATE CLUSTER c (REPLICAS = (r1 (SIZE = '2'), r2 (SIZE = '2')))"}: mz_adapter::coord::introspection: installing introspection subscribe id=t109 replica_id=u18 type_=ComputeErrorCounts
environmentd: 2024-07-11T10:33:17.361656Z INFO mz_orchestrator_process: launching cluster-u10-replica-u18-gen-0-0 via /usr/local/bin/clusterd --storage-controller-listen-addr=/tmp/a4e2575c53c41e710608bb7714e670742416b19a --compute-controller-listen-addr=/tmp/b9c9b50c3ab63d2849717cf590113c89eb821d73 --internal-http-listen-addr=/tmp/427e9e7ee1b41534647fc4c712ae17a5e5e0f6f4 --opentelemetry-resource=cluster_id=u10 --opentelemetry-resource=replica_id=u18 --persist-pubsub-url=http://localhost:6879 --persist-txn-tables=eager --environment-id=docker-container-f6159cb4-1c6c-4f25-a5e7-4c7635633d43-0 --secrets-reader=local-file --secrets-reader-local-file-dir=/mzdata/secrets --startup-log-filter=info --log-format=text --log-prefix=cluster-u10-replica-u18-gen-0...
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.390930Z INFO clusterd: booting os.os_type=Ubuntu os.version=22.04 os.bitness=64-bit build.version="0.107.2" build.sha="12c99b30eed467f5112beab6a39d8a47f2175c7c" build.time="2024-07-07T18:37:01Z" cpus.logical=16 cpus.physical=8 cpu0.brand="AMD Ryzen 7 7700 8-Core Processor" cpu0.frequency=5282 memory.total=134686658560 memory.used=15079116800 memory.limit=<unknown> swap.total=0 swap.used=0 swap.limit=<unknown> tracing.max_level=info
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.391130Z INFO clusterd: serving internal HTTP server on /tmp/e6e2c69c0194b8eb01739bd838681f86a68d0fda
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.391462Z INFO clusterd: booting os.os_type=Ubuntu os.version=22.04 os.bitness=64-bit build.version="0.107.2" build.sha="12c99b30eed467f5112beab6a39d8a47f2175c7c" build.time="2024-07-07T18:37:01Z" cpus.logical=16 cpus.physical=8 cpu0.brand="AMD Ryzen 7 7700 8-Core Processor" cpu0.frequency=5282 memory.total=134686658560 memory.used=15079116800 memory.limit=<unknown> swap.total=0 swap.used=0 swap.limit=<unknown> tracing.max_level=info
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.391610Z INFO clusterd: serving internal HTTP server on /tmp/427e9e7ee1b41534647fc4c712ae17a5e5e0f6f4
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.394290Z INFO clusterd: listening for storage controller connections on /tmp/1a380ce41d0b07d4f943a66a2d61bcbe9a4711b4
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.394308Z INFO mz_service::grpc: Starting to listen on /tmp/1a380ce41d0b07d4f943a66a2d61bcbe9a4711b4
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.394382Z INFO clusterd: listening for compute controller connections on /tmp/1c79659f469530488f8703a82923f658779f9a1f
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.394388Z INFO mz_service::grpc: Starting to listen on /tmp/1c79659f469530488f8703a82923f658779f9a1f
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.395904Z INFO clusterd: listening for storage controller connections on /tmp/a4e2575c53c41e710608bb7714e670742416b19a
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.395916Z INFO mz_service::grpc: Starting to listen on /tmp/a4e2575c53c41e710608bb7714e670742416b19a
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.395964Z INFO clusterd: listening for compute controller connections on /tmp/b9c9b50c3ab63d2849717cf590113c89eb821d73
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.395982Z INFO mz_service::grpc: Starting to listen on /tmp/b9c9b50c3ab63d2849717cf590113c89eb821d73
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.487520Z INFO mz_service::grpc: GrpcServer: remote client connected
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.487519Z INFO mz_service::grpc: GrpcServer: remote client connected
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.487519Z INFO mz_service::grpc: GrpcServer: remote client connected
environmentd: 2024-07-11T10:33:17.487645Z INFO mz_service::grpc: GrpcClient /tmp/a4e2575c53c41e710608bb7714e670742416b19a: connected
environmentd: 2024-07-11T10:33:17.487645Z INFO mz_service::grpc: GrpcClient /tmp/1c79659f469530488f8703a82923f658779f9a1f: connected
environmentd: 2024-07-11T10:33:17.487645Z INFO mz_service::grpc: GrpcClient /tmp/b9c9b50c3ab63d2849717cf590113c89eb821d73: connected
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.487827Z INFO mz_cluster::server: Building timely container with config TimelyConfig { workers: 2, process: 0, addresses: ["/tmp/16d6aaf7659a684aed60f19644b2f900bc72a331"], arrangement_exert_proportionality: 16 }
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.487838Z INFO mz_cluster::server: Building timely container with config TimelyConfig { workers: 2, process: 0, addresses: ["/tmp/941f58cabfa7ec8a2f02c474b6135eb660891a84"], arrangement_exert_proportionality: 16 }
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.487851Z INFO mz_cluster::communication: initializing network for timely instance, with 1 processes for epoch number (2, 1) process=0 addresses=["/tmp/16d6aaf7659a684aed60f19644b2f900bc72a331"]
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.487858Z INFO mz_cluster::communication: initializing network for timely instance, with 1 processes for epoch number (2, 1) process=0 addresses=["/tmp/941f58cabfa7ec8a2f02c474b6135eb660891a84"]
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.488222Z INFO mz_cluster::communication: successfully initialized network process=0
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488234Z INFO mz_cluster::server: Building timely container with config TimelyConfig { workers: 2, process: 0, addresses: ["/tmp/d64d6a5bf362a654fb011ac1210a197f186cec69"], arrangement_exert_proportionality: 1337 }
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488247Z INFO mz_cluster::communication: initializing network for timely instance, with 1 processes for epoch number (2, 1) process=0 addresses=["/tmp/d64d6a5bf362a654fb011ac1210a197f186cec69"]
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488293Z INFO mz_cluster::communication: successfully initialized network process=0
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488558Z INFO mz_cluster::communication: successfully initialized network process=0
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488944Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488946Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488950Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.488950Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.488982Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.488983Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.488988Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.488987Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489168Z INFO mz_storage::storage_state: Applying configuration update: StorageParameters { pg_source_connect_timeout: Some(30s), pg_source_tcp_keepalives_retries: Some(5), pg_source_tcp_keepalives_idle: Some(10s), pg_source_tcp_keepalives_interval: Some(10s), pg_source_tcp_user_timeout: Some(60s), pg_source_tcp_configure_server: false, pg_source_snapshot_statement_timeout: 0ns, pg_source_wal_sender_timeout: None, mysql_source_timeouts: TimeoutConfig { snapshot_max_execution_time: Some(0ns), snapshot_lock_wait_timeout: Some(3600s), tcp_keepalive: Some(60s) }, keep_n_source_status_history_entries: 5, keep_n_sink_status_history_entries: 5, keep_n_privatelink_status_history_entries: 5, upsert_rocksdb_tuning_config: RocksDBTuningParameters { compaction_style: Level, optimize_compaction_memtable_budget: 178956970, level_compaction_dynamic_level_bytes: true, universal_compaction_target_ratio: 200, parallelism: None, compression_type: Lz4, bottommost_compression_type: Lz4, batch_size: 20480, retry_max_duration: 1s, stats_log_interval_seconds: 600, stats_persist_interval_seconds: 600, point_lookup_block_cache_size_mb: None, shrink_buffers_by_ratio: 0, write_buffer_manager_memory_bytes: None, write_buffer_manager_memory_fraction: None, write_buffer_manager_allow_stall: false }, finalize_shards: true, tracing: TracingParameters { log_filter: Some(info), opentelemetry_filter: Some(info), log_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })], opentelemetry_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("h2"), level: LevelFilter::OFF }), SerializableDirective(Directive { in_span: None, fields: [], target: Some("hyper"), level: LevelFilter::OFF })], sentry_filters: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })] }, upsert_auto_spill_config: UpsertAutoSpillConfig { allow_spilling_to_disk: false, spill_to_disk_threshold_bytes: 89478484 }, storage_dataflow_max_inflight_bytes_config: StorageMaxInflightBytesConfig { max_inflight_bytes_default: Some(268435456), max_inflight_bytes_cluster_size_fraction: Some(0.0025), disk_only: true }, grpc_client: GrpcClientParameters { connect_timeout: Some(5s), http2_keep_alive_interval: Some(3s), http2_keep_alive_timeout: Some(5s) }, shrink_upsert_unused_buffers_by_ratio: 0, record_namespaced_errors: true, ssh_timeout_config: SshTimeoutConfig { check_interval: 30s, connect_timeout: 30s, keepalives_idle: 10s }, kafka_timeout_config: TimeoutConfig { keepalive: true, socket_timeout: 60s, transaction_timeout: 60s, socket_connection_setup_timeout: 30s, fetch_metadata_timeout: 10s, progress_record_fetch_timeout: 90s, default_metadata_fetch_interval: 60s }, statistics_interval: 60s, statistics_collection_interval: 10s, pg_snapshot_config: PgSourceSnapshotConfig { collect_strict_count: true, fallback_to_strict_count: true, wait_for_count: true }, user_storage_managed_collections_batch_duration: 1s, dyncfg_updates: ConfigUpdates { updates: {"allow_user_sessions": ProtoConfigVal { val: Some(Bool(true)) }, "compute_dataflow_max_inflight_bytes": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_dataflow_max_inflight_bytes_cc": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_hydration_concurrency": ProtoConfigVal { val: Some(Usize(4)) }, "compute_persist_sink_obey_read_only": ProtoConfigVal { val: Some(Bool(true)) }, "controller_past_generation_replica_cleanup_retry_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "copy_to_s3_arrow_builder_buffer_ratio": ProtoConfigVal { val: Some(Usize(150)) }, "copy_to_s3_multipart_part_size_bytes": ProtoConfigVal { val: Some(Usize(8388608)) }, "copy_to_s3_parquet_row_group_file_ratio": ProtoConfigVal { val: Some(Usize(20)) }, "crdb_connect_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 5, nanos: 0 })) }, "crdb_tcp_user_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "enable_0dt_deployment": ProtoConfigVal { val: Some(Bool(false)) }, "enable_columnation_lgalloc": ProtoConfigVal { val: Some(Bool(false)) }, "enable_compute_chunked_stack": ProtoConfigVal { val: Some(Bool(false)) }, "enable_introspection_subscribes": ProtoConfigVal { val: Some(Bool(true)) }, "enable_lgalloc_eager_reclamation": ProtoConfigVal { val: Some(Bool(true)) }, "enable_mz_join_core": ProtoConfigVal { val: Some(Bool(true)) }, "enable_statement_lifecycle_logging": ProtoConfigVal { val: Some(Bool(false)) }, "kafka_client_id_enrichment_rules": ProtoConfigVal { val: Some(Json("[]")) }, "kafka_poll_max_wait": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_background_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_slow_clear_bytes": ProtoConfigVal { val: Some(Usize(33554432)) }, "linear_join_yielding": ProtoConfigVal { val: Some(String("work:1000000,time:100")) }, "mysql_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "mysql_replication_heartbeat_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "persist_batch_columnar_format": ProtoConfigVal { val: Some(String("row")) }, "persist_batch_delete_enabled": ProtoConfigVal { val: Some(Bool(false)) }, "persist_batch_record_part_format": ProtoConfigVal { val: Some(Bool(false)) }, "persist_blob_cache_mem_limit_bytes": ProtoConfigVal { val: Some(Usize(1048576)) }, "persist_blob_target_size": ProtoConfigVal { val: Some(Usize(134217728)) }, "persist_claim_unclaimed_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_compaction_minimum_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 90, nanos: 0 })) }, "persist_consensus_connection_pool_ttl": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "persist_consensus_connection_pool_ttl_stagger": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 6, nanos: 0 })) }, "persist_enable_arrow_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_arrow_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_enable_s3_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_s3_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_fetch_semaphore_cost_adjustment": ProtoConfigVal { val: Some(F64(1.2)) }, "persist_fetch_semaphore_permit_adjustment": ProtoConfigVal { val: Some(F64(1.0)) }, "persist_inline_writes_single_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_inline_writes_total_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_next_listen_batch_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "persist_next_listen_batch_retryer_fixed_sleep": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 0 })) }, "persist_next_listen_batch_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 200000000 })) }, "persist_next_listen_batch_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_optimize_ignored_data_decode": ProtoConfigVal { val: Some(Bool(true)) }, "persist_optimize_ignored_data_fetch": ProtoConfigVal { val: Some(Bool(true)) }, "persist_part_decode_format": ProtoConfigVal { val: Some(String("row")) }, "persist_pubsub_client_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_pubsub_push_diff_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_reader_lease_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 900, nanos: 0 })) }, "persist_record_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_rollup_threshold": ProtoConfigVal { val: Some(Usize(128)) }, "persist_roundtrip_spine": ProtoConfigVal { val: Some(Bool(false)) }, "persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(0)) }, "persist_split_old_runs": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_audit_percent": ProtoConfigVal { val: Some(Usize(0)) }, "persist_stats_budget_bytes": ProtoConfigVal { val: Some(Usize(1024)) }, "persist_stats_collection_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_filter_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_untrimmable_columns_equals": ProtoConfigVal { val: Some(String("err,ts,receivedat,createdat,_fivetran_deleted")) }, "persist_stats_untrimmable_columns_prefix": ProtoConfigVal { val: Some(String("last_")) }, "persist_stats_untrimmable_columns_suffix": ProtoConfigVal { val: Some(String("timestamp,time,_at,_tstamp")) }, "persist_txns_data_shard_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 16, nanos: 0 })) }, "persist_txns_data_shard_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 24000000 })) }, "persist_txns_data_shard_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_use_critical_since_catalog": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_snapshot": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_source": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_txn": ProtoConfigVal { val: Some(Bool(true)) }, "persist_write_diffs_sum": ProtoConfigVal { val: Some(Bool(true)) }, "pg_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "plan_insights_notice fast_path_clusters_optimize_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 10000000 })) }, "postgres_fetch_slot_resume_lsn_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 500000000 })) }, "storage_cluster_shutdown_grace_period": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 600, nanos: 0 })) }, "storage_dataflow_delay_sources_past_rehydration": ProtoConfigVal { val: Some(Bool(true)) }, "storage_downgrade_since_during_finalization": ProtoConfigVal { val: Some(Bool(true)) }, "storage_enforce_external_addresses": ProtoConfigVal { val: Some(Bool(false)) }, "storage_persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(1024)) }, "storage_rocksdb_cleanup_tries": ProtoConfigVal { val: Some(Usize(5)) }, "storage_rocksdb_use_merge_operator": ProtoConfigVal { val: Some(Bool(false)) }, "storage_source_decode_fuel": ProtoConfigVal { val: Some(Usize(1000000)) }, "storage_upsert_max_snapshot_batch_buffering": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "storage_upsert_prevent_snapshot_buffering": ProtoConfigVal { val: Some(Bool(true)) }, "use_global_txn_cache_source": ProtoConfigVal { val: Some(Bool(true)) }} } }
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489168Z INFO mz_storage::storage_state: Applying configuration update: StorageParameters { pg_source_connect_timeout: Some(30s), pg_source_tcp_keepalives_retries: Some(5), pg_source_tcp_keepalives_idle: Some(10s), pg_source_tcp_keepalives_interval: Some(10s), pg_source_tcp_user_timeout: Some(60s), pg_source_tcp_configure_server: false, pg_source_snapshot_statement_timeout: 0ns, pg_source_wal_sender_timeout: None, mysql_source_timeouts: TimeoutConfig { snapshot_max_execution_time: Some(0ns), snapshot_lock_wait_timeout: Some(3600s), tcp_keepalive: Some(60s) }, keep_n_source_status_history_entries: 5, keep_n_sink_status_history_entries: 5, keep_n_privatelink_status_history_entries: 5, upsert_rocksdb_tuning_config: RocksDBTuningParameters { compaction_style: Level, optimize_compaction_memtable_budget: 178956970, level_compaction_dynamic_level_bytes: true, universal_compaction_target_ratio: 200, parallelism: None, compression_type: Lz4, bottommost_compression_type: Lz4, batch_size: 20480, retry_max_duration: 1s, stats_log_interval_seconds: 600, stats_persist_interval_seconds: 600, point_lookup_block_cache_size_mb: None, shrink_buffers_by_ratio: 0, write_buffer_manager_memory_bytes: None, write_buffer_manager_memory_fraction: None, write_buffer_manager_allow_stall: false }, finalize_shards: true, tracing: TracingParameters { log_filter: Some(info), opentelemetry_filter: Some(info), log_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })], opentelemetry_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("h2"), level: LevelFilter::OFF }), SerializableDirective(Directive { in_span: None, fields: [], target: Some("hyper"), level: LevelFilter::OFF })], sentry_filters: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })] }, upsert_auto_spill_config: UpsertAutoSpillConfig { allow_spilling_to_disk: false, spill_to_disk_threshold_bytes: 89478484 }, storage_dataflow_max_inflight_bytes_config: StorageMaxInflightBytesConfig { max_inflight_bytes_default: Some(268435456), max_inflight_bytes_cluster_size_fraction: Some(0.0025), disk_only: true }, grpc_client: GrpcClientParameters { connect_timeout: Some(5s), http2_keep_alive_interval: Some(3s), http2_keep_alive_timeout: Some(5s) }, shrink_upsert_unused_buffers_by_ratio: 0, record_namespaced_errors: true, ssh_timeout_config: SshTimeoutConfig { check_interval: 30s, connect_timeout: 30s, keepalives_idle: 10s }, kafka_timeout_config: TimeoutConfig { keepalive: true, socket_timeout: 60s, transaction_timeout: 60s, socket_connection_setup_timeout: 30s, fetch_metadata_timeout: 10s, progress_record_fetch_timeout: 90s, default_metadata_fetch_interval: 60s }, statistics_interval: 60s, statistics_collection_interval: 10s, pg_snapshot_config: PgSourceSnapshotConfig { collect_strict_count: true, fallback_to_strict_count: true, wait_for_count: true }, user_storage_managed_collections_batch_duration: 1s, dyncfg_updates: ConfigUpdates { updates: {"allow_user_sessions": ProtoConfigVal { val: Some(Bool(true)) }, "compute_dataflow_max_inflight_bytes": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_dataflow_max_inflight_bytes_cc": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_hydration_concurrency": ProtoConfigVal { val: Some(Usize(4)) }, "compute_persist_sink_obey_read_only": ProtoConfigVal { val: Some(Bool(true)) }, "controller_past_generation_replica_cleanup_retry_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "copy_to_s3_arrow_builder_buffer_ratio": ProtoConfigVal { val: Some(Usize(150)) }, "copy_to_s3_multipart_part_size_bytes": ProtoConfigVal { val: Some(Usize(8388608)) }, "copy_to_s3_parquet_row_group_file_ratio": ProtoConfigVal { val: Some(Usize(20)) }, "crdb_connect_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 5, nanos: 0 })) }, "crdb_tcp_user_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "enable_0dt_deployment": ProtoConfigVal { val: Some(Bool(false)) }, "enable_columnation_lgalloc": ProtoConfigVal { val: Some(Bool(false)) }, "enable_compute_chunked_stack": ProtoConfigVal { val: Some(Bool(false)) }, "enable_introspection_subscribes": ProtoConfigVal { val: Some(Bool(true)) }, "enable_lgalloc_eager_reclamation": ProtoConfigVal { val: Some(Bool(true)) }, "enable_mz_join_core": ProtoConfigVal { val: Some(Bool(true)) }, "enable_statement_lifecycle_logging": ProtoConfigVal { val: Some(Bool(false)) }, "kafka_client_id_enrichment_rules": ProtoConfigVal { val: Some(Json("[]")) }, "kafka_poll_max_wait": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_background_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_slow_clear_bytes": ProtoConfigVal { val: Some(Usize(33554432)) }, "linear_join_yielding": ProtoConfigVal { val: Some(String("work:1000000,time:100")) }, "mysql_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "mysql_replication_heartbeat_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "persist_batch_columnar_format": ProtoConfigVal { val: Some(String("row")) }, "persist_batch_delete_enabled": ProtoConfigVal { val: Some(Bool(false)) }, "persist_batch_record_part_format": ProtoConfigVal { val: Some(Bool(false)) }, "persist_blob_cache_mem_limit_bytes": ProtoConfigVal { val: Some(Usize(1048576)) }, "persist_blob_target_size": ProtoConfigVal { val: Some(Usize(134217728)) }, "persist_claim_unclaimed_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_compaction_minimum_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 90, nanos: 0 })) }, "persist_consensus_connection_pool_ttl": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "persist_consensus_connection_pool_ttl_stagger": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 6, nanos: 0 })) }, "persist_enable_arrow_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_arrow_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_enable_s3_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_s3_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_fetch_semaphore_cost_adjustment": ProtoConfigVal { val: Some(F64(1.2)) }, "persist_fetch_semaphore_permit_adjustment": ProtoConfigVal { val: Some(F64(1.0)) }, "persist_inline_writes_single_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_inline_writes_total_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_next_listen_batch_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "persist_next_listen_batch_retryer_fixed_sleep": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 0 })) }, "persist_next_listen_batch_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 200000000 })) }, "persist_next_listen_batch_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_optimize_ignored_data_decode": ProtoConfigVal { val: Some(Bool(true)) }, "persist_optimize_ignored_data_fetch": ProtoConfigVal { val: Some(Bool(true)) }, "persist_part_decode_format": ProtoConfigVal { val: Some(String("row")) }, "persist_pubsub_client_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_pubsub_push_diff_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_reader_lease_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 900, nanos: 0 })) }, "persist_record_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_rollup_threshold": ProtoConfigVal { val: Some(Usize(128)) }, "persist_roundtrip_spine": ProtoConfigVal { val: Some(Bool(false)) }, "persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(0)) }, "persist_split_old_runs": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_audit_percent": ProtoConfigVal { val: Some(Usize(0)) }, "persist_stats_budget_bytes": ProtoConfigVal { val: Some(Usize(1024)) }, "persist_stats_collection_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_filter_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_untrimmable_columns_equals": ProtoConfigVal { val: Some(String("err,ts,receivedat,createdat,_fivetran_deleted")) }, "persist_stats_untrimmable_columns_prefix": ProtoConfigVal { val: Some(String("last_")) }, "persist_stats_untrimmable_columns_suffix": ProtoConfigVal { val: Some(String("timestamp,time,_at,_tstamp")) }, "persist_txns_data_shard_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 16, nanos: 0 })) }, "persist_txns_data_shard_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 24000000 })) }, "persist_txns_data_shard_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_use_critical_since_catalog": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_snapshot": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_source": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_txn": ProtoConfigVal { val: Some(Bool(true)) }, "persist_write_diffs_sum": ProtoConfigVal { val: Some(Bool(true)) }, "pg_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "plan_insights_notice fast_path_clusters_optimize_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 10000000 })) }, "postgres_fetch_slot_resume_lsn_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 500000000 })) }, "storage_cluster_shutdown_grace_period": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 600, nanos: 0 })) }, "storage_dataflow_delay_sources_past_rehydration": ProtoConfigVal { val: Some(Bool(true)) }, "storage_downgrade_since_during_finalization": ProtoConfigVal { val: Some(Bool(true)) }, "storage_enforce_external_addresses": ProtoConfigVal { val: Some(Bool(false)) }, "storage_persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(1024)) }, "storage_rocksdb_cleanup_tries": ProtoConfigVal { val: Some(Usize(5)) }, "storage_rocksdb_use_merge_operator": ProtoConfigVal { val: Some(Bool(false)) }, "storage_source_decode_fuel": ProtoConfigVal { val: Some(Usize(1000000)) }, "storage_upsert_max_snapshot_batch_buffering": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "storage_upsert_prevent_snapshot_buffering": ProtoConfigVal { val: Some(Bool(true)) }, "use_global_txn_cache_source": ProtoConfigVal { val: Some(Bool(true)) }} } }
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489273Z INFO mz_persist_client::rpc: Connecting to Persist PubSub: http://localhost:6879
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489719Z INFO mz_persist_client::rpc: Connected to Persist PubSub: http://localhost:6879
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489799Z INFO mz_compute::compute_state: Applying configuration update: ComputeParameters { max_result_size: Some(1073741824), tracing: TracingParameters { log_filter: Some(info), opentelemetry_filter: Some(info), log_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })], opentelemetry_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("h2"), level: LevelFilter::OFF }), SerializableDirective(Directive { in_span: None, fields: [], target: Some("hyper"), level: LevelFilter::OFF })], sentry_filters: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })] }, grpc_client: GrpcClientParameters { connect_timeout: Some(5s), http2_keep_alive_interval: Some(3s), http2_keep_alive_timeout: Some(5s) }, dyncfg_updates: ConfigUpdates { updates: {"allow_user_sessions": ProtoConfigVal { val: Some(Bool(true)) }, "compute_dataflow_max_inflight_bytes": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_dataflow_max_inflight_bytes_cc": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_hydration_concurrency": ProtoConfigVal { val: Some(Usize(4)) }, "compute_persist_sink_obey_read_only": ProtoConfigVal { val: Some(Bool(true)) }, "controller_past_generation_replica_cleanup_retry_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "copy_to_s3_arrow_builder_buffer_ratio": ProtoConfigVal { val: Some(Usize(150)) }, "copy_to_s3_multipart_part_size_bytes": ProtoConfigVal { val: Some(Usize(8388608)) }, "copy_to_s3_parquet_row_group_file_ratio": ProtoConfigVal { val: Some(Usize(20)) }, "crdb_connect_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 5, nanos: 0 })) }, "crdb_tcp_user_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "enable_0dt_deployment": ProtoConfigVal { val: Some(Bool(false)) }, "enable_columnation_lgalloc": ProtoConfigVal { val: Some(Bool(false)) }, "enable_compute_chunked_stack": ProtoConfigVal { val: Some(Bool(false)) }, "enable_introspection_subscribes": ProtoConfigVal { val: Some(Bool(true)) }, "enable_lgalloc_eager_reclamation": ProtoConfigVal { val: Some(Bool(true)) }, "enable_mz_join_core": ProtoConfigVal { val: Some(Bool(true)) }, "enable_statement_lifecycle_logging": ProtoConfigVal { val: Some(Bool(false)) }, "kafka_client_id_enrichment_rules": ProtoConfigVal { val: Some(Json("[]")) }, "kafka_poll_max_wait": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_background_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_slow_clear_bytes": ProtoConfigVal { val: Some(Usize(33554432)) }, "linear_join_yielding": ProtoConfigVal { val: Some(String("work:1000000,time:100")) }, "mysql_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "mysql_replication_heartbeat_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "persist_batch_columnar_format": ProtoConfigVal { val: Some(String("row")) }, "persist_batch_delete_enabled": ProtoConfigVal { val: Some(Bool(false)) }, "persist_batch_record_part_format": ProtoConfigVal { val: Some(Bool(false)) }, "persist_blob_cache_mem_limit_bytes": ProtoConfigVal { val: Some(Usize(1048576)) }, "persist_blob_target_size": ProtoConfigVal { val: Some(Usize(134217728)) }, "persist_claim_unclaimed_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_compaction_minimum_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 90, nanos: 0 })) }, "persist_consensus_connection_pool_ttl": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "persist_consensus_connection_pool_ttl_stagger": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 6, nanos: 0 })) }, "persist_enable_arrow_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_arrow_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_enable_s3_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_s3_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_fetch_semaphore_cost_adjustment": ProtoConfigVal { val: Some(F64(1.2)) }, "persist_fetch_semaphore_permit_adjustment": ProtoConfigVal { val: Some(F64(1.0)) }, "persist_inline_writes_single_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_inline_writes_total_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_next_listen_batch_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "persist_next_listen_batch_retryer_fixed_sleep": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 0 })) }, "persist_next_listen_batch_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 200000000 })) }, "persist_next_listen_batch_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_optimize_ignored_data_decode": ProtoConfigVal { val: Some(Bool(true)) }, "persist_optimize_ignored_data_fetch": ProtoConfigVal { val: Some(Bool(true)) }, "persist_part_decode_format": ProtoConfigVal { val: Some(String("row")) }, "persist_pubsub_client_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_pubsub_push_diff_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_reader_lease_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 900, nanos: 0 })) }, "persist_record_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_rollup_threshold": ProtoConfigVal { val: Some(Usize(128)) }, "persist_roundtrip_spine": ProtoConfigVal { val: Some(Bool(false)) }, "persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(0)) }, "persist_split_old_runs": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_audit_percent": ProtoConfigVal { val: Some(Usize(0)) }, "persist_stats_budget_bytes": ProtoConfigVal { val: Some(Usize(1024)) }, "persist_stats_collection_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_filter_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_untrimmable_columns_equals": ProtoConfigVal { val: Some(String("err,ts,receivedat,createdat,_fivetran_deleted")) }, "persist_stats_untrimmable_columns_prefix": ProtoConfigVal { val: Some(String("last_")) }, "persist_stats_untrimmable_columns_suffix": ProtoConfigVal { val: Some(String("timestamp,time,_at,_tstamp")) }, "persist_txns_data_shard_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 16, nanos: 0 })) }, "persist_txns_data_shard_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 24000000 })) }, "persist_txns_data_shard_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_use_critical_since_catalog": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_snapshot": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_source": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_txn": ProtoConfigVal { val: Some(Bool(true)) }, "persist_write_diffs_sum": ProtoConfigVal { val: Some(Bool(true)) }, "pg_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "plan_insights_notice fast_path_clusters_optimize_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 10000000 })) }, "postgres_fetch_slot_resume_lsn_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 500000000 })) }, "storage_cluster_shutdown_grace_period": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 600, nanos: 0 })) }, "storage_dataflow_delay_sources_past_rehydration": ProtoConfigVal { val: Some(Bool(true)) }, "storage_downgrade_since_during_finalization": ProtoConfigVal { val: Some(Bool(true)) }, "storage_enforce_external_addresses": ProtoConfigVal { val: Some(Bool(false)) }, "storage_persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(1024)) }, "storage_rocksdb_cleanup_tries": ProtoConfigVal { val: Some(Usize(5)) }, "storage_rocksdb_use_merge_operator": ProtoConfigVal { val: Some(Bool(false)) }, "storage_source_decode_fuel": ProtoConfigVal { val: Some(Usize(1000000)) }, "storage_upsert_max_snapshot_batch_buffering": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "storage_upsert_prevent_snapshot_buffering": ProtoConfigVal { val: Some(Bool(true)) }, "use_global_txn_cache_source": ProtoConfigVal { val: Some(Bool(true)) }} } }
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489795Z INFO mz_compute::compute_state: Applying configuration update: ComputeParameters { max_result_size: Some(1073741824), tracing: TracingParameters { log_filter: Some(info), opentelemetry_filter: Some(info), log_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })], opentelemetry_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("h2"), level: LevelFilter::OFF }), SerializableDirective(Directive { in_span: None, fields: [], target: Some("hyper"), level: LevelFilter::OFF })], sentry_filters: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })] }, grpc_client: GrpcClientParameters { connect_timeout: Some(5s), http2_keep_alive_interval: Some(3s), http2_keep_alive_timeout: Some(5s) }, dyncfg_updates: ConfigUpdates { updates: {"allow_user_sessions": ProtoConfigVal { val: Some(Bool(true)) }, "compute_dataflow_max_inflight_bytes": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_dataflow_max_inflight_bytes_cc": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_hydration_concurrency": ProtoConfigVal { val: Some(Usize(4)) }, "compute_persist_sink_obey_read_only": ProtoConfigVal { val: Some(Bool(true)) }, "controller_past_generation_replica_cleanup_retry_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "copy_to_s3_arrow_builder_buffer_ratio": ProtoConfigVal { val: Some(Usize(150)) }, "copy_to_s3_multipart_part_size_bytes": ProtoConfigVal { val: Some(Usize(8388608)) }, "copy_to_s3_parquet_row_group_file_ratio": ProtoConfigVal { val: Some(Usize(20)) }, "crdb_connect_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 5, nanos: 0 })) }, "crdb_tcp_user_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "enable_0dt_deployment": ProtoConfigVal { val: Some(Bool(false)) }, "enable_columnation_lgalloc": ProtoConfigVal { val: Some(Bool(false)) }, "enable_compute_chunked_stack": ProtoConfigVal { val: Some(Bool(false)) }, "enable_introspection_subscribes": ProtoConfigVal { val: Some(Bool(true)) }, "enable_lgalloc_eager_reclamation": ProtoConfigVal { val: Some(Bool(true)) }, "enable_mz_join_core": ProtoConfigVal { val: Some(Bool(true)) }, "enable_statement_lifecycle_logging": ProtoConfigVal { val: Some(Bool(false)) }, "kafka_client_id_enrichment_rules": ProtoConfigVal { val: Some(Json("[]")) }, "kafka_poll_max_wait": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_background_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_slow_clear_bytes": ProtoConfigVal { val: Some(Usize(33554432)) }, "linear_join_yielding": ProtoConfigVal { val: Some(String("work:1000000,time:100")) }, "mysql_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "mysql_replication_heartbeat_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "persist_batch_columnar_format": ProtoConfigVal { val: Some(String("row")) }, "persist_batch_delete_enabled": ProtoConfigVal { val: Some(Bool(false)) }, "persist_batch_record_part_format": ProtoConfigVal { val: Some(Bool(false)) }, "persist_blob_cache_mem_limit_bytes": ProtoConfigVal { val: Some(Usize(1048576)) }, "persist_blob_target_size": ProtoConfigVal { val: Some(Usize(134217728)) }, "persist_claim_unclaimed_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_compaction_minimum_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 90, nanos: 0 })) }, "persist_consensus_connection_pool_ttl": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "persist_consensus_connection_pool_ttl_stagger": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 6, nanos: 0 })) }, "persist_enable_arrow_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_arrow_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_enable_s3_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_s3_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_fetch_semaphore_cost_adjustment": ProtoConfigVal { val: Some(F64(1.2)) }, "persist_fetch_semaphore_permit_adjustment": ProtoConfigVal { val: Some(F64(1.0)) }, "persist_inline_writes_single_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_inline_writes_total_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_next_listen_batch_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "persist_next_listen_batch_retryer_fixed_sleep": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 0 })) }, "persist_next_listen_batch_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 200000000 })) }, "persist_next_listen_batch_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_optimize_ignored_data_decode": ProtoConfigVal { val: Some(Bool(true)) }, "persist_optimize_ignored_data_fetch": ProtoConfigVal { val: Some(Bool(true)) }, "persist_part_decode_format": ProtoConfigVal { val: Some(String("row")) }, "persist_pubsub_client_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_pubsub_push_diff_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_reader_lease_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 900, nanos: 0 })) }, "persist_record_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_rollup_threshold": ProtoConfigVal { val: Some(Usize(128)) }, "persist_roundtrip_spine": ProtoConfigVal { val: Some(Bool(false)) }, "persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(0)) }, "persist_split_old_runs": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_audit_percent": ProtoConfigVal { val: Some(Usize(0)) }, "persist_stats_budget_bytes": ProtoConfigVal { val: Some(Usize(1024)) }, "persist_stats_collection_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_filter_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_untrimmable_columns_equals": ProtoConfigVal { val: Some(String("err,ts,receivedat,createdat,_fivetran_deleted")) }, "persist_stats_untrimmable_columns_prefix": ProtoConfigVal { val: Some(String("last_")) }, "persist_stats_untrimmable_columns_suffix": ProtoConfigVal { val: Some(String("timestamp,time,_at,_tstamp")) }, "persist_txns_data_shard_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 16, nanos: 0 })) }, "persist_txns_data_shard_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 24000000 })) }, "persist_txns_data_shard_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_use_critical_since_catalog": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_snapshot": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_source": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_txn": ProtoConfigVal { val: Some(Bool(true)) }, "persist_write_diffs_sum": ProtoConfigVal { val: Some(Bool(true)) }, "pg_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "plan_insights_notice fast_path_clusters_optimize_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 10000000 })) }, "postgres_fetch_slot_resume_lsn_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 500000000 })) }, "storage_cluster_shutdown_grace_period": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 600, nanos: 0 })) }, "storage_dataflow_delay_sources_past_rehydration": ProtoConfigVal { val: Some(Bool(true)) }, "storage_downgrade_since_during_finalization": ProtoConfigVal { val: Some(Bool(true)) }, "storage_enforce_external_addresses": ProtoConfigVal { val: Some(Bool(false)) }, "storage_persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(1024)) }, "storage_rocksdb_cleanup_tries": ProtoConfigVal { val: Some(Usize(5)) }, "storage_rocksdb_use_merge_operator": ProtoConfigVal { val: Some(Bool(false)) }, "storage_source_decode_fuel": ProtoConfigVal { val: Some(Usize(1000000)) }, "storage_upsert_max_snapshot_batch_buffering": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "storage_upsert_prevent_snapshot_buffering": ProtoConfigVal { val: Some(Bool(true)) }, "use_global_txn_cache_source": ProtoConfigVal { val: Some(Bool(true)) }} } }
environmentd: 2024-07-11T10:33:17.489846Z INFO persist::rpc::server: mz_persist_client::rpc: Received Persist PubSub connection from: "f50feabed3a9"
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489859Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489861Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489816Z INFO mz_compute::compute_state: Applying configuration update: ComputeParameters { max_result_size: Some(1073741824), tracing: TracingParameters { log_filter: Some(info), opentelemetry_filter: Some(info), log_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })], opentelemetry_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("h2"), level: LevelFilter::OFF }), SerializableDirective(Directive { in_span: None, fields: [], target: Some("hyper"), level: LevelFilter::OFF })], sentry_filters: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })] }, grpc_client: GrpcClientParameters { connect_timeout: Some(5s), http2_keep_alive_interval: Some(3s), http2_keep_alive_timeout: Some(5s) }, dyncfg_updates: ConfigUpdates { updates: {"allow_user_sessions": ProtoConfigVal { val: Some(Bool(true)) }, "compute_dataflow_max_inflight_bytes": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_dataflow_max_inflight_bytes_cc": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_hydration_concurrency": ProtoConfigVal { val: Some(Usize(4)) }, "compute_persist_sink_obey_read_only": ProtoConfigVal { val: Some(Bool(true)) }, "controller_past_generation_replica_cleanup_retry_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "copy_to_s3_arrow_builder_buffer_ratio": ProtoConfigVal { val: Some(Usize(150)) }, "copy_to_s3_multipart_part_size_bytes": ProtoConfigVal { val: Some(Usize(8388608)) }, "copy_to_s3_parquet_row_group_file_ratio": ProtoConfigVal { val: Some(Usize(20)) }, "crdb_connect_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 5, nanos: 0 })) }, "crdb_tcp_user_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "enable_0dt_deployment": ProtoConfigVal { val: Some(Bool(false)) }, "enable_columnation_lgalloc": ProtoConfigVal { val: Some(Bool(false)) }, "enable_compute_chunked_stack": ProtoConfigVal { val: Some(Bool(false)) }, "enable_introspection_subscribes": ProtoConfigVal { val: Some(Bool(true)) }, "enable_lgalloc_eager_reclamation": ProtoConfigVal { val: Some(Bool(true)) }, "enable_mz_join_core": ProtoConfigVal { val: Some(Bool(true)) }, "enable_statement_lifecycle_logging": ProtoConfigVal { val: Some(Bool(false)) }, "kafka_client_id_enrichment_rules": ProtoConfigVal { val: Some(Json("[]")) }, "kafka_poll_max_wait": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_background_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_slow_clear_bytes": ProtoConfigVal { val: Some(Usize(33554432)) }, "linear_join_yielding": ProtoConfigVal { val: Some(String("work:1000000,time:100")) }, "mysql_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "mysql_replication_heartbeat_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "persist_batch_columnar_format": ProtoConfigVal { val: Some(String("row")) }, "persist_batch_delete_enabled": ProtoConfigVal { val: Some(Bool(false)) }, "persist_batch_record_part_format": ProtoConfigVal { val: Some(Bool(false)) }, "persist_blob_cache_mem_limit_bytes": ProtoConfigVal { val: Some(Usize(1048576)) }, "persist_blob_target_size": ProtoConfigVal { val: Some(Usize(134217728)) }, "persist_claim_unclaimed_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_compaction_minimum_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 90, nanos: 0 })) }, "persist_consensus_connection_pool_ttl": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "persist_consensus_connection_pool_ttl_stagger": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 6, nanos: 0 })) }, "persist_enable_arrow_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_arrow_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_enable_s3_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_s3_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_fetch_semaphore_cost_adjustment": ProtoConfigVal { val: Some(F64(1.2)) }, "persist_fetch_semaphore_permit_adjustment": ProtoConfigVal { val: Some(F64(1.0)) }, "persist_inline_writes_single_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_inline_writes_total_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_next_listen_batch_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "persist_next_listen_batch_retryer_fixed_sleep": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 0 })) }, "persist_next_listen_batch_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 200000000 })) }, "persist_next_listen_batch_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_optimize_ignored_data_decode": ProtoConfigVal { val: Some(Bool(true)) }, "persist_optimize_ignored_data_fetch": ProtoConfigVal { val: Some(Bool(true)) }, "persist_part_decode_format": ProtoConfigVal { val: Some(String("row")) }, "persist_pubsub_client_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_pubsub_push_diff_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_reader_lease_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 900, nanos: 0 })) }, "persist_record_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_rollup_threshold": ProtoConfigVal { val: Some(Usize(128)) }, "persist_roundtrip_spine": ProtoConfigVal { val: Some(Bool(false)) }, "persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(0)) }, "persist_split_old_runs": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_audit_percent": ProtoConfigVal { val: Some(Usize(0)) }, "persist_stats_budget_bytes": ProtoConfigVal { val: Some(Usize(1024)) }, "persist_stats_collection_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_filter_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_untrimmable_columns_equals": ProtoConfigVal { val: Some(String("err,ts,receivedat,createdat,_fivetran_deleted")) }, "persist_stats_untrimmable_columns_prefix": ProtoConfigVal { val: Some(String("last_")) }, "persist_stats_untrimmable_columns_suffix": ProtoConfigVal { val: Some(String("timestamp,time,_at,_tstamp")) }, "persist_txns_data_shard_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 16, nanos: 0 })) }, "persist_txns_data_shard_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 24000000 })) }, "persist_txns_data_shard_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_use_critical_since_catalog": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_snapshot": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_source": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_txn": ProtoConfigVal { val: Some(Bool(true)) }, "persist_write_diffs_sum": ProtoConfigVal { val: Some(Bool(true)) }, "pg_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "plan_insights_notice fast_path_clusters_optimize_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 10000000 })) }, "postgres_fetch_slot_resume_lsn_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 500000000 })) }, "storage_cluster_shutdown_grace_period": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 600, nanos: 0 })) }, "storage_dataflow_delay_sources_past_rehydration": ProtoConfigVal { val: Some(Bool(true)) }, "storage_downgrade_since_during_finalization": ProtoConfigVal { val: Some(Bool(true)) }, "storage_enforce_external_addresses": ProtoConfigVal { val: Some(Bool(false)) }, "storage_persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(1024)) }, "storage_rocksdb_cleanup_tries": ProtoConfigVal { val: Some(Usize(5)) }, "storage_rocksdb_use_merge_operator": ProtoConfigVal { val: Some(Bool(false)) }, "storage_source_decode_fuel": ProtoConfigVal { val: Some(Usize(1000000)) }, "storage_upsert_max_snapshot_batch_buffering": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "storage_upsert_prevent_snapshot_buffering": ProtoConfigVal { val: Some(Bool(true)) }, "use_global_txn_cache_source": ProtoConfigVal { val: Some(Bool(true)) }} } }
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489866Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u18-gen-0: 2024-07-11T10:33:17.489868Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489823Z INFO mz_compute::compute_state: Applying configuration update: ComputeParameters { max_result_size: Some(1073741824), tracing: TracingParameters { log_filter: Some(info), opentelemetry_filter: Some(info), log_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })], opentelemetry_filter_defaults: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("h2"), level: LevelFilter::OFF }), SerializableDirective(Directive { in_span: None, fields: [], target: Some("hyper"), level: LevelFilter::OFF })], sentry_filters: [SerializableDirective(Directive { in_span: None, fields: [], target: Some("kube_client::client::builder"), level: LevelFilter::OFF })] }, grpc_client: GrpcClientParameters { connect_timeout: Some(5s), http2_keep_alive_interval: Some(3s), http2_keep_alive_timeout: Some(5s) }, dyncfg_updates: ConfigUpdates { updates: {"allow_user_sessions": ProtoConfigVal { val: Some(Bool(true)) }, "compute_dataflow_max_inflight_bytes": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_dataflow_max_inflight_bytes_cc": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "compute_hydration_concurrency": ProtoConfigVal { val: Some(Usize(4)) }, "compute_persist_sink_obey_read_only": ProtoConfigVal { val: Some(Bool(true)) }, "controller_past_generation_replica_cleanup_retry_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "copy_to_s3_arrow_builder_buffer_ratio": ProtoConfigVal { val: Some(Usize(150)) }, "copy_to_s3_multipart_part_size_bytes": ProtoConfigVal { val: Some(Usize(8388608)) }, "copy_to_s3_parquet_row_group_file_ratio": ProtoConfigVal { val: Some(Usize(20)) }, "crdb_connect_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 5, nanos: 0 })) }, "crdb_tcp_user_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "enable_0dt_deployment": ProtoConfigVal { val: Some(Bool(false)) }, "enable_columnation_lgalloc": ProtoConfigVal { val: Some(Bool(false)) }, "enable_compute_chunked_stack": ProtoConfigVal { val: Some(Bool(false)) }, "enable_introspection_subscribes": ProtoConfigVal { val: Some(Bool(true)) }, "enable_lgalloc_eager_reclamation": ProtoConfigVal { val: Some(Bool(true)) }, "enable_mz_join_core": ProtoConfigVal { val: Some(Bool(true)) }, "enable_statement_lifecycle_logging": ProtoConfigVal { val: Some(Bool(false)) }, "kafka_client_id_enrichment_rules": ProtoConfigVal { val: Some(Json("[]")) }, "kafka_poll_max_wait": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_background_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "lgalloc_slow_clear_bytes": ProtoConfigVal { val: Some(Usize(33554432)) }, "linear_join_yielding": ProtoConfigVal { val: Some(String("work:1000000,time:100")) }, "mysql_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "mysql_replication_heartbeat_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 30, nanos: 0 })) }, "persist_batch_columnar_format": ProtoConfigVal { val: Some(String("row")) }, "persist_batch_delete_enabled": ProtoConfigVal { val: Some(Bool(false)) }, "persist_batch_record_part_format": ProtoConfigVal { val: Some(Bool(false)) }, "persist_blob_cache_mem_limit_bytes": ProtoConfigVal { val: Some(Usize(1048576)) }, "persist_blob_target_size": ProtoConfigVal { val: Some(Usize(134217728)) }, "persist_claim_unclaimed_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_compaction_minimum_timeout": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 90, nanos: 0 })) }, "persist_consensus_connection_pool_ttl": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 300, nanos: 0 })) }, "persist_consensus_connection_pool_ttl_stagger": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 6, nanos: 0 })) }, "persist_enable_arrow_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_arrow_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_enable_s3_lgalloc_cc_sizes": ProtoConfigVal { val: Some(Bool(true)) }, "persist_enable_s3_lgalloc_noncc_sizes": ProtoConfigVal { val: Some(Bool(false)) }, "persist_fetch_semaphore_cost_adjustment": ProtoConfigVal { val: Some(F64(1.2)) }, "persist_fetch_semaphore_permit_adjustment": ProtoConfigVal { val: Some(F64(1.0)) }, "persist_inline_writes_single_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_inline_writes_total_max_bytes": ProtoConfigVal { val: Some(Usize(0)) }, "persist_next_listen_batch_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 0 })) }, "persist_next_listen_batch_retryer_fixed_sleep": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 0 })) }, "persist_next_listen_batch_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 200000000 })) }, "persist_next_listen_batch_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_optimize_ignored_data_decode": ProtoConfigVal { val: Some(Bool(true)) }, "persist_optimize_ignored_data_fetch": ProtoConfigVal { val: Some(Bool(true)) }, "persist_part_decode_format": ProtoConfigVal { val: Some(String("row")) }, "persist_pubsub_client_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_pubsub_push_diff_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_reader_lease_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 900, nanos: 0 })) }, "persist_record_compactions": ProtoConfigVal { val: Some(Bool(false)) }, "persist_rollup_threshold": ProtoConfigVal { val: Some(Usize(128)) }, "persist_roundtrip_spine": ProtoConfigVal { val: Some(Bool(false)) }, "persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(0)) }, "persist_split_old_runs": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_audit_percent": ProtoConfigVal { val: Some(Usize(0)) }, "persist_stats_budget_bytes": ProtoConfigVal { val: Some(Usize(1024)) }, "persist_stats_collection_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_filter_enabled": ProtoConfigVal { val: Some(Bool(true)) }, "persist_stats_untrimmable_columns_equals": ProtoConfigVal { val: Some(String("err,ts,receivedat,createdat,_fivetran_deleted")) }, "persist_stats_untrimmable_columns_prefix": ProtoConfigVal { val: Some(String("last_")) }, "persist_stats_untrimmable_columns_suffix": ProtoConfigVal { val: Some(String("timestamp,time,_at,_tstamp")) }, "persist_txns_data_shard_retryer_clamp": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 16, nanos: 0 })) }, "persist_txns_data_shard_retryer_initial_backoff": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 1, nanos: 24000000 })) }, "persist_txns_data_shard_retryer_multiplier": ProtoConfigVal { val: Some(U32(2)) }, "persist_use_critical_since_catalog": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_snapshot": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_source": ProtoConfigVal { val: Some(Bool(false)) }, "persist_use_critical_since_txn": ProtoConfigVal { val: Some(Bool(true)) }, "persist_write_diffs_sum": ProtoConfigVal { val: Some(Bool(true)) }, "pg_offset_known_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 10, nanos: 0 })) }, "plan_insights_notice fast_path_clusters_optimize_duration": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 10000000 })) }, "postgres_fetch_slot_resume_lsn_interval": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 0, nanos: 500000000 })) }, "storage_cluster_shutdown_grace_period": ProtoConfigVal { val: Some(Duration(ProtoDuration { secs: 600, nanos: 0 })) }, "storage_dataflow_delay_sources_past_rehydration": ProtoConfigVal { val: Some(Bool(true)) }, "storage_downgrade_since_during_finalization": ProtoConfigVal { val: Some(Bool(true)) }, "storage_enforce_external_addresses": ProtoConfigVal { val: Some(Bool(false)) }, "storage_persist_sink_minimum_batch_updates": ProtoConfigVal { val: Some(Usize(1024)) }, "storage_rocksdb_cleanup_tries": ProtoConfigVal { val: Some(Usize(5)) }, "storage_rocksdb_use_merge_operator": ProtoConfigVal { val: Some(Bool(false)) }, "storage_source_decode_fuel": ProtoConfigVal { val: Some(Usize(1000000)) }, "storage_upsert_max_snapshot_batch_buffering": ProtoConfigVal { val: Some(OptUsize(ProtoOptionU64 { val: None })) }, "storage_upsert_prevent_snapshot_buffering": ProtoConfigVal { val: Some(Bool(true)) }, "use_global_txn_cache_source": ProtoConfigVal { val: Some(Bool(true)) }} } }
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489904Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489906Z INFO mz_compute::compute_state: disabling lgalloc
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489908Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489909Z INFO mz_compute::compute_state: using chunked stack: false
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.489925Z INFO mz_persist_client::rpc: Connecting to Persist PubSub: http://localhost:6879
cluster-u10-replica-u17-gen-0: 2024-07-11T10:33:17.490315Z INFO mz_persist_client::rpc: Connected to Persist PubSub: http://localhost:6879
environmentd: 2024-07-11T10:33:17.490404Z INFO persist::rpc::server: mz_persist_client::rpc: Received Persist PubSub connection from: "f50feabed3a9"
environmentd: 2024-07-11T10:33:18.050336Z INFO coord::handle_message{kind="command-execute"}:message_command:coord::handle_execute{session="68f4f7f9-c9f8-4e0b-bc6f-d821a7ca2ad5"}:coord::handle_execute_inner{stmt="DROP CLUSTER REPLICA c.r1"}:sequence_drop_objects:coord::catalog_transact:coord::catalog_transact_conn:coord::catalog_transact_inner:coord::catalog_transact_with::finalize: mz_adapter::coord::introspection: dropping introspection subscribe id=t107 replica_id=u17 type_=ComputeErrorCounts
environmentd: 2024-07-11T10:33:18.050439Z INFO mz_compute_client::controller::replica: stopped replica task replica=User(17)
environmentd: 2024-07-11T10:33:18.050947Z INFO mz_orchestrator_process: terminating orphaned process for cluster-u10-replica-u17-gen-0 with PID 1845
environmentd: 2024-07-11T10:33:18.054162Z WARN persist::rpc::server:connection{caller_id="f50feabed3a9"}: mz_persist_client::rpc: pubsub connection err: status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }
environmentd: 2024-07-11T10:33:18.054171Z INFO persist::rpc::server:connection{caller_id="f50feabed3a9"}: mz_persist_client::rpc: Persist PubSub connection ended: "f50feabed3a9"
environmentd: 2024-07-11T10:33:18.499823Z INFO coord::handle_message{kind="purified_statement_ready"}:sequence_create_source:coord::catalog_transact_with_side_effects:coord::catalog_transact_inner:catalog::transact:catalog::transact_inner:transact_op: mz_adapter::catalog::transact: create source materialize.public.counter_progress (u14)
environmentd: 2024-07-11T10:33:18.500086Z INFO coord::handle_message{kind="purified_statement_ready"}:sequence_create_source:coord::catalog_transact_with_side_effects:coord::catalog_transact_inner:catalog::transact:catalog::transact_inner:transact_op: mz_adapter::catalog::transact: create source materialize.public.counter (u15)
environmentd: 2024-07-11T10:33:18.778313Z INFO coord::handle_message{kind="purified_statement_ready"}:sequence_create_source:coord::catalog_transact_with_side_effects:coord::catalog_transact_with_side_effects::side_effects_fut:storage::create_collections: mz_storage_controller: starting ingestion id=u15
cluster-s1-replica-s1-gen-0: 2024-07-11T10:33:20.888092Z INFO mz_persist_client::internal::machine: snapshot saa0337ad-b3e6-4441-a357-a8d95bb71907 as of [1720693998583] not yet available for v5 upper [0]
environmentd: 2024-07-11T10:33:21.325694Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:26.276494Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:31.228086Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:36.259445Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:41.261935Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:46.271684Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:51.228058Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:33:56.273360Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:34:01.270646Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:34:06.270803Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:34:11.259540Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
environmentd: 2024-07-11T10:34:16.254856Z WARN mz_storage_client::storage_collections: error during finalization of shard s4ee9a640-4dd1-4441-9290-bfd3fd27ca5e: FinalizationError { since: Antichain { elements: [0] }, upper: Antichain { elements: [] } }
This is not a regression, happens also in old versions. Maybe this is not supposed to work? I'll disable dropping cluster replicas for now in parallel-workload.
The issue is that the Controller
's handling of storage replicas becomes confused. Storage only supports single-replica clusters, but the top-level Controller
supports multi-replica clusters because it also wraps compute, and it presents a multi-replica API to its clients.
The storage-related logic performed by the Controller
when replicas are added/removed is less smart than it needs to be:
So there are two reasons why the storage controller can end up in a disconnected state even when the cluster has replicas installed.
What version of Materialize are you using?
708f88f6cb1459f0c3f85753386a7941c845a78f
What is the issue?
I found this while looking into https://github.com/MaterializeInc/materialize/issues/23582 by just running
bin/mzcompose --find parallel-workload down && bin/mzcompose --find parallel-workload run default
. I don't think it explains all the stuck queries, but at least a subset:But the cluster reports to be in a good state and has a replica:
In truth creating new sources on the cluster will have the same problem:
All queries that ran: parallel-workload-queries.log There were a bunch of attempts to drop the cluster and replica, which should have failed since it had sources:
But since the replica is still shown as existing the drop somehow didn't complete fully. Logs: services.zip
I'll try to create a smaller reproducer.