search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.78k stars 561 forks source link

Issue with new files on minio storage #17991

Open adrianhaj opened 1 month ago

adrianhaj commented 1 month ago

Describe the bug

After creating a source and a materialized view in the RisingWave application, new data does not appear in the sink, despite uploading new files to the MinIO storage.

Error message/log

No logs.

To Reproduce

First clone https://github.com/risingwavelabs/risingwave go to docer and run:

docker-compose up

Create new bucket on minio examples

Create source:

CREATE
SOURCE minio_source (
                   id int,
                   name TEXT,
                   amount NUMERIC,
                   operation_time TIMESTAMP WITH TIME ZONE
)
INCLUDE offset AS scoring_offset WITH (connector = 's3_v2',
    s3.region_name = 'us-east-1',
    s3.bucket_name = 'examples',
    s3.credentials.access = 'hummockadmin',
    s3.credentials.secret = 'hummockadmin',
    match_pattern = '*. ndjson',
    s3.endpoint_url = 'http://minio-0:9301') ORMAT PLAIN ENCODE JSON;

Then create view:

CREATE MATERIALIZED VIEW mv AS SELECT DATE_TRUNC('month', minio_source.operation_time)::date AS month, avg(minio_source.amount) avg_expenses FROM minio_source GROUP BY DATE_TRUNC('month', minio_source.operation_time);

Please create table and sink:

CREATE TABLE stats (
    month DATE,
    avg_expenses NUMERIC
);

CREATE SINK sql_sink INTO stats FROM mv WITH (type = 'append-only', force_append_only = 'true');

Then upload file expanses.ndjson with content:

{"id":1,"name":"Adair","amount":75.48,"operation_time":"2024-05-04T01:16:02Z"}
{"id":2,"name":"Galven","amount":54.04,"operation_time":"2024-05-13T18:04:29Z"}
{"id":3,"name":"Birgit","amount":85.01,"operation_time":"2023-12-10T22:02:56Z"}
{"id":4,"name":"Lona","amount":33.81,"operation_time":"2024-07-26T12:42:21Z"}
{"id":5,"name":"Ned","amount":66.47,"operation_time":"2024-03-18T07:09:41Z"}
{"id":6,"name":"Clarice","amount":30.65,"operation_time":"2024-04-28T16:54:03Z"}
{"id":7,"name":"Sherill","amount":84.0,"operation_time":"2023-08-21T02:53:07Z"}
{"id":8,"name":"Dyan","amount":13.1,"operation_time":"2024-01-20T01:36:26Z"}
{"id":9,"name":"Cordelia","amount":94.45,"operation_time":"2023-09-07T16:14:03Z"}
{"id":10,"name":"Georgiana","amount":75.13,"operation_time":"2023-09-26T05:14:53Z"}

Please run sql after few seconds SELECT * FROM stats;

Expected behavior

I expect to see new records in table stats.

How did you deploy RisingWave?

docker-compose up, docker compose is:

---
x-image: &image
  image: ${RW_IMAGE:-risingwavelabs/risingwave:v1.10.0}
services:
  risingwave-standalone:
    <<: *image
    command: "standalone --meta-opts=\" \
                    --listen-addr 0.0.0.0:5690 \
                    --advertise-addr 0.0.0.0:5690 \
                    --dashboard-host 0.0.0.0:5691 \
                    --prometheus-host 0.0.0.0:1250 \
                    --prometheus-endpoint http://prometheus-0:9500 \
                    --backend sql \
                    --sql-endpoint postgres://postgres:@postgres-0:5432/metadata \
                    --state-store hummock+minio://hummockadmin:hummockadmin@minio-0:9301/hummock001 \
                    --data-directory hummock_001 \
                    --config-path /risingwave.toml\" \
                 --compute-opts=\" \
                    --config-path /risingwave.toml \
                    --listen-addr 0.0.0.0:5688 \
                    --prometheus-listener-addr 0.0.0.0:1250 \
                    --advertise-addr 0.0.0.0:5688 \
                    --async-stack-trace verbose \
                    #--parallelism 4 \
                    #--total-memory-bytes 8589934592 \
                    --role both \
                    --meta-address http://0.0.0.0:5690\" \
                 --frontend-opts=\" \
                   --config-path /risingwave.toml \
                   --listen-addr 0.0.0.0:4566 \
                   --advertise-addr 0.0.0.0:4566 \
                   --prometheus-listener-addr 0.0.0.0:1250 \
                   --health-check-listener-addr 0.0.0.0:6786 \
                   --meta-addr http://0.0.0.0:5690\" \
                 --compactor-opts=\" \
                   --listen-addr 0.0.0.0:6660 \
                   --prometheus-listener-addr 0.0.0.0:1250 \
                   --advertise-addr 0.0.0.0:6660 \
                   --meta-address http://0.0.0.0:5690\""
    expose:
      - "6660"
      - "4566"
      - "5688"
      - "5690"
      - "1250"
      - "5691"
    ports:
      - "4566:4566"
      - "5690:5690"
      - "5691:5691"
      - "1250:1250"
    depends_on:
      - postgres-0
      - minio-0
    volumes:
      - "./risingwave.toml:/risingwave.toml"
    environment:
      RUST_BACKTRACE: "1"
      # If ENABLE_TELEMETRY is not set, telemetry will start by default
      ENABLE_TELEMETRY: ${ENABLE_TELEMETRY:-true}
      RW_TELEMETRY_TYPE: ${RW_TELEMETRY_TYPE:-"docker-compose"}
      RW_SECRET_STORE_PRIVATE_KEY_HEX: ${RW_SECRET_STORE_PRIVATE_KEY_HEX:-0123456789abcdef}
    container_name: risingwave-standalone
    healthcheck:
      test:
        - CMD-SHELL
        - bash -c 'printf \"GET / HTTP/1.1\n\n\" > /dev/tcp/127.0.0.1/6660; exit $$?;'
        - bash -c 'printf \"GET / HTTP/1.1\n\n\" > /dev/tcp/127.0.0.1/5688; exit $$?;'
        - bash -c 'printf \"GET / HTTP/1.1\n\n\" > /dev/tcp/127.0.0.1/4566; exit $$?;'
        - bash -c 'printf \"GET / HTTP/1.1\n\n\" > /dev/tcp/127.0.0.1/5690; exit $$?;'
      interval: 1s
      timeout: 5s
    restart: always
    deploy:
      resources:
        limits:
          memory: 28G
        reservations:
          memory: 28G

  postgres-0:
    image: "postgres:15-alpine"
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
      - POSTGRES_USER=postgres
      - POSTGRES_DB=metadata
      - POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
    expose:
      - "5432"
    ports:
      - "8432:5432"
    volumes:
      - "postgres-0:/var/lib/postgresql/data"
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 2s
      timeout: 5s
      retries: 5
    restart: always

  grafana-0:
    image: "grafana/grafana-oss:latest"
    command: [ ]
    expose:
      - "3001"
    ports:
      - "3001:3001"
    depends_on: [ ]
    volumes:
      - "grafana-0:/var/lib/grafana"
      - "./grafana.ini:/etc/grafana/grafana.ini"
      - "./grafana-risedev-datasource.yml:/etc/grafana/provisioning/datasources/grafana-risedev-datasource.yml"
      - "./grafana-risedev-dashboard.yml:/etc/grafana/provisioning/dashboards/grafana-risedev-dashboard.yml"
      - "./dashboards:/dashboards"
    environment: { }
    container_name: grafana-0
    healthcheck:
      test:
        - CMD-SHELL
        - bash -c 'printf \"GET / HTTP/1.1\n\n\" > /dev/tcp/127.0.0.1/3001; exit $$?;'
      interval: 1s
      timeout: 5s
      retries: 5
    restart: always

  minio-0:
    image: "quay.io/minio/minio:latest"
    command:
      - server
      - "--address"
      - "0.0.0.0:9301"
      - "--console-address"
      - "0.0.0.0:9400"
      - /data
    expose:
      - "9301"
      - "9400"
    ports:
      - "9301:9301"
      - "9400:9400"
    depends_on: [ ]
    volumes:
      - "minio-0:/data"
    entrypoint: "

      /bin/sh -c '

      set -e

      mkdir -p \"/data/hummock001\"

      /usr/bin/docker-entrypoint.sh \"$$0\" \"$$@\"

      '"
    environment:
      MINIO_CI_CD: "1"
      MINIO_PROMETHEUS_AUTH_TYPE: public
      MINIO_PROMETHEUS_URL: "http://prometheus-0:9500"
      MINIO_ROOT_PASSWORD: hummockadmin
      MINIO_ROOT_USER: hummockadmin
      MINIO_DOMAIN: "minio-0"
    container_name: minio-0
    healthcheck:
      test:
        - CMD-SHELL
        - bash -c 'printf \"GET / HTTP/1.1\n\n\" > /dev/tcp/127.0.0.1/9301; exit $$?;'
      interval: 1s
      timeout: 5s
      retries: 5
    restart: always

  prometheus-0:
    image: "prom/prometheus:latest"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      - "--web.listen-address=0.0.0.0:9500"
      - "--storage.tsdb.retention.time=30d"
    expose:
      - "9500"
    ports:
      - "9500:9500"
    depends_on: [ ]
    volumes:
      - "prometheus-0:/prometheus"
      - "./prometheus.yaml:/etc/prometheus/prometheus.yml"
    environment: { }
    container_name: prometheus-0
    healthcheck:
      test:
        - CMD-SHELL
        - sh -c 'printf "GET /-/healthy HTTP/1.0\n\n" | nc localhost 9500; exit $$?;'
      interval: 1s
      timeout: 5s
      retries: 5
    restart: always

  message_queue:
    image: "docker.vectorized.io/vectorized/redpanda:latest"
    command:
      - redpanda
      - start
      - "--smp"
      - "1"
      - "--reserve-memory"
      - 0M
      - "--memory"
      - 4G
      - "--overprovisioned"
      - "--node-id"
      - "0"
      - "--check=false"
      - "--kafka-addr"
      - "PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092"
      - "--advertise-kafka-addr"
      - "PLAINTEXT://message_queue:29092,OUTSIDE://localhost:9092"
    expose:
      - "29092"
      - "9092"
      - "9644"
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9644:9644"
      - "8081:8081"
    depends_on: [ ]
    volumes:
      - "message_queue:/var/lib/redpanda/data"
    environment: { }
    container_name: message_queue
    healthcheck:
      test: curl -f localhost:9644/v1/status/ready
      interval: 1s
      timeout: 5s
      retries: 5
    restart: always
volumes:
  postgres-0:
    external: false
  grafana-0:
    external: false
  minio-0:
    external: false
  prometheus-0:
    external: false
  message_queue:
    external: false

The version of RisingWave

RisingWave 1.10.

Additional context

No response

hzxa21 commented 1 month ago

Would you mind sharing the logs for the compute pods?

adrianhaj commented 1 month ago

Would you mind sharing the logs for the compute pods?

Logs:

risingwave-standalone  | opentelemetry tracing will be exported to `http://0.0.0.0:5690/` if enabled
risingwave-standalone  | 2024-08-12T06:11:40.706343935Z  INFO main risingwave_rt::panic_hook: coredump on panic is likely to be enabled limit=18446744073709551615
risingwave-standalone  | 2024-08-12T06:11:40.706776604Z  INFO main risingwave_cmd_all::standalone: launching Risingwave in standalone mode
risingwave-standalone  | 2024-08-12T06:11:40.706798312Z  INFO main risingwave_cmd_all::standalone: starting meta-node thread with cli args: MetaNodeOpts { listen_addr: "0.0.0.0:5690", advertise_addr: "0.0.0.0:5690", dashboard_host: Some("0.0.0.0:5691"), prometheus_listener_addr: Some("0.0.0.0:1250"), etcd_endpoints: "", etcd_auth: false, etcd_username: "", etcd_password: [REDACTED alloc::string::String], sql_endpoint: Some([REDACTED alloc::string::String]), sql_username: "", sql_password: [REDACTED alloc::string::String], sql_database: "", prometheus_endpoint: None, prometheus_selector: None, privatelink_endpoint_default_tags: None, vpc_id: None, security_group_id: None, config_path: "/risingwave.toml", backend: Some(Sql), barrier_interval_ms: None, sstable_size_mb: None, block_size_kb: None, bloom_false_positive: None, state_store: Some("hummock+minio://mlops:password@minio:9000/hummock001"), data_directory: Some("hummock_001"), do_not_config_object_storage_lifecycle: None, backup_storage_url: None, backup_storage_directory: None, heap_profiling_dir: None, dangerous_max_idle_secs: None, connector_rpc_endpoint: None }
risingwave-standalone  | 2024-08-12T06:11:40.706830562Z  INFO main risingwave_cmd_all::standalone: waiting for meta service to be ready...
risingwave-standalone  | 2024-08-12T06:11:40.706908271Z  INFO rw-main risingwave_meta_node: Starting meta node
risingwave-standalone  | 2024-08-12T06:11:40.706934313Z  INFO rw-main risingwave_meta_node: > options: MetaNodeOpts { listen_addr: "0.0.0.0:5690", advertise_addr: "0.0.0.0:5690", dashboard_host: Some("0.0.0.0:5691"), prometheus_listener_addr: Some("0.0.0.0:1250"), etcd_endpoints: "", etcd_auth: false, etcd_username: "", etcd_password: [REDACTED alloc::string::String], sql_endpoint: Some([REDACTED alloc::string::String]), sql_username: "", sql_password: [REDACTED alloc::string::String], sql_database: "", prometheus_endpoint: None, prometheus_selector: None, privatelink_endpoint_default_tags: None, vpc_id: None, security_group_id: None, config_path: "/risingwave.toml", backend: Some(Sql), barrier_interval_ms: None, sstable_size_mb: None, block_size_kb: None, bloom_false_positive: None, state_store: Some("hummock+minio://mlops:password@minio:9000/hummock001"), data_directory: Some("hummock_001"), do_not_config_object_storage_lifecycle: None, backup_storage_url: None, backup_storage_directory: None, heap_profiling_dir: None, dangerous_max_idle_secs: None, connector_rpc_endpoint: None }
risingwave-standalone  | 2024-08-12T06:11:40.707586525Z  INFO rw-main risingwave_meta_node: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: Info, telemetry_enabled: true, heap_profiling: HeapProfilingConfig { enable_auto: true, threshold_auto: 0.9, dir: "./" }, grpc_max_reset_stream: 200, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 86400, full_gc_interval_sec: 86400, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, vacuum_spin_interval_ms: 10, hummock_version_checkpoint_interval_sec: 30, enable_hummock_data_archive: false, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 60, disable_recovery: false, disable_automatic_parallelism_control: false, parallelism_control_batch_size: 10, parallelism_control_trigger_period_sec: 10, parallelism_control_trigger_first_delay_sec: 30, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Sql, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_tombstone_reclaim_compaction_interval_sec: 600, periodic_split_compact_group_interval_sec: 10, move_table_size_limit: 10737418240, split_group_size_limit: 68719476736, cut_table_size_limit: 1073741824, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 16, table_write_throughput_threshold: 16777216, min_table_split_write_throughput: 4194304, compaction_task_max_heartbeat_interval_secs: 30, compaction_task_max_progress_interval_secs: 600, compaction_config: CompactionConfig { max_bytes_for_level_base: 536870912, max_bytes_for_level_multiplier: 5, max_compaction_bytes: 2147483648, sub_level_max_compaction_bytes: 134217728, level0_tier_compact_file_number: 12, target_file_size_base: 33554432, compaction_filter_mask: 6, max_sub_compaction: 4, level0_stop_write_threshold_sub_level_number: 300, level0_sub_level_compact_level_count: 3, level0_overlapping_sub_level_compact_level_count: 12, max_space_reclaim_bytes: 536870912, level0_max_compact_file_number: 100, tombstone_reclaim_ratio: 40, enable_emergency_picker: true, max_level: 6 }, hybrid_partition_vnode_count: 4, compact_task_table_size_partition_threshold_low: 134217728, compact_task_table_size_partition_threshold_high: 536870912, event_log_enabled: true, event_log_channel_max_size: 10, developer: MetaDeveloperConfig { cached_traces_num: 256, cached_traces_memory_limit_bytes: 134217728, enable_trivial_move: true, enable_check_task_level_overlap: false, max_trivial_move_task_count_per_loop: 256, max_get_task_probe_times: 5 }, enable_dropped_column_reclaim: false, secret_store_private_key: [100, 101, 109, 111, 45, 115, 101, 99, 114, 101, 116, 45, 112, 114, 105, 118, 97, 116, 101, 45, 107, 101, 121] }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024, exchange_connection_pool_size: None }, distributed_query_limit: None, max_batch_queries_per_frontend_node: None, enable_barrier_read: false, statement_timeout_in_sec: 3600, unrecognized: {}, frontend_compute_runtime_worker_threads: 4, mask_worker_temporary_secs: 30, redact_sql_option_keywords: ["credential", "key", "password", "private", "secret", "token"], enable_spill: true }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: None, async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { enable_executor_row_count: false, connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, exchange_concurrent_dispatchers: 0, dml_channel_initial_permits: 32768, hash_agg_max_dirty_groups_heap_size: 67108864, memory_controller_threshold_aggressive: 0.9, memory_controller_threshold_graceful: 0.8, memory_controller_threshold_stable: 0.7, memory_controller_eviction_factor_aggressive: 2.0, memory_controller_eviction_factor_graceful: 1.5, memory_controller_eviction_factor_stable: 1.0, memory_controller_sequence_tls_step: 128, memory_controller_sequence_tls_lag: 32, enable_arrangement_backfill: true, high_join_amplification_threshold: 2048, exchange_connection_pool_size: Some(1) }, unique_user_stream_errors: 10, unsafe_enable_strict_consistency: true, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, shared_buffer_min_batch_flush_size_mb: 800, imm_merge_threshold: 0, write_conflict_detection_enabled: false, cache: CacheConfig { block_cache_capacity_mb: None, block_cache_shard_num: None, block_cache_eviction: Lru { high_priority_ratio_in_percent: None }, meta_cache_capacity_mb: None, meta_cache_shard_num: None, meta_cache_eviction: Lru { high_priority_ratio_in_percent: None } }, block_cache_capacity_mb: None, meta_cache_capacity_mb: None, high_priority_ratio_in_percent: None, prefetch_buffer_capacity_mb: None, max_prefetch_block_number: 16, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, compactor_max_task_multiplier: 3.0, compactor_memory_available_proportion: 0.8, sstable_id_remote_fetch_number: 10, min_sstable_size_mb: 32, data_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, meta_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, cache_refill: CacheRefillConfig { data_refill_levels: [], timeout_ms: 6000, concurrency: 10, unit: 64, threshold: 0.5, recent_filter_layers: 6, recent_filter_rotate_interval_ms: 10000, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 0, max_version_pinning_duration_sec: 10800, compactor_max_sst_key_count: 2097152, compact_iter_recreate_timeout_ms: 600000, compactor_max_sst_size: 536870912, enable_fast_compaction: true, check_compaction_result: false, max_preload_io_retry_times: 3, compactor_fast_max_compact_delete_ratio: 40, compactor_fast_max_compact_task_size: 2147483648, compactor_iter_max_io_retry_times: 8, unrecognized: {}, mem_table_spill_threshold: 4194304, compactor_concurrent_uploading_sst_count: None, object_store: ObjectStoreConfig { set_atomic_write_dir: false, retry: ObjectStoreRetryConfig { req_backoff_interval_ms: 1000, req_backoff_max_delay_ms: 10000, req_backoff_factor: 2, upload_attempt_timeout_ms: 8000, upload_retry_attempts: 3, streaming_upload_attempt_timeout_ms: 5000, streaming_upload_retry_attempts: 3, read_attempt_timeout_ms: 8000, read_retry_attempts: 3, streaming_read_attempt_timeout_ms: 3000, streaming_read_retry_attempts: 3, metadata_attempt_timeout_ms: 60000, metadata_retry_attempts: 3, delete_attempt_timeout_ms: 5000, delete_retry_attempts: 3, delete_objects_attempt_timeout_ms: 5000, delete_objects_retry_attempts: 3, list_attempt_timeout_ms: 600000, list_retry_attempts: 3 }, s3: S3ObjectStoreConfig { keepalive_ms: Some(600000), recv_buffer_size: Some(2097152), send_buffer_size: None, nodelay: Some(true), retry_unknown_service_error: false, identity_resolution_timeout_s: 5, developer: S3ObjectStoreDeveloperConfig { retry_unknown_service_error: false, retryable_service_error_codes: ["SlowDown", "TooManyRequests"], use_opendal: false } }, opendal_upload_concurrency: 8, opendal_writer_abort_on_err: false, upload_part_size: 16777216 } }, unrecognized: {} }
risingwave-standalone  | 2024-08-12T06:11:40.707636317Z  INFO rw-main risingwave_meta_node: > version: 1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c)
risingwave-standalone  | 2024-08-12T06:11:40.707874901Z  INFO rw-main risingwave_common_heap_profiling::profiler: Auto memory dump is disabled.
risingwave-standalone  | 2024-08-12T06:11:40.70788186Z  INFO rw-main risingwave_common::config: initialize backup_storage_url based on state_store
risingwave-standalone  | 2024-08-12T06:11:40.70788536Z  INFO rw-main risingwave_common::config: initialize backup_storage_directory based on data_directory
risingwave-standalone  | 2024-08-12T06:11:40.721921516Z  INFO rw-main risingwave_meta::rpc::election::sql: initializing database for Sql backend election client
risingwave-standalone  | 2024-08-12T06:11:40.72509545Z  INFO rw-main risingwave_meta_node: Meta server listening at 0.0.0.0:5690
risingwave-standalone  | 2024-08-12T06:11:40.731622609Z  INFO rw-main risingwave_meta_node::server: Shutting down follower services
risingwave-standalone  | 2024-08-12T06:11:40.731726276Z  INFO rw-main risingwave_meta_node::server: Defining leader services
risingwave-standalone  | 2024-08-12T06:11:40.73384312Z  INFO rw-main sea_orm_migration::migrator: Applying all pending migrations
risingwave-standalone  | 2024-08-12T06:11:40.735294211Z  INFO rw-main sea_orm_migration::migrator: No pending migrations
risingwave-standalone  | 2024-08-12T06:11:40.739643859Z  INFO rw-main risingwave_meta::controller::system_param: system parameters: SystemParams { barrier_interval_ms: Some(1000), checkpoint_frequency: Some(1), sstable_size_mb: Some(256), block_size_kb: Some(64), bloom_false_positive: Some(0.001), state_store: Some("hummock+minio://mlops:password@minio:9000/hummock001"), data_directory: Some("hummock_001"), backup_storage_url: Some("minio://mlops:password@minio:9000/hummock001"), backup_storage_directory: Some("hummock_001/backup"), telemetry_enabled: None, parallel_compact_size_mb: Some(512), max_concurrent_creating_streaming_jobs: Some(1), pause_on_next_bootstrap: Some(false), wasm_storage_url: None, enable_tracing: Some(false) }
risingwave-standalone  | 2024-08-12T06:11:40.739721776Z  INFO rw-main risingwave_rt::logger: opentelemetry tracing disabled
risingwave-standalone  | 2024-08-12T06:11:40.742540666Z  INFO rw-main risingwave_meta::controller::session_params: session parameters init_params=SessionConfig { implicit_flush: false, create_compaction_group_for_mv: false, query_mode: Auto, extra_float_digits: 1, application_name: "", date_style: "", batch_enable_lookup_join: true, batch_enable_sort_agg: true, batch_enable_distributed_dml: false, max_split_range_gap: 8, search_path: SearchPath { origin_str: "\"$user\", public", path: ["pg_catalog", "rw_catalog", "\"$user\"", "public"], real_path: ["\"$user\"", "public"] }, visibility_mode: Default, transaction_isolation: ReadCommitted, query_epoch: ConfigNonZeroU64(None), timezone: "UTC", streaming_parallelism: ConfigNonZeroU64(None), streaming_enable_delta_join: false, streaming_enable_bushy_join: true, streaming_use_arrangement_backfill: true, streaming_allow_jsonb_in_stream_key: false, enable_join_ordering: true, enable_two_phase_agg: true, force_two_phase_agg: false, enable_share_plan: true, force_split_distinct_agg: false, interval_style: "", batch_parallelism: ConfigNonZeroU64(None), server_version: "13.14.0", server_version_num: 130014, client_min_messages: "notice", client_encoding: "UTF8", sink_decouple: Default, synchronize_seqscans: false, statement_timeout: 0, idle_in_transaction_session_timeout: 60000, lock_timeout: 0, cdc_source_wait_streaming_start_timeout: 30, row_security: true, standard_conforming_strings: "on", streaming_rate_limit: -1, streaming_over_window_cache_policy: Full, background_ddl: false, rw_enable_shared_source: false, server_encoding: "UTF8", bytea_output: "hex" }
risingwave-standalone  | 2024-08-12T06:11:40.750604166Z  WARN rw-main aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
risingwave-standalone  | 2024-08-12T06:11:40.750767001Z  INFO rw-main risingwave_common_metrics::monitor::connection: monitoring connector hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector> with type S3
risingwave-standalone  | 2024-08-12T06:11:40.75548115Z  INFO rw-main lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took Ok(11.291µs))
risingwave-standalone  | 2024-08-12T06:11:40.762165144Z  WARN rw-main risingwave_object_store::object::s3: Failed to configure life cycle rule for S3 bucket: "hummock001". It is recommended to configure it manually to avoid unnecessary storage cost.
risingwave-standalone  | 2024-08-12T06:11:40.763958236Z  INFO rw-main risingwave_meta::hummock::manager: init hummock version checkpoint
risingwave-standalone  | 2024-08-12T06:11:40.771784652Z  INFO rw-main risingwave_meta::barrier::schedule: Starting barrier scheduler with: checkpoint_frequency=1
risingwave-standalone  | 2024-08-12T06:11:40.77240128Z  INFO rw-main risingwave_meta_dashboard: using embedded dashboard assets
risingwave-standalone  | 2024-08-12T06:11:40.775212879Z  WARN rw-main aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
risingwave-standalone  | 2024-08-12T06:11:40.775338463Z  INFO rw-main risingwave_common_metrics::monitor::connection: monitoring connector hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector> with type S3
risingwave-standalone  | 2024-08-12T06:11:40.779244441Z  INFO rw-main lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took Ok(7.75µs))
risingwave-standalone  | 2024-08-12T06:11:40.780141863Z  INFO rw-main risingwave_meta::backup_restore::backup_manager: backup manager initialized: url=minio://mlops:password@minio:9000/hummock001, dir=hummock_001/backup
risingwave-standalone  | 2024-08-12T06:11:40.780299988Z  INFO rw-main risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1250
risingwave-standalone  | 2024-08-12T06:11:40.780575157Z  INFO rw-main risingwave_meta::hummock::manager::timer_task: Hummock timer task tracing [GroupSplit interval 10 sec] [CheckDeadTask interval 300 sec] [Report interval 20 sec] [CompactionHeartBeat interval 1 sec]
risingwave-standalone  | 2024-08-12T06:11:40.780634449Z  INFO rw-main risingwave_meta::stream::scale: Automatic parallelism scale-out is enabled for streaming jobs
risingwave-standalone  | 2024-08-12T06:11:40.780703241Z  INFO rw-main risingwave_meta_node::server: Assigned cluster id ClusterId("2263ba2d-2d3a-4cdf-bf4c-d1e0833de20e")
risingwave-standalone  | 2024-08-12T06:11:40.780776449Z  INFO rw-main risingwave_meta_node::server: Starting meta services
risingwave-standalone  | 2024-08-12T06:11:40.780737116Z  INFO rw-main risingwave_meta::barrier: Starting barrier manager with: interval=1s, enable_recovery=true, in_flight_barrier_nums=10000
risingwave-standalone  | 2024-08-12T06:11:40.780759199Z  INFO rw-main risingwave_common::telemetry::report: start telemetry reporting
risingwave-standalone  | 2024-08-12T06:11:40.780765158Z  INFO rw-main risingwave_meta::stream::scale: starting automatic parallelism control monitor
risingwave-standalone  | 2024-08-12T06:11:40.780990575Z  INFO rw-main risingwave_common::telemetry::report: Telemetry tracking_id is not set, event reporting disabled
risingwave-standalone  | 2024-08-12T06:11:40.781067076Z  INFO rw-main bootstrap_recovery{prev_epoch=0}: risingwave_meta::barrier::recovery: recovery start!
risingwave-standalone  | 2024-08-12T06:11:40.787690027Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: recovering mview progress
risingwave-standalone  | 2024-08-12T06:11:40.789365328Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: recovered mview progress
risingwave-standalone  | 2024-08-12T06:11:40.796872492Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-stream-client
risingwave-standalone  | 2024-08-12T06:11:40.797145952Z  WARN rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: reset compute nodes failed error=failed to create RPC client to 0.0.0.0:5688: transport error: error trying to connect: tcp connect error: Connection refused (os error 111)
risingwave-standalone  | 2024-08-12T06:11:40.807491297Z  INFO    main risingwave_cmd_all::standalone: starting compute-node thread with cli args: ComputeNodeOpts { listen_addr: "0.0.0.0:5688", advertise_addr: Some("0.0.0.0:5688"), prometheus_listener_addr: "0.0.0.0:1250", meta_address: List([http://127.0.0.1:5690/]), config_path: "/risingwave.toml", total_memory_bytes: 16379089305, reserved_memory_bytes: None, parallelism: 8, role: Both, metrics_level: None, data_file_cache_dir: None, meta_file_cache_dir: None, async_stack_trace: Some(ReleaseVerbose), heap_profiling_dir: None, connector_rpc_endpoint: None }
risingwave-standalone  | 2024-08-12T06:11:40.807508922Z  INFO    main risingwave_cmd_all::standalone: starting frontend-node thread with cli args: FrontendOpts { listen_addr: "0.0.0.0:4566", advertise_addr: Some("0.0.0.0:4566"), meta_addr: List([http://0.0.0.0:5690/]), prometheus_listener_addr: "0.0.0.0:1250", health_check_listener_addr: "0.0.0.0:6786", config_path: "/risingwave.toml", metrics_level: None, enable_barrier_read: None }
risingwave-standalone  | 2024-08-12T06:11:40.807512047Z  INFO    main risingwave_cmd_all::standalone: starting compactor-node thread with cli args: CompactorOpts { listen_addr: "0.0.0.0:6660", advertise_addr: Some("0.0.0.0:6660"), port: None, prometheus_listener_addr: "0.0.0.0:1250", meta_address: List([http://0.0.0.0:5690/]), compaction_worker_threads_number: None, config_path: "", metrics_level: None, async_stack_trace: None, heap_profiling_dir: None, compactor_mode: None, proxy_rpc_endpoint: "" }
risingwave-standalone  | 2024-08-12T06:11:40.807565339Z  INFO rw-main risingwave_compute: options: ComputeNodeOpts { listen_addr: "0.0.0.0:5688", advertise_addr: Some("0.0.0.0:5688"), prometheus_listener_addr: "0.0.0.0:1250", meta_address: List([http://127.0.0.1:5690/]), config_path: "/risingwave.toml", total_memory_bytes: 16379089305, reserved_memory_bytes: None, parallelism: 8, role: Both, metrics_level: None, data_file_cache_dir: None, meta_file_cache_dir: None, async_stack_trace: Some(ReleaseVerbose), heap_profiling_dir: None, connector_rpc_endpoint: None }
risingwave-standalone  | 2024-08-12T06:11:40.807600215Z  INFO rw-main risingwave_compactor: Compactor node options: CompactorOpts { listen_addr: "0.0.0.0:6660", advertise_addr: Some("0.0.0.0:6660"), port: None, prometheus_listener_addr: "0.0.0.0:1250", meta_address: List([http://0.0.0.0:5690/]), compaction_worker_threads_number: None, config_path: "", metrics_level: None, async_stack_trace: None, heap_profiling_dir: None, compactor_mode: None, proxy_rpc_endpoint: "" }
risingwave-standalone  | 2024-08-12T06:11:40.807632006Z  INFO rw-main risingwave_compactor: meta address: http://0.0.0.0:5690/
risingwave-standalone  | 2024-08-12T06:11:40.80764609Z  INFO rw-main risingwave_compactor:  address is 0.0.0.0:6660
risingwave-standalone  | 2024-08-12T06:11:40.807656382Z  WARN rw-main risingwave_common::config: risingwave.toml not found, using default config.
risingwave-standalone  | 2024-08-12T06:11:40.80766109Z  INFO rw-main risingwave_compactor::server: Starting compactor node
risingwave-standalone  | 2024-08-12T06:11:40.807662423Z  INFO rw-main risingwave_compactor::server: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: Info, telemetry_enabled: true, heap_profiling: HeapProfilingConfig { enable_auto: true, threshold_auto: 0.9, dir: "./" }, grpc_max_reset_stream: 200, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 86400, full_gc_interval_sec: 86400, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, vacuum_spin_interval_ms: 10, hummock_version_checkpoint_interval_sec: 30, enable_hummock_data_archive: false, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 60, disable_recovery: false, disable_automatic_parallelism_control: false, parallelism_control_batch_size: 10, parallelism_control_trigger_period_sec: 10, parallelism_control_trigger_first_delay_sec: 30, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Mem, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_tombstone_reclaim_compaction_interval_sec: 600, periodic_split_compact_group_interval_sec: 10, move_table_size_limit: 10737418240, split_group_size_limit: 68719476736, cut_table_size_limit: 1073741824, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 16, table_write_throughput_threshold: 16777216, min_table_split_write_throughput: 4194304, compaction_task_max_heartbeat_interval_secs: 30, compaction_task_max_progress_interval_secs: 600, compaction_config: CompactionConfig { max_bytes_for_level_base: 536870912, max_bytes_for_level_multiplier: 5, max_compaction_bytes: 2147483648, sub_level_max_compaction_bytes: 134217728, level0_tier_compact_file_number: 12, target_file_size_base: 33554432, compaction_filter_mask: 6, max_sub_compaction: 4, level0_stop_write_threshold_sub_level_number: 300, level0_sub_level_compact_level_count: 3, level0_overlapping_sub_level_compact_level_count: 12, max_space_reclaim_bytes: 536870912, level0_max_compact_file_number: 100, tombstone_reclaim_ratio: 40, enable_emergency_picker: true, max_level: 6 }, hybrid_partition_vnode_count: 4, compact_task_table_size_partition_threshold_low: 134217728, compact_task_table_size_partition_threshold_high: 536870912, event_log_enabled: true, event_log_channel_max_size: 10, developer: MetaDeveloperConfig { cached_traces_num: 256, cached_traces_memory_limit_bytes: 134217728, enable_trivial_move: true, enable_check_task_level_overlap: false, max_trivial_move_task_count_per_loop: 256, max_get_task_probe_times: 5 }, enable_dropped_column_reclaim: false, secret_store_private_key: [100, 101, 109, 111, 45, 115, 101, 99, 114, 101, 116, 45, 112, 114, 105, 118, 97, 116, 101, 45, 107, 101, 121] }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024, exchange_connection_pool_size: None }, distributed_query_limit: None, max_batch_queries_per_frontend_node: None, enable_barrier_read: false, statement_timeout_in_sec: 3600, unrecognized: {}, frontend_compute_runtime_worker_threads: 4, mask_worker_temporary_secs: 30, redact_sql_option_keywords: ["credential", "key", "password", "private", "secret", "token"], enable_spill: true }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: None, async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { enable_executor_row_count: false, connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, exchange_concurrent_dispatchers: 0, dml_channel_initial_permits: 32768, hash_agg_max_dirty_groups_heap_size: 67108864, memory_controller_threshold_aggressive: 0.9, memory_controller_threshold_graceful: 0.8, memory_controller_threshold_stable: 0.7, memory_controller_eviction_factor_aggressive: 2.0, memory_controller_eviction_factor_graceful: 1.5, memory_controller_eviction_factor_stable: 1.0, memory_controller_sequence_tls_step: 128, memory_controller_sequence_tls_lag: 32, enable_arrangement_backfill: true, high_join_amplification_threshold: 2048, exchange_connection_pool_size: Some(1) }, unique_user_stream_errors: 10, unsafe_enable_strict_consistency: true, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, shared_buffer_min_batch_flush_size_mb: 800, imm_merge_threshold: 0, write_conflict_detection_enabled: false, cache: CacheConfig { block_cache_capacity_mb: None, block_cache_shard_num: None, block_cache_eviction: Lru { high_priority_ratio_in_percent: None }, meta_cache_capacity_mb: None, meta_cache_shard_num: None, meta_cache_eviction: Lru { high_priority_ratio_in_percent: None } }, block_cache_capacity_mb: None, meta_cache_capacity_mb: None, high_priority_ratio_in_percent: None, prefetch_buffer_capacity_mb: None, max_prefetch_block_number: 16, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, compactor_max_task_multiplier: 3.0, compactor_memory_available_proportion: 0.8, sstable_id_remote_fetch_number: 10, min_sstable_size_mb: 32, data_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, meta_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, cache_refill: CacheRefillConfig { data_refill_levels: [], timeout_ms: 6000, concurrency: 10, unit: 64, threshold: 0.5, recent_filter_layers: 6, recent_filter_rotate_interval_ms: 10000, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 0, max_version_pinning_duration_sec: 10800, compactor_max_sst_key_count: 2097152, compact_iter_recreate_timeout_ms: 600000, compactor_max_sst_size: 536870912, enable_fast_compaction: true, check_compaction_result: false, max_preload_io_retry_times: 3, compactor_fast_max_compact_delete_ratio: 40, compactor_fast_max_compact_task_size: 2147483648, compactor_iter_max_io_retry_times: 8, unrecognized: {}, mem_table_spill_threshold: 4194304, compactor_concurrent_uploading_sst_count: None, object_store: ObjectStoreConfig { set_atomic_write_dir: false, retry: ObjectStoreRetryConfig { req_backoff_interval_ms: 1000, req_backoff_max_delay_ms: 10000, req_backoff_factor: 2, upload_attempt_timeout_ms: 8000, upload_retry_attempts: 3, streaming_upload_attempt_timeout_ms: 5000, streaming_upload_retry_attempts: 3, read_attempt_timeout_ms: 8000, read_retry_attempts: 3, streaming_read_attempt_timeout_ms: 3000, streaming_read_retry_attempts: 3, metadata_attempt_timeout_ms: 60000, metadata_retry_attempts: 3, delete_attempt_timeout_ms: 5000, delete_retry_attempts: 3, delete_objects_attempt_timeout_ms: 5000, delete_objects_retry_attempts: 3, list_attempt_timeout_ms: 600000, list_retry_attempts: 3 }, s3: S3ObjectStoreConfig { keepalive_ms: Some(600000), recv_buffer_size: Some(2097152), send_buffer_size: None, nodelay: Some(true), retry_unknown_service_error: false, identity_resolution_timeout_s: 5, developer: S3ObjectStoreDeveloperConfig { retry_unknown_service_error: false, retryable_service_error_codes: ["SlowDown", "TooManyRequests"], use_opendal: false } }, opendal_upload_concurrency: 8, opendal_writer_abort_on_err: false, upload_part_size: 16777216 } }, unrecognized: {} }
risingwave-standalone  | 2024-08-12T06:11:40.807695757Z  INFO rw-main risingwave_compactor::server: > debug assertions: off
risingwave-standalone  | 2024-08-12T06:11:40.807696132Z  INFO rw-main risingwave_compute: advertise addr is 0.0.0.0:5688
risingwave-standalone  | 2024-08-12T06:11:40.807697465Z  INFO rw-main risingwave_compactor::server: > version: 1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c)
risingwave-standalone  | 2024-08-12T06:11:40.80770059Z  INFO rw-main risingwave_rpc_client::meta_client: register meta client using strategy: http://0.0.0.0:5690/
risingwave-standalone  | 2024-08-12T06:11:40.80799705Z  INFO rw-main risingwave_rpc_client::meta_client: Connect to meta server http://0.0.0.0:5690/ successfully
risingwave-standalone  | 2024-08-12T06:11:40.808132509Z  INFO rw-main risingwave_compute::server: Starting compute node
risingwave-standalone  | 2024-08-12T06:11:40.808135759Z  INFO rw-main risingwave_compute::server: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: Info, telemetry_enabled: true, heap_profiling: HeapProfilingConfig { enable_auto: true, threshold_auto: 0.9, dir: "./" }, grpc_max_reset_stream: 200, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 86400, full_gc_interval_sec: 86400, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, vacuum_spin_interval_ms: 10, hummock_version_checkpoint_interval_sec: 30, enable_hummock_data_archive: false, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 60, disable_recovery: false, disable_automatic_parallelism_control: false, parallelism_control_batch_size: 10, parallelism_control_trigger_period_sec: 10, parallelism_control_trigger_first_delay_sec: 30, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Mem, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_tombstone_reclaim_compaction_interval_sec: 600, periodic_split_compact_group_interval_sec: 10, move_table_size_limit: 10737418240, split_group_size_limit: 68719476736, cut_table_size_limit: 1073741824, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 16, table_write_throughput_threshold: 16777216, min_table_split_write_throughput: 4194304, compaction_task_max_heartbeat_interval_secs: 30, compaction_task_max_progress_interval_secs: 600, compaction_config: CompactionConfig { max_bytes_for_level_base: 536870912, max_bytes_for_level_multiplier: 5, max_compaction_bytes: 2147483648, sub_level_max_compaction_bytes: 134217728, level0_tier_compact_file_number: 12, target_file_size_base: 33554432, compaction_filter_mask: 6, max_sub_compaction: 4, level0_stop_write_threshold_sub_level_number: 300, level0_sub_level_compact_level_count: 3, level0_overlapping_sub_level_compact_level_count: 12, max_space_reclaim_bytes: 536870912, level0_max_compact_file_number: 100, tombstone_reclaim_ratio: 40, enable_emergency_picker: true, max_level: 6 }, hybrid_partition_vnode_count: 4, compact_task_table_size_partition_threshold_low: 134217728, compact_task_table_size_partition_threshold_high: 536870912, event_log_enabled: true, event_log_channel_max_size: 10, developer: MetaDeveloperConfig { cached_traces_num: 256, cached_traces_memory_limit_bytes: 134217728, enable_trivial_move: true, enable_check_task_level_overlap: false, max_trivial_move_task_count_per_loop: 256, max_get_task_probe_times: 5 }, enable_dropped_column_reclaim: false, secret_store_private_key: [100, 101, 109, 111, 45, 115, 101, 99, 114, 101, 116, 45, 112, 114, 105, 118, 97, 116, 101, 45, 107, 101, 121] }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024, exchange_connection_pool_size: None }, distributed_query_limit: None, max_batch_queries_per_frontend_node: None, enable_barrier_read: false, statement_timeout_in_sec: 3600, unrecognized: {}, frontend_compute_runtime_worker_threads: 4, mask_worker_temporary_secs: 30, redact_sql_option_keywords: ["credential", "key", "password", "private", "secret", "token"], enable_spill: true }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: Some(8), async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { enable_executor_row_count: false, connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, exchange_concurrent_dispatchers: 0, dml_channel_initial_permits: 32768, hash_agg_max_dirty_groups_heap_size: 67108864, memory_controller_threshold_aggressive: 0.9, memory_controller_threshold_graceful: 0.8, memory_controller_threshold_stable: 0.7, memory_controller_eviction_factor_aggressive: 2.0, memory_controller_eviction_factor_graceful: 1.5, memory_controller_eviction_factor_stable: 1.0, memory_controller_sequence_tls_step: 128, memory_controller_sequence_tls_lag: 32, enable_arrangement_backfill: true, high_join_amplification_threshold: 2048, exchange_connection_pool_size: Some(1) }, unique_user_stream_errors: 10, unsafe_enable_strict_consistency: true, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, shared_buffer_min_batch_flush_size_mb: 800, imm_merge_threshold: 0, write_conflict_detection_enabled: false, cache: CacheConfig { block_cache_capacity_mb: None, block_cache_shard_num: None, block_cache_eviction: Lru { high_priority_ratio_in_percent: None }, meta_cache_capacity_mb: None, meta_cache_shard_num: None, meta_cache_eviction: Lru { high_priority_ratio_in_percent: None } }, block_cache_capacity_mb: None, meta_cache_capacity_mb: None, high_priority_ratio_in_percent: None, prefetch_buffer_capacity_mb: None, max_prefetch_block_number: 16, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, compactor_max_task_multiplier: 3.0, compactor_memory_available_proportion: 0.8, sstable_id_remote_fetch_number: 10, min_sstable_size_mb: 32, data_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, meta_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, cache_refill: CacheRefillConfig { data_refill_levels: [], timeout_ms: 6000, concurrency: 10, unit: 64, threshold: 0.5, recent_filter_layers: 6, recent_filter_rotate_interval_ms: 10000, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 0, max_version_pinning_duration_sec: 10800, compactor_max_sst_key_count: 2097152, compact_iter_recreate_timeout_ms: 600000, compactor_max_sst_size: 536870912, enable_fast_compaction: true, check_compaction_result: false, max_preload_io_retry_times: 3, compactor_fast_max_compact_delete_ratio: 40, compactor_fast_max_compact_task_size: 2147483648, compactor_iter_max_io_retry_times: 8, unrecognized: {}, mem_table_spill_threshold: 4194304, compactor_concurrent_uploading_sst_count: None, object_store: ObjectStoreConfig { set_atomic_write_dir: false, retry: ObjectStoreRetryConfig { req_backoff_interval_ms: 1000, req_backoff_max_delay_ms: 10000, req_backoff_factor: 2, upload_attempt_timeout_ms: 8000, upload_retry_attempts: 3, streaming_upload_attempt_timeout_ms: 5000, streaming_upload_retry_attempts: 3, read_attempt_timeout_ms: 8000, read_retry_attempts: 3, streaming_read_attempt_timeout_ms: 3000, streaming_read_retry_attempts: 3, metadata_attempt_timeout_ms: 60000, metadata_retry_attempts: 3, delete_attempt_timeout_ms: 5000, delete_retry_attempts: 3, delete_objects_attempt_timeout_ms: 5000, delete_objects_retry_attempts: 3, list_attempt_timeout_ms: 600000, list_retry_attempts: 3 }, s3: S3ObjectStoreConfig { keepalive_ms: Some(600000), recv_buffer_size: Some(2097152), send_buffer_size: None, nodelay: Some(true), retry_unknown_service_error: false, identity_resolution_timeout_s: 5, developer: S3ObjectStoreDeveloperConfig { retry_unknown_service_error: false, retryable_service_error_codes: ["SlowDown", "TooManyRequests"], use_opendal: false } }, opendal_upload_concurrency: 8, opendal_writer_abort_on_err: false, upload_part_size: 16777216 } }, unrecognized: {} }
risingwave-standalone  | 2024-08-12T06:11:40.808171843Z  INFO rw-main risingwave_compute::server: > debug assertions: off
risingwave-standalone  | 2024-08-12T06:11:40.808177051Z  INFO rw-main risingwave_compute::server: > version: 1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c)
risingwave-standalone  | 2024-08-12T06:11:40.808181259Z  INFO rw-main risingwave_rpc_client::meta_client: register meta client using strategy: http://127.0.0.1:5690/
risingwave-standalone  | 2024-08-12T06:11:40.808397177Z  INFO rw-main risingwave_rpc_client::meta_client: Connect to meta server http://127.0.0.1:5690/ successfully
risingwave-standalone  | 2024-08-12T06:11:40.808478761Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: recovering mview progress
risingwave-standalone  | 2024-08-12T06:11:40.808528011Z  INFO rw-main risingwave_frontend::session: Starting frontend node
risingwave-standalone  | 2024-08-12T06:11:40.808532303Z  INFO rw-main risingwave_frontend::session: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: Info, telemetry_enabled: true, heap_profiling: HeapProfilingConfig { enable_auto: true, threshold_auto: 0.9, dir: "./" }, grpc_max_reset_stream: 200, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 86400, full_gc_interval_sec: 86400, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, vacuum_spin_interval_ms: 10, hummock_version_checkpoint_interval_sec: 30, enable_hummock_data_archive: false, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 60, disable_recovery: false, disable_automatic_parallelism_control: false, parallelism_control_batch_size: 10, parallelism_control_trigger_period_sec: 10, parallelism_control_trigger_first_delay_sec: 30, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Mem, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_tombstone_reclaim_compaction_interval_sec: 600, periodic_split_compact_group_interval_sec: 10, move_table_size_limit: 10737418240, split_group_size_limit: 68719476736, cut_table_size_limit: 1073741824, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 16, table_write_throughput_threshold: 16777216, min_table_split_write_throughput: 4194304, compaction_task_max_heartbeat_interval_secs: 30, compaction_task_max_progress_interval_secs: 600, compaction_config: CompactionConfig { max_bytes_for_level_base: 536870912, max_bytes_for_level_multiplier: 5, max_compaction_bytes: 2147483648, sub_level_max_compaction_bytes: 134217728, level0_tier_compact_file_number: 12, target_file_size_base: 33554432, compaction_filter_mask: 6, max_sub_compaction: 4, level0_stop_write_threshold_sub_level_number: 300, level0_sub_level_compact_level_count: 3, level0_overlapping_sub_level_compact_level_count: 12, max_space_reclaim_bytes: 536870912, level0_max_compact_file_number: 100, tombstone_reclaim_ratio: 40, enable_emergency_picker: true, max_level: 6 }, hybrid_partition_vnode_count: 4, compact_task_table_size_partition_threshold_low: 134217728, compact_task_table_size_partition_threshold_high: 536870912, event_log_enabled: true, event_log_channel_max_size: 10, developer: MetaDeveloperConfig { cached_traces_num: 256, cached_traces_memory_limit_bytes: 134217728, enable_trivial_move: true, enable_check_task_level_overlap: false, max_trivial_move_task_count_per_loop: 256, max_get_task_probe_times: 5 }, enable_dropped_column_reclaim: false, secret_store_private_key: [100, 101, 109, 111, 45, 115, 101, 99, 114, 101, 116, 45, 112, 114, 105, 118, 97, 116, 101, 45, 107, 101, 121] }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024, exchange_connection_pool_size: None }, distributed_query_limit: None, max_batch_queries_per_frontend_node: None, enable_barrier_read: false, statement_timeout_in_sec: 3600, unrecognized: {}, frontend_compute_runtime_worker_threads: 4, mask_worker_temporary_secs: 30, redact_sql_option_keywords: ["credential", "key", "password", "private", "secret", "token"], enable_spill: true }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: None, async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { enable_executor_row_count: false, connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, exchange_concurrent_dispatchers: 0, dml_channel_initial_permits: 32768, hash_agg_max_dirty_groups_heap_size: 67108864, memory_controller_threshold_aggressive: 0.9, memory_controller_threshold_graceful: 0.8, memory_controller_threshold_stable: 0.7, memory_controller_eviction_factor_aggressive: 2.0, memory_controller_eviction_factor_graceful: 1.5, memory_controller_eviction_factor_stable: 1.0, memory_controller_sequence_tls_step: 128, memory_controller_sequence_tls_lag: 32, enable_arrangement_backfill: true, high_join_amplification_threshold: 2048, exchange_connection_pool_size: Some(1) }, unique_user_stream_errors: 10, unsafe_enable_strict_consistency: true, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, shared_buffer_min_batch_flush_size_mb: 800, imm_merge_threshold: 0, write_conflict_detection_enabled: false, cache: CacheConfig { block_cache_capacity_mb: None, block_cache_shard_num: None, block_cache_eviction: Lru { high_priority_ratio_in_percent: None }, meta_cache_capacity_mb: None, meta_cache_shard_num: None, meta_cache_eviction: Lru { high_priority_ratio_in_percent: None } }, block_cache_capacity_mb: None, meta_cache_capacity_mb: None, high_priority_ratio_in_percent: None, prefetch_buffer_capacity_mb: None, max_prefetch_block_number: 16, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, compactor_max_task_multiplier: 3.0, compactor_memory_available_proportion: 0.8, sstable_id_remote_fetch_number: 10, min_sstable_size_mb: 32, data_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, meta_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, flushers: 4, reclaimers: 4, recover_concurrency: 8, insert_rate_limit_mb: 0, indexer_shards: 64, compression: "none", flush_buffer_threshold_mb: None, unrecognized: {} }, cache_refill: CacheRefillConfig { data_refill_levels: [], timeout_ms: 6000, concurrency: 10, unit: 64, threshold: 0.5, recent_filter_layers: 6, recent_filter_rotate_interval_ms: 10000, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 0, max_version_pinning_duration_sec: 10800, compactor_max_sst_key_count: 2097152, compact_iter_recreate_timeout_ms: 600000, compactor_max_sst_size: 536870912, enable_fast_compaction: true, check_compaction_result: false, max_preload_io_retry_times: 3, compactor_fast_max_compact_delete_ratio: 40, compactor_fast_max_compact_task_size: 2147483648, compactor_iter_max_io_retry_times: 8, unrecognized: {}, mem_table_spill_threshold: 4194304, compactor_concurrent_uploading_sst_count: None, object_store: ObjectStoreConfig { set_atomic_write_dir: false, retry: ObjectStoreRetryConfig { req_backoff_interval_ms: 1000, req_backoff_max_delay_ms: 10000, req_backoff_factor: 2, upload_attempt_timeout_ms: 8000, upload_retry_attempts: 3, streaming_upload_attempt_timeout_ms: 5000, streaming_upload_retry_attempts: 3, read_attempt_timeout_ms: 8000, read_retry_attempts: 3, streaming_read_attempt_timeout_ms: 3000, streaming_read_retry_attempts: 3, metadata_attempt_timeout_ms: 60000, metadata_retry_attempts: 3, delete_attempt_timeout_ms: 5000, delete_retry_attempts: 3, delete_objects_attempt_timeout_ms: 5000, delete_objects_retry_attempts: 3, list_attempt_timeout_ms: 600000, list_retry_attempts: 3 }, s3: S3ObjectStoreConfig { keepalive_ms: Some(600000), recv_buffer_size: Some(2097152), send_buffer_size: None, nodelay: Some(true), retry_unknown_service_error: false, identity_resolution_timeout_s: 5, developer: S3ObjectStoreDeveloperConfig { retry_unknown_service_error: false, retryable_service_error_codes: ["SlowDown", "TooManyRequests"], use_opendal: false } }, opendal_upload_concurrency: 8, opendal_writer_abort_on_err: false, upload_part_size: 16777216 } }, unrecognized: {} }
risingwave-standalone  | 2024-08-12T06:11:40.808590761Z  INFO rw-main risingwave_frontend::session: > debug assertions: off
risingwave-standalone  | 2024-08-12T06:11:40.80859272Z  INFO rw-main risingwave_frontend::session: > version: 1.10.0 (0542faf14d9a1871e9385c82748d7102aeeea42c)
risingwave-standalone  | 2024-08-12T06:11:40.80859847Z  INFO rw-main risingwave_frontend::session: advertise addr is 0.0.0.0:4566
risingwave-standalone  | 2024-08-12T06:11:40.808600053Z  INFO rw-main risingwave_rpc_client::meta_client: register meta client using strategy: http://0.0.0.0:5690/
risingwave-standalone  | 2024-08-12T06:11:40.808844888Z  INFO rw-main risingwave_rpc_client::meta_client: Connect to meta server http://0.0.0.0:5690/ successfully
risingwave-standalone  | 2024-08-12T06:11:40.808876346Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: recovered mview progress
risingwave-standalone  | 2024-08-12T06:11:40.810082894Z  INFO rw-main risingwave_rpc_client::meta_client: new meta member joined: http://0.0.0.0:5690/
risingwave-standalone  | 2024-08-12T06:11:40.810093894Z  INFO rw-main risingwave_rpc_client::meta_client: new meta leader http://0.0.0.0:5690/ discovered
risingwave-standalone  | 2024-08-12T06:11:40.814310333Z  INFO rw-main risingwave_compute::server: 
risingwave-standalone  | Memory outline:
risingwave-standalone  | > total_memory: 15.25 GiB
risingwave-standalone  | >     storage_memory: 4.06 GiB
risingwave-standalone  | >         block_cache_capacity: 1.28 GiB
risingwave-standalone  | >         meta_cache_capacity: 1.49 GiB
risingwave-standalone  | >         shared_buffer_capacity: 1.28 GiB
risingwave-standalone  | >         compactor_memory_limit: Not enabled
risingwave-standalone  | >     compute_memory: 6.62 GiB
risingwave-standalone  | >     reserved_memory: 4.58 GiB
risingwave-standalone  | 2024-08-12T06:11:40.81432425Z  INFO rw-main risingwave_compute::server: Assigned worker node id 3
risingwave-standalone  | 2024-08-12T06:11:40.814713335Z  INFO rw-main risingwave_frontend::session: Assigned worker node id 2
risingwave-standalone  | 2024-08-12T06:11:40.814897044Z  INFO rw-main risingwave_rt::logger: opentelemetry tracing disabled
risingwave-standalone  | 2024-08-12T06:11:40.815339713Z  WARN rw-main foyer_storage::store: [store builder]: No device config set. Use `NoneStore` which always returns `None` for queries.
risingwave-standalone  | 2024-08-12T06:11:40.815449131Z  INFO rw-main risingwave_compactor::server: Assigned compactor id 1
risingwave-standalone  | 2024-08-12T06:11:40.815508423Z  WARN rw-main foyer_storage::store: [store builder]: No device config set. Use `NoneStore` which always returns `None` for queries.
risingwave-standalone  | 2024-08-12T06:11:40.815518548Z  INFO rw-main risingwave_meta_service::cluster_service: resolve host addr socket_addr=0.0.0.0:6660 host=HostAddress { host: "0.0.0.0", port: 6660 }
risingwave-standalone  | 2024-08-12T06:11:40.8159178Z  WARN rw-main aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
risingwave-standalone  | 2024-08-12T06:11:40.816064217Z  INFO rw-main risingwave_common_metrics::monitor::connection: monitoring connector hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector> with type S3
risingwave-standalone  | 2024-08-12T06:11:40.817876893Z  WARN rw-main aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
risingwave-standalone  | 2024-08-12T06:11:40.817985977Z  INFO rw-main risingwave_common_metrics::monitor::connection: monitoring connector hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector> with type S3
risingwave-standalone  | 2024-08-12T06:11:40.818041853Z  WARN rw-main risingwave_meta::manager::notification: Failed to notify local subscriber error=channel closed
risingwave-standalone  | 2024-08-12T06:11:40.818182728Z  INFO rw-main risingwave_compactor::server: Compactor non_reserved_memory_bytes 18718959206 meta_cache_capacity_bytes 134217728 compactor_memory_limit_bytes 18584741478 sstable_size_bytes 268435456 block_size_bytes 65536
risingwave-standalone  | 2024-08-12T06:11:40.818485647Z  WARN rw-main aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
risingwave-standalone  | 2024-08-12T06:11:40.818546855Z  INFO rw-main risingwave_common_metrics::monitor::connection: monitoring connector hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector> with type S3
risingwave-standalone  | 2024-08-12T06:11:40.81982882Z  INFO rw-main lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took Ok(5.125µs))
risingwave-standalone  | 2024-08-12T06:11:40.820118738Z  WARN rw-main foyer_storage::store: [store builder]: No device config set. Use `NoneStore` which always returns `None` for queries.
risingwave-standalone  | 2024-08-12T06:11:40.820157697Z  WARN rw-main foyer_storage::store: [store builder]: No device config set. Use `NoneStore` which always returns `None` for queries.
risingwave-standalone  | 2024-08-12T06:11:40.820305823Z  INFO rw-main risingwave_rt::logger: opentelemetry tracing disabled
risingwave-standalone  | 2024-08-12T06:11:40.828776867Z  INFO rw-main risingwave_common_heap_profiling::profiler: Auto memory dump is disabled.
risingwave-standalone  | 2024-08-12T06:11:40.829117411Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-stream-client
risingwave-standalone  | 2024-08-12T06:11:40.829209911Z  WARN rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: reset compute nodes failed error=failed to create RPC client to 0.0.0.0:5688: transport error: error trying to connect: tcp connect error: Connection refused (os error 111)
risingwave-standalone  | 2024-08-12T06:11:40.829508246Z  INFO rw-main risingwave_rt::logger: opentelemetry tracing disabled
risingwave-standalone  | 2024-08-12T06:11:40.829571913Z  INFO rw-main risingwave_compactor: Server listening at 0.0.0.0:6660
risingwave-standalone  | 2024-08-12T06:11:40.829597788Z  INFO rw-main risingwave_common::telemetry::report: start telemetry reporting
risingwave-standalone  | 2024-08-12T06:11:40.829679538Z  WARN rw-main risingwave_storage::monitor::hummock_state_store_metrics: unable to monitor cache. May have been registered if in all-in-one deployment: Duplicate metrics collector registration attempted
risingwave-standalone  | 2024-08-12T06:11:40.829886956Z  INFO rw-main risingwave_frontend::catalog::system_catalog: found 93 catalogs
risingwave-standalone  | 2024-08-12T06:11:40.830267792Z  INFO rw-main risingwave_meta_service::cluster_service: resolve host addr socket_addr=0.0.0.0:4566 host=HostAddress { host: "0.0.0.0", port: 4566 }
risingwave-standalone  | 2024-08-12T06:11:40.830591877Z  INFO rw-main risingwave_compute::memory::manager: LRU watermark controller: Mutex { data: LruWatermarkController { threshold_stable: 8025753759, threshold_graceful: 9172290011, threshold_aggressive: 10318826262 }, poisoned: false, .. }
risingwave-standalone  | 2024-08-12T06:11:40.830633752Z  INFO rw-main risingwave_common_heap_profiling::profiler: Auto memory dump is disabled.
risingwave-standalone  | 2024-08-12T06:11:40.830767961Z  WARN rw-main risingwave_common::telemetry::report: Telemetry failed to set tracking_id, event reporting will be disabled
risingwave-standalone  | 2024-08-12T06:11:40.831030462Z  INFO rw-main risingwave_compute::memory::manager: start running MemoryManager with interval 1000ms
risingwave-standalone  | 2024-08-12T06:11:40.831103338Z  WARN rw-main risingwave_meta::manager::notification: Failed to notify local subscriber error=channel closed
risingwave-standalone  | 2024-08-12T06:11:40.831166921Z  INFO rw-main risingwave_frontend::session: Health Check RPC Listener is set up on 0.0.0.0:6786
risingwave-standalone  | 2024-08-12T06:11:40.83124488Z  INFO rw-main risingwave_common::telemetry::report: start telemetry reporting
risingwave-standalone  | 2024-08-12T06:11:40.831325714Z  INFO rw-main risingwave_common::telemetry::report: start telemetry reporting
risingwave-standalone  | 2024-08-12T06:11:40.831982884Z  WARN rw-main risingwave_common::telemetry::report: Telemetry failed to set tracking_id, event reporting will be disabled
risingwave-standalone  | 2024-08-12T06:11:40.832007426Z  INFO rw-main risingwave_meta_service::cluster_service: resolve host addr socket_addr=0.0.0.0:5688 host=HostAddress { host: "0.0.0.0", port: 5688 }
risingwave-standalone  | 2024-08-12T06:11:40.832079259Z  INFO rw-main risingwave_meta::hummock::compactor_manager: Added compactor session 1
risingwave-standalone  | 2024-08-12T06:11:40.832115676Z  INFO rw-main risingwave_meta::hummock::manager::compaction: compactor 1 enters the cluster
risingwave-standalone  | 2024-08-12T06:11:40.83214476Z  WARN rw-main risingwave_common::telemetry::report: Telemetry failed to set tracking_id, event reporting will be disabled
risingwave-standalone  | 2024-08-12T06:11:40.834373438Z  INFO rw-main risingwave_common_heap_profiling::profiler: Auto memory dump is disabled.
risingwave-standalone  | 2024-08-12T06:11:40.834421188Z  INFO rw-main pgwire::pg_server: server started addr="0.0.0.0:4566"
risingwave-standalone  | 2024-08-12T06:11:40.836617491Z  INFO rw-main risingwave_compute: Server listening at 0.0.0.0:5688
risingwave-standalone  | 2024-08-12T06:11:40.890584774Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: recovering mview progress
risingwave-standalone  | 2024-08-12T06:11:40.891123027Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::barrier::recovery: recovered mview progress
risingwave-standalone  | 2024-08-12T06:11:40.893121287Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-stream-client
risingwave-standalone  | 2024-08-12T06:11:40.893588123Z  INFO rw-main risingwave_meta::manager::sink_coordination::manager: sink manager worker start cleaning up
risingwave-standalone  | 2024-08-12T06:11:40.893596331Z  INFO rw-main risingwave_meta::manager::sink_coordination::manager: sink manager worker finished cleaning up
risingwave-standalone  | 2024-08-12T06:11:40.893599123Z  INFO rw-main bootstrap_recovery{prev_epoch=0}:recovery_attempt: risingwave_meta::manager::sink_coordination::manager: successfully stop coordinator: None
risingwave-standalone  | 2024-08-12T06:11:40.893629956Z  INFO rw-main bootstrap_recovery{prev_epoch=0}: risingwave_meta::barrier::recovery: recovery success epoch=6960440992923648 paused=None
risingwave-standalone  | 2024-08-12T06:11:42.051887015Z  INFO rw-main risingwave_common::telemetry::report: Telemetry post success, id 2263ba2d-2d3a-4cdf-bf4c-d1e0833de20e
risingwave-standalone  | 2024-08-12T06:11:42.060714395Z  INFO rw-main risingwave_common::telemetry::report: Telemetry post success, id 2263ba2d-2d3a-4cdf-bf4c-d1e0833de20e
risingwave-standalone  | 2024-08-12T06:11:42.086734572Z  INFO rw-main risingwave_common::telemetry::report: Telemetry post success, id 2263ba2d-2d3a-4cdf-bf4c-d1e0833de20e
risingwave-standalone  | 2024-08-12T06:11:42.089357503Z  INFO rw-main risingwave_common::telemetry::report: Telemetry post success, id 2263ba2d-2d3a-4cdf-bf4c-d1e0833de20e
risingwave-standalone  | 2024-08-12T06:17:15.88116959Z  INFO rw-acceptor pgwire::pg_server: accept connection peer_addr=192.168.127.1:41355
risingwave-standalone  | 2024-08-12T06:17:16.05912594Z  INFO     rw-main risingwave_meta::stream::source_manager: spawning new watcher for source 25
risingwave-standalone  | 2024-08-12T06:17:31.539932375Z  INFO     rw-main handle_query{mode="extended query execute" session_id=0 sql=CREATE MATERIALIZED VIEW mv AS SELECT CAST(DATE_TRUNC('month', minio_source.operation_time) AS DATE) AS month, avg(minio_source.amount) AS avg_expendes FROM minio_source GROUP BY DATE_TRUNC('month', minio_source.operation_time)}: risingwave_expr::sig: found 1169 functions
risingwave-standalone  | 2024-08-12T06:17:31.599134284Z  INFO     rw-main risingwave_meta::stream::source_manager: new discovered splits fragment_id=3 new_discovered_splits={"empty_split"}
risingwave-standalone  | 2024-08-12T06:17:32.197082202Z  INFO rw-streaming actor{otel.name="Actor 19" actor_id=19 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1300002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.19820025Z  INFO rw-streaming actor{otel.name="Actor 20" actor_id=20 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1400002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.198588002Z  INFO rw-streaming actor{otel.name="Actor 17" actor_id=17 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1100002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.203235444Z  INFO rw-streaming actor{otel.name="Actor 24" actor_id=24 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1800002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.203910073Z  INFO rw-streaming actor{otel.name="Actor 21" actor_id=21 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1500002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.204217991Z  INFO rw-streaming actor{otel.name="Actor 23" actor_id=23 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1700002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.204519909Z  INFO rw-streaming actor{otel.name="Actor 22" actor_id=22 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1600002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.204564868Z  INFO rw-streaming actor{otel.name="Actor 18" actor_id=18 prev_epoch=6960463931375616 curr_epoch=6960463977447424}:executor{otel.name="Source 1200002738"}: risingwave_connector::source::filesystem::opendal_source::opendal_enumerator: list object completed.
risingwave-standalone  | 2024-08-12T06:17:32.279055737Z  INFO  rw-acceptor pgwire::pg_server: accept connection peer_addr=192.168.127.1:41356
risingwave-standalone  | 2024-08-12T06:17:32.351525844Z ERROR      rw-main handle_query{mode="extended query parse" session_id=1 sql=SELECT CAST(CAST(L.transactionid AS CHARACTER VARYING) AS BIGINT) AS transaction_id FROM pg_catalog.pg_locks AS L WHERE L.transactionid IS NOT NULL ORDER BY pg_catalog.age(L.transactionid) DESC LIMIT 1}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: pg_catalog.age(L.transactionid): function age(integer) does not exist
risingwave-standalone  | 2024-08-12T06:17:32.374705513Z ERROR      rw-main handle_query{mode="extended query parse" session_id=1 sql=select name, is_dst from pg_catalog.pg_timezone_names
risingwave-standalone  | union distinct
risingwave-standalone  | select abbrev as name, is_dst from pg_catalog.pg_timezone_abbrevs}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: sql parser error: expected SELECT, VALUES, or a subquery in the query body, found: distinct
risingwave-standalone  | LINE 2: union distinct
risingwave-standalone  |               ^
risingwave-standalone  | 2024-08-12T06:17:32.381520301Z ERROR      rw-main handle_query{mode="extended query parse" session_id=1 sql=SELECT CAST(R.oid AS BIGINT) AS role_id, rolname AS role_name, rolsuper AS is_super, rolinherit AS is_inherit, rolcreaterole AS can_createrole, rolcreatedb AS can_createdb, rolcanlogin AS can_login, rolreplication AS is_replication, rolconnlimit AS conn_limit, rolvaliduntil AS valid_until, rolbypassrls AS bypass_rls, rolconfig AS config, D.description FROM pg_catalog.pg_roles AS R LEFT JOIN pg_catalog.pg_shdescription AS D ON D.objoid = R.oid}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: rolconfig: Item not found: Invalid column: rolconfig
risingwave-standalone  | 2024-08-12T06:17:32.396370758Z ERROR      rw-main handle_query{mode="extended query parse" session_id=1 sql=SELECT CAST(T.oid AS BIGINT) AS id, T.spcname AS name, T.xmin AS state_number, pg_catalog.pg_get_userbyid(T.spcowner) AS owner, pg_catalog.pg_tablespace_location(T.oid) AS location, T.spcoptions AS options, D.description AS comment FROM pg_catalog.pg_tablespace AS T LEFT JOIN pg_catalog.pg_shdescription AS D ON D.objoid = T.oid}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: T.xmin: Item not found: Invalid column: xmin
risingwave-standalone  | 2024-08-12T06:17:32.426986259Z ERROR      rw-main handle_query{mode="extended query parse" session_id=1 sql=SELECT CAST(N.oid AS BIGINT) AS id, N.xmin AS state_number, nspname AS name, D.description, pg_catalog.pg_get_userbyid(N.nspowner) AS "owner" FROM pg_catalog.pg_namespace AS N LEFT JOIN pg_catalog.pg_description AS D ON N.oid = D.objoid ORDER BY CASE WHEN nspname = pg_catalog.current_schema() THEN - CAST(1 AS BIGINT) ELSE CAST(N.oid AS BIGINT) END}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: N.xmin: Item not found: Invalid column: xmin
risingwave-standalone  | 2024-08-12T06:17:47.007707672Z  INFO  rw-acceptor pgwire::pg_server: accept connection peer_addr=192.168.127.1:41357
risingwave-standalone  | 2024-08-12T06:17:47.033418313Z ERROR      rw-main handle_query{mode="extended query parse" session_id=2 sql=SELECT CAST(CAST(L.transactionid AS CHARACTER VARYING) AS BIGINT) AS transaction_id FROM pg_catalog.pg_locks AS L WHERE L.transactionid IS NOT NULL ORDER BY pg_catalog.age(L.transactionid) DESC LIMIT 1}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: pg_catalog.age(L.transactionid): function age(integer) does not exist
risingwave-standalone  | 2024-08-12T06:17:47.040288851Z ERROR      rw-main handle_query{mode="extended query parse" session_id=2 sql=select name, is_dst from pg_catalog.pg_timezone_names
risingwave-standalone  | union distinct
risingwave-standalone  | select abbrev as name, is_dst from pg_catalog.pg_timezone_abbrevs}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: sql parser error: expected SELECT, VALUES, or a subquery in the query body, found: distinct
risingwave-standalone  | LINE 2: union distinct
risingwave-standalone  |               ^
risingwave-standalone  | 2024-08-12T06:17:47.042746115Z ERROR      rw-main handle_query{mode="extended query parse" session_id=2 sql=SELECT CAST(R.oid AS BIGINT) AS role_id, rolname AS role_name, rolsuper AS is_super, rolinherit AS is_inherit, rolcreaterole AS can_createrole, rolcreatedb AS can_createdb, rolcanlogin AS can_login, rolreplication AS is_replication, rolconnlimit AS conn_limit, rolvaliduntil AS valid_until, rolbypassrls AS bypass_rls, rolconfig AS config, D.description FROM pg_catalog.pg_roles AS R LEFT JOIN pg_catalog.pg_shdescription AS D ON D.objoid = R.oid}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: rolconfig: Item not found: Invalid column: rolconfig
risingwave-standalone  | 2024-08-12T06:17:47.048648189Z ERROR      rw-main handle_query{mode="extended query parse" session_id=2 sql=SELECT CAST(T.oid AS BIGINT) AS id, T.spcname AS name, T.xmin AS state_number, pg_catalog.pg_get_userbyid(T.spcowner) AS owner, pg_catalog.pg_tablespace_location(T.oid) AS location, T.spcoptions AS options, D.description AS comment FROM pg_catalog.pg_tablespace AS T LEFT JOIN pg_catalog.pg_shdescription AS D ON D.objoid = T.oid}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: T.xmin: Item not found: Invalid column: xmin
risingwave-standalone  | 2024-08-12T06:17:47.063061893Z ERROR      rw-main handle_query{mode="extended query parse" session_id=2 sql=SELECT CAST(N.oid AS BIGINT) AS id, N.xmin AS state_number, nspname AS name, D.description, pg_catalog.pg_get_userbyid(N.nspowner) AS "owner" FROM pg_catalog.pg_namespace AS N LEFT JOIN pg_catalog.pg_description AS D ON N.oid = D.objoid ORDER BY CASE WHEN nspname = pg_catalog.current_schema() THEN - CAST(1 AS BIGINT) ELSE CAST(N.oid AS BIGINT) END}: pgwire::pg_protocol: error when process message error=Failed to prepare the statement: Failed to bind expression: N.xmin: Item not found: Invalid column: xmin
risingwave-standalone  | 2024-08-12T06:18:07.608116231Z  INFO rw-streaming actor{otel.name="Actor 41" actor_id=41}:executor{otel.name="Project 2900000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 41, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.608485692Z  INFO rw-streaming actor{otel.name="Actor 43" actor_id=43}:executor{otel.name="Project 2B00000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 43, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.608544817Z  INFO rw-streaming actor{otel.name="Actor 42" actor_id=42}:executor{otel.name="Project 2A00000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 42, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.608600234Z  INFO rw-streaming actor{otel.name="Actor 44" actor_id=44}:executor{otel.name="Project 2C00000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 44, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.608660693Z  INFO rw-streaming actor{otel.name="Actor 45" actor_id=45}:executor{otel.name="Project 2D00000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 45, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.608717943Z  INFO rw-streaming actor{otel.name="Actor 46" actor_id=46}:executor{otel.name="Project 2E00000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 46, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.608769277Z  INFO rw-streaming actor{otel.name="Actor 47" actor_id=47}:executor{otel.name="Project 2F00000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 47, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:07.60882411Z  INFO rw-streaming actor{otel.name="Actor 48" actor_id=48}:executor{otel.name="Project 3000000002"}: risingwave_stream::executor::sink: Sink info: sink_id: 32 actor_id: 48, need_advance_delete: true, re_construct_with_sink_pk: false
risingwave-standalone  | 2024-08-12T06:18:08.642063675Z  INFO rw-compaction lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took Ok(34.834µs))
risingwave-standalone  | 2024-08-12T06:18:47.095356819Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.09556357Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095617862Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095661154Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095705655Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095740655Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095780405Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095818947Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095850697Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095886156Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095917072Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095955323Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.095984865Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.096018406Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.096053365Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client
risingwave-standalone  | 2024-08-12T06:18:47.096087782Z  INFO rw-batch-local handle_query{mode="extended query execute" session_id=0 sql=SELECT * FROM stats, params = Some([])}:distributed_execute{query_id="80ade2dc-9027-4572-8d77-d639628ee07d" epoch=BatchQueryEpoch { epoch: Some(Committed(6960468828487680)) }}:stage{otel.name="Stage 80ade2dc-9027-4572-8d77-d639628ee07d-1" query_id="80ade2dc-9027-4572-8d77-d639628ee07d" stage_id=1}: risingwave_common_metrics::monitor::connection: monitoring connector hyper::client::connect::http::HttpConnector<risingwave_common_metrics::monitor::connection::MonitoredGaiResolver> with type grpc-compute-client

After few minutes:

risingwave-standalone  | ----------------------------------------
risingwave-standalone  | | RisingWave standalone mode is ready. |
risingwave-standalone  | ----------------------------------------
risingwave-standalone  |
risingwave-standalone  | Connect to the RisingWave instance via psql:
risingwave-standalone  |   psql -h 0.0.0.0 -p 4566 -d dev -U root
postgres-1             | 2024-08-12 06:16:40.804 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:16:47.193 UTC [22] LOG:  checkpoint complete: wrote 67 buffers (0.4%); 0 WAL file(s) added, 0 removed, 0 recycled; write=6.378 s, sync=0.008 s, total=6.390 s; sync files=40, longest=0.002 s, average=0.001 s; distance=545 kB, estimate=545 kB
postgres-1             | 2024-08-12 06:21:40.292 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:21:49.495 UTC [22] LOG:  checkpoint complete: wrote 93 buffers (0.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=9.192 s, sync=0.007 s, total=9.203 s; sync files=56, longest=0.002 s, average=0.001 s; distance=673 kB, estimate=673 kB
postgres-1             | 2024-08-12 06:26:40.593 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:26:45.257 UTC [22] LOG:  checkpoint complete: wrote 48 buffers (0.3%); 0 WAL file(s) added, 0 removed, 1 recycled; write=4.662 s, sync=0.001 s, total=4.665 s; sync files=23, longest=0.001 s, average=0.001 s; distance=532 kB, estimate=659 kB
risingwave-standalone  | thread 'rw-main' panicked at src/meta/src/hummock/manager/transaction.rs:120:17:
risingwave-standalone  | assertion failed: self.orig_deltas.insert(delta.id, delta.clone()).is_none()
risingwave-standalone  | stack backtrace:
risingwave-standalone  |    0: rust_begin_unwind
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:645:5
risingwave-standalone  |    1: core::panicking::panic_fmt
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:72:14
risingwave-standalone  |    2: core::panicking::panic
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panicking.rs:145:5
risingwave-standalone  |    3: <risingwave_meta::hummock::manager::transaction::HummockVersionTransaction as risingwave_meta::model::InMemValTransaction>::commit
risingwave-standalone  |              at ./risingwave/src/meta/src/hummock/manager/transaction.rs:120:17
risingwave-standalone  |    4: risingwave_meta::hummock::manager::commit_epoch::<impl risingwave_meta::hummock::manager::HummockManager>::commit_epoch::{{closure}}::{{closure}}
risingwave-standalone  |              at ./risingwave/src/meta/src/hummock/manager/utils.rs:46:29
risingwave-standalone  |    5: risingwave_meta::hummock::manager::commit_epoch::<impl risingwave_meta::hummock::manager::HummockManager>::commit_epoch::{{closure}}
risingwave-standalone  |              at ./risingwave/src/meta/src/hummock/manager/commit_epoch.rs:376:9
risingwave-standalone  |    6: risingwave_meta::barrier::GlobalBarrierManagerContext::update_snapshot::{{closure}}
risingwave-standalone  |              at ./risingwave/src/meta/src/barrier/mod.rs:930:30
risingwave-standalone  |    7: risingwave_meta::barrier::GlobalBarrierManagerContext::complete_barrier::{{closure}}
risingwave-standalone  |              at ./risingwave/src/meta/src/barrier/mod.rs:884:67
risingwave-standalone  |    8: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
risingwave-standalone  |    9: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
risingwave-standalone  |   10: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
risingwave-standalone  |   11: tokio::runtime::task::core::Core<T,S>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
risingwave-standalone  |   12: tokio::runtime::task::harness::poll_future::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
risingwave-standalone  |   13: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
risingwave-standalone  |   14: std::panicking::try::do_call
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
risingwave-standalone  |   15: std::panicking::try
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
risingwave-standalone  |   16: std::panic::catch_unwind
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
risingwave-standalone  |   17: tokio::runtime::task::harness::poll_future
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
risingwave-standalone  |   18: tokio::runtime::task::harness::Harness<T,S>::poll_inner
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
risingwave-standalone  |   19: tokio::runtime::task::harness::Harness<T,S>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
risingwave-standalone  |   20: tokio::runtime::task::raw::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
risingwave-standalone  |   21: tokio::runtime::task::raw::RawTask::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
risingwave-standalone  |   22: tokio::runtime::task::LocalNotified<S>::run
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
risingwave-standalone  |   23: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:576:18
risingwave-standalone  |   24: tokio::runtime::coop::with_budget
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
risingwave-standalone  |   25: tokio::runtime::coop::budget
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
risingwave-standalone  |   26: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
risingwave-standalone  |   27: tokio::runtime::scheduler::multi_thread::worker::Context::run
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
risingwave-standalone  |   28: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
risingwave-standalone  |   29: tokio::runtime::context::scoped::Scoped<T>::set
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
risingwave-standalone  |   30: tokio::runtime::context::set_scheduler::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
risingwave-standalone  |   31: std::thread::local::LocalKey<T>::try_with
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:284:16
risingwave-standalone  |   32: std::thread::local::LocalKey<T>::with
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/thread/local.rs:260:9
risingwave-standalone  |   33: tokio::runtime::context::set_scheduler
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17
risingwave-standalone  |   34: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
risingwave-standalone  |   35: tokio::runtime::context::runtime::enter_runtime
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
risingwave-standalone  |   36: tokio::runtime::scheduler::multi_thread::worker::run
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
risingwave-standalone  |   37: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
risingwave-standalone  |   38: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
risingwave-standalone  |   39: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
risingwave-standalone  |   40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
risingwave-standalone  |   41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
risingwave-standalone  |   42: tokio::runtime::task::core::Core<T,S>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30
risingwave-standalone  |   43: tokio::runtime::task::harness::poll_future::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
risingwave-standalone  |   44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/core/src/panic/unwind_safe.rs:272:9
risingwave-standalone  |   45: std::panicking::try::do_call
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:552:40
risingwave-standalone  |   46: std::panicking::try
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panicking.rs:516:19
risingwave-standalone  |   47: std::panic::catch_unwind
risingwave-standalone  |              at ./rustc/4a0cc881dcc4d800f10672747f61a94377ff6662/library/std/src/panic.rs:146:14
risingwave-standalone  |   48: tokio::runtime::task::harness::poll_future
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
risingwave-standalone  |   49: tokio::runtime::task::harness::Harness<T,S>::poll_inner
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
risingwave-standalone  |   50: tokio::runtime::task::harness::Harness<T,S>::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
risingwave-standalone  |   51: tokio::runtime::task::raw::RawTask::poll
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
risingwave-standalone  |   52: tokio::runtime::task::UnownedTask<S>::run
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
risingwave-standalone  |   53: tokio::runtime::blocking::pool::Task::run
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
risingwave-standalone  |   54: tokio::runtime::blocking::pool::Inner::run
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
risingwave-standalone  |   55: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
risingwave-standalone  |              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
risingwave-standalone  | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
risingwave-standalone  | ----------------------------------------
risingwave-standalone  | | RisingWave standalone mode is ready. |
risingwave-standalone  | ----------------------------------------
risingwave-standalone  |
risingwave-standalone  | Connect to the RisingWave instance via psql:
risingwave-standalone  |   psql -h 0.0.0.0 -p 4566 -d dev -U root
postgres-1             | 2024-08-12 06:31:40.292 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:31:45.741 UTC [22] LOG:  checkpoint complete: wrote 55 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=5.442 s, sync=0.006 s, total=5.449 s; sync files=32, longest=0.002 s, average=0.001 s; distance=545 kB, estimate=648 kB
postgres-1             | 2024-08-12 06:36:40.824 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:36:45.687 UTC [22] LOG:  checkpoint complete: wrote 50 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.855 s, sync=0.006 s, total=4.864 s; sync files=24, longest=0.002 s, average=0.001 s; distance=559 kB, estimate=639 kB
postgres-1             | 2024-08-12 06:41:40.758 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:41:45.414 UTC [22] LOG:  checkpoint complete: wrote 47 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.651 s, sync=0.003 s, total=4.656 s; sync files=22, longest=0.001 s, average=0.001 s; distance=543 kB, estimate=629 kB
postgres-1             | 2024-08-12 06:46:40.435 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:46:44.677 UTC [22] LOG:  checkpoint complete: wrote 44 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.234 s, sync=0.005 s, total=4.243 s; sync files=21, longest=0.002 s, average=0.001 s; distance=537 kB, estimate=620 kB
postgres-1             | 2024-08-12 06:51:40.775 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:51:45.104 UTC [22] LOG:  checkpoint complete: wrote 45 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.321 s, sync=0.005 s, total=4.330 s; sync files=21, longest=0.002 s, average=0.001 s; distance=530 kB, estimate=611 kB
postgres-1             | 2024-08-12 06:56:40.200 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 06:56:44.743 UTC [22] LOG:  checkpoint complete: wrote 46 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.534 s, sync=0.004 s, total=4.543 s; sync files=21, longest=0.002 s, average=0.001 s; distance=544 kB, estimate=604 kB
postgres-1             | 2024-08-12 07:01:40.842 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 07:01:44.591 UTC [22] LOG:  checkpoint complete: wrote 39 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=3.746 s, sync=0.002 s, total=3.750 s; sync files=19, longest=0.001 s, average=0.001 s; distance=518 kB, estimate=596 kB
postgres-1             | 2024-08-12 07:06:40.691 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 07:06:45.732 UTC [22] LOG:  checkpoint complete: wrote 52 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=5.036 s, sync=0.003 s, total=5.042 s; sync files=23, longest=0.001 s, average=0.001 s; distance=568 kB, estimate=593 kB
postgres-1             | 2024-08-12 07:11:40.820 UTC [22] LOG:  checkpoint starting: time
postgres-1             | 2024-08-12 07:11:45.064 UTC [22] LOG:  checkpoint complete: wrote 43 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.236 s, sync=0.005 s, total=4.244 s; sync files=20, longest=0.002 s, average=0.001 s; distance=526 kB, estimate=586 kB
adrianhaj commented 1 month ago

From diagnose:

report created at: 2024-08-12 06:23:17.993774007 UTC

number of fragment: 3
number of actor: 48
number of source: 1
number of table: 1
number of materialized view: 1
number of sink: 1
number of index: 0
number of function: 0

+----+--------------+--------------------------+---------+-------------+--------------+------------+------------+--------------------+-----------------+---------------------------+-------------+
| id | host         | type                     | state   | parallelism | is_streaming | is_serving | rw_version | total_memory_bytes | total_cpu_cores | started_at                | actor_count |
+================================================================================================================================================================================================+
| 0  | 0.0.0.0:5690 | WORKER_TYPE_COMPACTOR    | RUNNING | 0           |              |            | 1.10.0     | 23398699008        | 8               | 2024-08-12 06:11:40+00:00 |             |
|----+--------------+--------------------------+---------+-------------+--------------+------------+------------+--------------------+-----------------+---------------------------+-------------|
| 1  | 0.0.0.0:6660 | WORKER_TYPE_COMPACTOR    | RUNNING | 0           |              |            | 1.10.0     | 23398699008        | 8               | 2024-08-12 06:11:40+00:00 |             |
|----+--------------+--------------------------+---------+-------------+--------------+------------+------------+--------------------+-----------------+---------------------------+-------------|
| 2  | 0.0.0.0:4566 | WORKER_TYPE_FRONTEND     | RUNNING | 0           |              |            | 1.10.0     | 23398699008        | 8               | 2024-08-12 06:11:40+00:00 |             |
|----+--------------+--------------------------+---------+-------------+--------------+------------+------------+--------------------+-----------------+---------------------------+-------------|
| 3  | 0.0.0.0:5688 | WORKER_TYPE_COMPUTE_NODE | RUNNING | 8           | true         | true       | 1.10.0     | 23398699008        | 8               | 2024-08-12 06:11:40+00:00 | 48          |
+----+--------------+--------------------------+---------+-------------+--------------+------------+------------+--------------------+-----------------+---------------------------+-------------+

top sources by throughput (rows/s)

top materialized views by throughput (rows/s)

top join executor by matched rows

number of SSTables: 2
total size of SSTables (byte): 8335
number of compaction groups: 2

top tombstone delete ratio
+---------------------+------------+--------------+
| compaction group id | sstable id | delete ratio |
+=================================================+
| 2                   | 2          | 0.00%        |
|---------------------+------------+--------------|
| 2                   | 1          | 0.00%        |
+---------------------+------------+--------------+

top range delete ratio
+---------------------+------------+--------------+
| compaction group id | sstable id | delete ratio |
+=================================================+
| 2                   | 2          | 0.00%        |
|---------------------+------------+--------------|
| 2                   | 1          | 0.00%        |
+---------------------+------------+--------------+

top Hummock Get by duration (second)

top Hummock Iter Init by duration (second)

top table commit flush by size (byte)

object store read throughput (bytes/s)

object store write throughput (bytes/s)

object store operation rate

object store operation duration (second)

--- Actor Traces ---
>> Actor 1
Actor 1: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 100000009 [341.002ms]
      Project 100000008 [341.002ms]
        HashAgg 100000007 [341.002ms]
          Merge 100000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 16) [341.002ms]
            LocalInput (actor 12) [341.002ms]

>> Actor 2
Actor 2: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 200000009 [341.002ms]
      Project 200000008 [341.002ms]
        HashAgg 200000007 [341.002ms]
          Merge 200000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 16) [341.002ms]
            LocalInput (actor 12) [341.002ms]

>> Actor 3
Actor 3: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 300000009 [341.002ms]
      Project 300000008 [341.002ms]
        HashAgg 300000007 [341.002ms]
          Merge 300000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 16) [341.002ms]
            LocalInput (actor 12) [341.002ms]

>> Actor 4
Actor 4: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 400000009 [341.002ms]
      Project 400000008 [341.002ms]
        HashAgg 400000007 [341.002ms]
          Merge 400000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 16) [341.002ms]
            LocalInput (actor 12) [341.002ms]

>> Actor 5
Actor 5: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 500000009 [341.002ms]
      Project 500000008 [341.002ms]
        HashAgg 500000007 [341.002ms]
          Merge 500000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 16) [341.002ms]
            LocalInput (actor 12) [341.002ms]

>> Actor 6
Actor 6: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 600000009 [341.002ms]
      Project 600000008 [341.002ms]
        HashAgg 600000007 [341.002ms]
          Merge 600000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 12) [341.002ms]
            LocalInput (actor 16) [341.002ms]

>> Actor 7
Actor 7: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 700000009 [341.002ms]
      Project 700000008 [341.002ms]
        HashAgg 700000007 [341.002ms]
          Merge 700000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 12) [341.002ms]
            LocalInput (actor 16) [341.002ms]

>> Actor 8
Actor 8: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 800000009 [341.002ms]
      Project 800000008 [341.002ms]
        HashAgg 800000007 [341.002ms]
          Merge 800000006 [341.002ms]
            LocalInput (actor 10) [341.002ms]
            LocalInput (actor 13) [341.002ms]
            LocalInput (actor 11) [341.002ms]
            LocalInput (actor 15) [341.002ms]
            LocalInput (actor 9) [341.002ms]
            LocalInput (actor 14) [341.002ms]
            LocalInput (actor 12) [341.002ms]
            LocalInput (actor 16) [341.002ms]

>> Actor 9
Actor 9: `mv` [346.413s]
  Epoch 6960486656114688 [342.002ms]
    Project 900000005 [342.002ms]
      RowIdGen 900000004 [342.002ms]
        StreamFsFetch 900000003 [342.002ms]
          AppendOnlyDedup 900000002 [342.002ms]
            Merge 900000001 [342.002ms]
              LocalInput (actor 24) [342.002ms]
              LocalInput (actor 19) [342.002ms]
              LocalInput (actor 21) [342.002ms]
              LocalInput (actor 18) [342.002ms]
              LocalInput (actor 23) [342.002ms]
              LocalInput (actor 17) [342.002ms]
              LocalInput (actor 20) [342.002ms]
              LocalInput (actor 22) [342.002ms]

>> Actor 10
Actor 10: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Project A00000005 [341.002ms]
      RowIdGen A00000004 [341.002ms]
        StreamFsFetch A00000003 [341.002ms]
          AppendOnlyDedup A00000002 [341.002ms]
            Merge A00000001 [341.002ms]
              LocalInput (actor 24) [341.002ms]
              LocalInput (actor 19) [341.002ms]
              LocalInput (actor 21) [341.002ms]
              LocalInput (actor 18) [341.002ms]
              LocalInput (actor 23) [341.002ms]
              LocalInput (actor 17) [341.002ms]
              LocalInput (actor 20) [341.002ms]
              LocalInput (actor 22) [341.002ms]

>> Actor 11
Actor 11: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Project B00000005 [341.002ms]
      RowIdGen B00000004 [341.002ms]
        StreamFsFetch B00000003 [341.002ms]
          AppendOnlyDedup B00000002 [341.002ms]
            Merge B00000001 [341.002ms]
              LocalInput (actor 24) [341.002ms]
              LocalInput (actor 19) [341.002ms]
              LocalInput (actor 21) [341.002ms]
              LocalInput (actor 18) [341.002ms]
              LocalInput (actor 23) [341.002ms]
              LocalInput (actor 17) [341.002ms]
              LocalInput (actor 22) [341.002ms]
              LocalInput (actor 20) [341.002ms]

>> Actor 12
Actor 12: `mv` [346.413s]
  Epoch 6960486656114688 [342.002ms]
    Project C00000005 [342.002ms]
      RowIdGen C00000004 [342.002ms]
        StreamFsFetch C00000003 [342.002ms]
          AppendOnlyDedup C00000002 [342.002ms]
            Merge C00000001 [342.002ms]
              LocalInput (actor 24) [342.002ms]
              LocalInput (actor 19) [342.002ms]
              LocalInput (actor 21) [342.002ms]
              LocalInput (actor 18) [342.002ms]
              LocalInput (actor 23) [342.002ms]
              LocalInput (actor 17) [342.002ms]
              LocalInput (actor 22) [342.002ms]
              LocalInput (actor 20) [342.002ms]

>> Actor 13
Actor 13: `mv` [346.413s]
  Epoch 6960486656114688 [341.002ms]
    Project D00000005 [341.002ms]
      RowIdGen D00000004 [341.002ms]
        StreamFsFetch D00000003 [341.002ms]
          AppendOnlyDedup D00000002 [341.002ms]
            Merge D00000001 [341.002ms]
              LocalInput (actor 24) [341.002ms]
              LocalInput (actor 19) [341.002ms]
              LocalInput (actor 21) [341.002ms]
              LocalInput (actor 18) [341.002ms]
              LocalInput (actor 23) [341.002ms]
              LocalInput (actor 17) [341.002ms]
              LocalInput (actor 22) [341.002ms]
              LocalInput (actor 20) [341.002ms]

>> Actor 14
Actor 14: `mv` [346.413s]
  Epoch 6960486656114688 [342.002ms]
    Project E00000005 [342.002ms]
      RowIdGen E00000004 [342.002ms]
        StreamFsFetch E00000003 [342.002ms]
          AppendOnlyDedup E00000002 [342.002ms]
            Merge E00000001 [342.002ms]
              LocalInput (actor 24) [342.002ms]
              LocalInput (actor 19) [342.002ms]
              LocalInput (actor 21) [342.002ms]
              LocalInput (actor 18) [342.002ms]
              LocalInput (actor 23) [342.002ms]
              LocalInput (actor 17) [342.002ms]
              LocalInput (actor 22) [342.002ms]
              LocalInput (actor 20) [342.002ms]

>> Actor 15
Actor 15: `mv` [346.413s]
  Epoch 6960486656114688 [342.002ms]
    Project F00000005 [342.002ms]
      RowIdGen F00000004 [342.002ms]
        StreamFsFetch F00000003 [342.002ms]
          AppendOnlyDedup F00000002 [342.002ms]
            Merge F00000001 [342.002ms]
              LocalInput (actor 24) [342.002ms]
              LocalInput (actor 19) [342.002ms]
              LocalInput (actor 21) [342.002ms]
              LocalInput (actor 18) [342.002ms]
              LocalInput (actor 23) [342.002ms]
              LocalInput (actor 17) [342.002ms]
              LocalInput (actor 22) [342.002ms]
              LocalInput (actor 20) [342.002ms]

>> Actor 16
Actor 16: `mv` [346.413s]
  Epoch 6960486656114688 [342.002ms]
    Project 1000000005 [342.002ms]
      RowIdGen 1000000004 [342.002ms]
        StreamFsFetch 1000000003 [342.002ms]
          AppendOnlyDedup 1000000002 [342.002ms]
            Merge 1000000001 [342.002ms]
              LocalInput (actor 24) [342.002ms]
              LocalInput (actor 19) [342.002ms]
              LocalInput (actor 21) [342.002ms]
              LocalInput (actor 18) [342.002ms]
              LocalInput (actor 23) [342.002ms]
              LocalInput (actor 17) [342.002ms]
              LocalInput (actor 20) [342.002ms]
              LocalInput (actor 22) [342.002ms]

>> Actor 17
Actor 17: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1100002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 18
Actor 18: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1200002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 19
Actor 19: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1300002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 20
Actor 20: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1400002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 21
Actor 21: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1500002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 22
Actor 22: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1600002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 23
Actor 23: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1700002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 24
Actor 24: `mv` [346.413s]
  Epoch 6960486656114688 [343.002ms]
    Source 1800002738 [343.002ms]
      receive_barrier [343.002ms]

>> Actor 41
Actor 41: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2900000002 [341.002ms]
      Sink 2900000001 [341.002ms]
        StreamScan 2900002725 [341.002ms]
          Merge 2900000000 [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 4) [341.002ms]
            LocalInput (actor 2) [341.002ms]
            LocalInput (actor 8) [341.002ms]

>> Actor 42
Actor 42: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2A00000002 [341.002ms]
      Sink 2A00000001 [341.002ms]
        StreamScan 2A00002725 [341.002ms]
          Merge 2A00000000 [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 4) [341.002ms]
            LocalInput (actor 2) [341.002ms]
            LocalInput (actor 8) [341.002ms]

>> Actor 43
Actor 43: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2B00000002 [341.002ms]
      Sink 2B00000001 [341.002ms]
        StreamScan 2B00002725 [341.002ms]
          Merge 2B00000000 [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 4) [341.002ms]
            LocalInput (actor 2) [341.002ms]
            LocalInput (actor 8) [341.002ms]

>> Actor 44
Actor 44: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2C00000002 [341.002ms]
      Sink 2C00000001 [341.002ms]
        StreamScan 2C00002725 [341.002ms]
          Merge 2C00000000 [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 4) [341.002ms]
            LocalInput (actor 2) [341.002ms]
            LocalInput (actor 8) [341.002ms]

>> Actor 45
Actor 45: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2D00000002 [341.002ms]
      Sink 2D00000001 [341.002ms]
        StreamScan 2D00002725 [341.002ms]
          Merge 2D00000000 [340.002ms]
            LocalInput (actor 7) [340.002ms]
            LocalInput (actor 6) [340.002ms]
            LocalInput (actor 3) [340.002ms]
            LocalInput (actor 5) [340.002ms]
            LocalInput (actor 1) [340.002ms]
            LocalInput (actor 8) [340.002ms]
            LocalInput (actor 4) [340.002ms]
            LocalInput (actor 2) [340.002ms]

>> Actor 46
Actor 46: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2E00000002 [341.002ms]
      Sink 2E00000001 [341.002ms]
        StreamScan 2E00002725 [341.002ms]
          Merge 2E00000000 [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 8) [341.002ms]
            LocalInput (actor 4) [341.002ms]
            LocalInput (actor 2) [341.002ms]

>> Actor 47
Actor 47: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 2F00000002 [341.002ms]
      Sink 2F00000001 [341.002ms]
        StreamScan 2F00002725 [341.002ms]
          Merge 2F00000000 [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 8) [341.002ms]
            LocalInput (actor 4) [341.002ms]
            LocalInput (actor 2) [341.002ms]

>> Actor 48
Actor 48: `sql_sink` [310.693s]
  Epoch 6960486656114688 [341.002ms]
    Project 3000000002 [341.002ms]
      Sink 3000000001 [341.002ms]
        StreamScan 3000002725 [341.002ms]
          Merge 3000000000 [341.002ms]
            LocalInput (actor 7) [341.002ms]
            LocalInput (actor 6) [341.002ms]
            LocalInput (actor 3) [341.002ms]
            LocalInput (actor 5) [341.002ms]
            LocalInput (actor 1) [341.002ms]
            LocalInput (actor 8) [341.002ms]
            LocalInput (actor 2) [341.002ms]
            LocalInput (actor 4) [341.002ms]

>> Actor 49
Actor 49: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3100000005 [341.002ms]
      RowIdGen 3100000004 [341.002ms]
        Union 3100000003 [341.002ms]
          Merge 3100000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 48) [341.002ms]
            LocalInput (actor 42) [341.002ms]
          Merge 3100000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 50
Actor 50: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3200000005 [341.002ms]
      RowIdGen 3200000004 [341.002ms]
        Union 3200000003 [341.002ms]
          Merge 3200000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 48) [341.002ms]
            LocalInput (actor 42) [341.002ms]
          Merge 3200000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 51
Actor 51: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3300000005 [341.002ms]
      RowIdGen 3300000004 [341.002ms]
        Union 3300000003 [341.002ms]
          Merge 3300000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 48) [341.002ms]
            LocalInput (actor 42) [341.002ms]
          Merge 3300000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 52
Actor 52: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3400000005 [341.002ms]
      RowIdGen 3400000004 [341.002ms]
        Union 3400000003 [341.002ms]
          Merge 3400000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 42) [341.002ms]
            LocalInput (actor 48) [341.002ms]
          Merge 3400000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 53
Actor 53: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3500000005 [341.002ms]
      RowIdGen 3500000004 [341.002ms]
        Union 3500000003 [341.002ms]
          Merge 3500000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 48) [341.002ms]
            LocalInput (actor 42) [341.002ms]
          Merge 3500000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 54
Actor 54: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3600000005 [341.002ms]
      RowIdGen 3600000004 [341.002ms]
        Union 3600000003 [341.002ms]
          Merge 3600000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 48) [341.002ms]
            LocalInput (actor 42) [341.002ms]
          Merge 3600000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 55
Actor 55: `stats` [310.392s]
  Epoch 6960486656114688 [340.002ms]
    Materialize 3700000005 [340.002ms]
      RowIdGen 3700000004 [340.002ms]
        Union 3700000003 [340.002ms]
          Merge 3700000000 [340.002ms]
            LocalInput (actor 45) [340.002ms]
            LocalInput (actor 46) [340.002ms]
            LocalInput (actor 47) [340.002ms]
            LocalInput (actor 44) [340.002ms]
            LocalInput (actor 43) [340.002ms]
            LocalInput (actor 41) [340.002ms]
            LocalInput (actor 48) [340.002ms]
            LocalInput (actor 42) [340.002ms]
          Merge 3700000002 [340.002ms]
            LocalInput (actor 61) [340.002ms]
            LocalInput (actor 57) [340.002ms]
            LocalInput (actor 58) [340.002ms]
            LocalInput (actor 59) [340.002ms]
            LocalInput (actor 63) [340.002ms]
            LocalInput (actor 60) [340.002ms]
            LocalInput (actor 64) [340.002ms]
            LocalInput (actor 62) [340.002ms]

>> Actor 56
Actor 56: `stats` [310.392s]
  Epoch 6960486656114688 [341.002ms]
    Materialize 3800000005 [341.002ms]
      RowIdGen 3800000004 [341.002ms]
        Union 3800000003 [341.002ms]
          Merge 3800000000 [341.002ms]
            LocalInput (actor 45) [341.002ms]
            LocalInput (actor 46) [341.002ms]
            LocalInput (actor 47) [341.002ms]
            LocalInput (actor 44) [341.002ms]
            LocalInput (actor 43) [341.002ms]
            LocalInput (actor 41) [341.002ms]
            LocalInput (actor 48) [341.002ms]
            LocalInput (actor 42) [341.002ms]
          Merge 3800000002 [341.002ms]
            LocalInput (actor 61) [341.002ms]
            LocalInput (actor 57) [341.002ms]
            LocalInput (actor 58) [341.002ms]
            LocalInput (actor 59) [341.002ms]
            LocalInput (actor 63) [341.002ms]
            LocalInput (actor 60) [341.002ms]
            LocalInput (actor 64) [341.002ms]
            LocalInput (actor 62) [341.002ms]

>> Actor 57
Actor 57: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 3900000001 [343.002ms]
      Source 3900002715 [343.002ms]

>> Actor 58
Actor 58: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 3A00000001 [343.002ms]
      Source 3A00002715 [343.002ms]

>> Actor 59
Actor 59: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 3B00000001 [343.002ms]
      Source 3B00002715 [343.002ms]

>> Actor 60
Actor 60: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 3C00000001 [343.002ms]
      Source 3C00002715 [343.002ms]

>> Actor 61
Actor 61: `stats` [310.392s]
  Epoch 6960486656114688 [342.002ms]
    Dml 3D00000001 [342.002ms]
      Source 3D00002715 [342.002ms]

>> Actor 62
Actor 62: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 3E00000001 [343.002ms]
      Source 3E00002715 [343.002ms]

>> Actor 63
Actor 63: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 3F00000001 [343.002ms]
      Source 3F00002715 [343.002ms]

>> Actor 64
Actor 64: `stats` [310.392s]
  Epoch 6960486656114688 [343.002ms]
    Dml 4000000001 [343.002ms]
      Source 4000002715 [343.002ms]

--- Barrier Worker States ---
>> Worker 3
LocalBarrierWorkerDebugInfo {
    actor_to_send: {
        17,
        18,
        19,
        20,
        21,
        22,
        23,
        24,
        57,
        58,
        59,
        60,
        61,
        62,
        63,
        64,
    },
    running_actors: {
        1,
        2,
        3,
        4,
        5,
        6,
        7,
        8,
        9,
        10,
        11,
        12,
        13,
        14,
        15,
        16,
        17,
        18,
        19,
        20,
        21,
        22,
        23,
        24,
        41,
        42,
        43,
        44,
        45,
        46,
        47,
        48,
        49,
        50,
        51,
        52,
        53,
        54,
        55,
        56,
        57,
        58,
        59,
        60,
        61,
        62,
        63,
        64,
    },
    creating_actors: [],
    managed_barrier_state: ManagedBarrierStateDebugInfo {
        epoch_barrier_state_map: {},
        create_mview_progress: {},
    },
    has_control_stream_connected: true,
}

latest barrier completions
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| created_at                    | info                                                                                                                                  |
+=======================================================================================================================================================================+
| 2024-08-12 06:23:17.667+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486656114688","durationSec":0.008261255,"prevEpoch":"6960486590709760"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:16.667+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486590709760","durationSec":0.007299958,"prevEpoch":"6960486524977152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:15.665+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486524977152","durationSec":0.008004337,"prevEpoch":"6960486459441152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:14.664+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486459441152","durationSec":0.007280291,"prevEpoch":"6960486393905152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:13.662+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486393905152","durationSec":0.004786193,"prevEpoch":"6960486328369152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:12.664+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486328369152","durationSec":0.006820871,"prevEpoch":"6960486262833152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:11.665+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486262833152","durationSec":0.008249796,"prevEpoch":"6960486197362688"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:10.667+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486197362688","durationSec":0.009164343,"prevEpoch":"6960486131761152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:09.660+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486131761152","durationSec":0.002724099,"prevEpoch":"6960486066225152"} |
|-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| 2024-08-12 06:23:08.670+00:00 | {"barrierKind":"Checkpoint","command":"Plain","curEpoch":"6960486066225152","durationSec":0.012501861,"prevEpoch":"6960486000754688"} |
+-------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+

latest barrier collection failures
+------------+------+
| created_at | info |
+===================+
+------------+------+

latest barrier injection failures
+------------+------+
| created_at | info |
+===================+
+------------+------+

latest worker node panics
+------------+------+
| created_at | info |
+===================+
+------------+------+

latest create stream job failures
+------------+------+
| created_at | info |
+===================+
+------------+------+

latest dirty stream job clear-ups
+------------+------+
| created_at | info |
+===================+
+------------+------+
hzxa21 commented 1 month ago

I think there are two issues here:

  1. Empty records in SELECT * FROM stats; after new files are added.
  2. thread 'rw-main' panicked at src/meta/src/hummock/manager/transaction.rs:120:17:.

For issue 1, I can successfully reproduce and it seems to be related to #18017. Let's wait for the PR to see whether it can resolve the issue.

For issue 2, I cannot successfully reproduce. Based on the implementation, I think the panic can happen only when there is some corruption in the metadata. Can you consistently reproduce issue 2? Would you mind double checking whether the meta store backend pg db is healthy and never crashes ?

adrianhaj commented 1 month ago

I think there are two issues here:

  1. Empty records in SELECT * FROM stats; after new files are added.
  2. thread 'rw-main' panicked at src/meta/src/hummock/manager/transaction.rs:120:17:.

For issue 1, I can successfully reproduce and it seems to be related to #18017. Let's wait for the PR to see whether it can resolve the issue.

For issue 2, I cannot successfully reproduce. Based on the implementation, I think the panic can happen only when there is some corruption in the metadata. Can you consistently reproduce issue 2? Would you mind double checking whether the meta store backend pg db is healthy and never crashes ?

For issue 2, It's difficult to say because it was one time and right now I'm not able to reproduce it.

tabVersion commented 1 month ago

new data does not appear in the sink, despite uploading new files to the MinIO storage.

I think this can be fixed in https://github.com/risingwavelabs/risingwave/pull/18017