risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.09k stars 585 forks source link

ch-benchmark supplier table can't sync via PG CDC #14250

Closed cyliu0 closed 11 months ago

cyliu0 commented 11 months ago

Describe the bug

In RW:

dev=> select count(1) from supplier;
 count
-------
     0
(1 row)

In PG:

postgres=# select count(1) from supplier;
 count
-------
 10000
(1 row)

The alive namespace: ch-pg-cdc-cy https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=ebec273b-0774-4ccd-90a9-c2a22144d623&var-namespace=ch-pg-cdc-cy&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1703731200000&to=now&refresh=10s

Can be reproduced via running kube-bench with following env.override.toml

[benchmark]
namespace = "ch-pg-cdc-cy"
system = "risingwave"
[benchmark.aws]
ask_before_proceed = false
[benchmark.job]
name = "benchmark-job"
type = "ch-benchmark-pg-cdc"
[benchmark.job.wait]
retry_limit = 500
retry_interval = 10
[benchmark.pods.distribution]
node_selectors = "risingwave.cloud/nodegroup-name:ondemand-ng-test-32c64g-x86-0"
mutual_exclusive_policy = "global"
mutual_exclusive_key = ""
tolerations = { key = "node_group", operator = "Equal", value = "bench_test", effect ="NoSchedule" }
all_pods_affinity_enabled = "true"
compactor_compute_affinity_enabled = false
[benchmark.ch_benchmark]
#query = "q1,q2,q3,q4,q5,q6,q7,q8,q9,q10,q11,q12,q13,q14,q15,q16,q17,q18,q19,q20,q21,q22"
query = "q9"
threads = 8
warehouses = "1"
enable_blackhole = "false"
skip_prepare_data = false
enable_pg_views = true
enable_pg_sinks = "false"
modify_queries = "true"
tpcc_execution_duration = "5m"
tpcc_execution_time = "5m"
[benchmark.risingwave]
version = "nightly-20231227"
standalone_enabled = "false"
[benchmark.risingwave.wait]
retry_limit = 60
retry_interval = 10
[benchmark.risingwave.replicas]
meta = 1
frontend = 1
compute = 1
compactor = 1
connector = 0
[benchmark.risingwave.node.selector]
compute = ""
standalone = ""
[benchmark.risingwave.resources]
cpu = { limit = "8", request = "7" }
mem = { limit = "16Gi", request = "13Gi" }
[benchmark.risingwave.mem.resources]
frontend = { limit = "1Gi", request = "1Gi" }
meta = { limit = "4Gi", request = "4Gi" }
compactor = { limit = "4Gi", request = "2Gi" }
compute = { limit = "13Gi", request = "13Gi" }
[benchmark.risingwave.meta.memory_profiling]
enable = "true"
lg_prof_interval = 32
lg_prof_sample = 19
[benchmark.risingwave.compactor.memory_profiling]
enable = "true"
lg_prof_interval = 38
lg_prof_sample = 19
[benchmark.risingwave.compute.memory_profiling]
enable = "true"
lg_prof_interval = 40
lg_prof_sample = 19
[benchmark.risingwave.storage]
type = "s3"
[benchmark.risingwave.storage.s3]
region = "us-east-1"
access_key = ""
access_secret = ""
bucket = "ch-benchmark-pg-cdc-20231214-023805-19747"
data_directory = "hummock_001"
[benchmark.s3.bucket]
create_enabled = true
delete_enabled = true
[benchmark.etcd.resources]
cpu = { limit = "0", request = "0" }
mem = { limit = "4Gi", request = "4Gi" }
[benchmark.postgresql]
enabled = true
name = "benchmark-postgresql"
wal_level = "logical"
shared_preload_libraries = "decoderbufs"
[benchmark.postgresql.resources]
cpu = { limit = "0", request = "0" }
mem = { limit = "8Gi", request = "8Gi" }
[benchmark.postgresql.persistence]
enabled = true
size = "120Gi"

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20231227

Additional context

No response

StrikeW commented 11 months ago

Should be same problem of https://github.com/risingwavelabs/risingwave/issues/14248, we can rerun the test after the issue get fixed.

StrikeW commented 11 months ago

Verified due to #14248, which has been fixed.