ch-benchmark q9 compute OOM with 3 compute nodes PG CDC

[cyliu0]

Describe the bug

ch-benchmark q9 will lead to compute node OOM when there are 3 compute nodes. But it won't be OOM when there is only one compute node. The compute node has the same memory size.

https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=ebec273b-0774-4ccd-90a9-c2a22144d623&var-namespace=ch-pg-cdc-cy&var-instance=benchmark-risingwave&var-pod=All&var-component=All&var-table=All&from=1703731200000&to=now&refresh=10s

The first part was running with 3 compute nodes. And there is only 1 compute node in the second part. The memory consumption will be much higher when there are 3 compute nodes.

You can use the following configuration to reproduce the OOM. env.override.toml

[benchmark]
namespace = "ch-pg-cdc-cy"
system = "risingwave"
[benchmark.aws]
ask_before_proceed = false
[benchmark.job]
name = "benchmark-job"
type = "ch-benchmark-pg-cdc"
[benchmark.job.wait]
retry_limit = 500
retry_interval = 10
[benchmark.pods.distribution]
node_selectors = "risingwave.cloud/nodegroup-name:ondemand-ng-test-32c64g-x86-0"
mutual_exclusive_policy = "global"
mutual_exclusive_key = ""
tolerations = { key = "node_group", operator = "Equal", value = "bench_test", effect ="NoSchedule" }
all_pods_affinity_enabled = "true"
compactor_compute_affinity_enabled = false
[benchmark.ch_benchmark]
#query = "q1,q2,q3,q4,q5,q6,q7,q8,q9,q10,q11,q12,q13,q14,q15,q16,q17,q18,q19,q20,q21,q22"
query = "q9"
threads = 8
warehouses = "1"
enable_blackhole = "false"
skip_prepare_data = false
enable_pg_views = true
enable_pg_sinks = "false"
modify_queries = "true"
tpcc_execution_duration = "5m"
tpcc_execution_time = "5m"
[benchmark.risingwave]
version = "nightly-20231227"
standalone_enabled = "false"
[benchmark.risingwave.wait]
retry_limit = 60
retry_interval = 10
[benchmark.risingwave.replicas]
meta = 1
frontend = 1
compute = 3
compactor = 1
connector = 0
[benchmark.risingwave.node.selector]
compute = ""
standalone = ""
[benchmark.risingwave.resources]
cpu = { limit = "8", request = "7" }
mem = { limit = "16Gi", request = "13Gi" }
[benchmark.risingwave.mem.resources]
frontend = { limit = "1Gi", request = "1Gi" }
meta = { limit = "4Gi", request = "4Gi" }
compactor = { limit = "4Gi", request = "2Gi" }
compute = { limit = "13Gi", request = "13Gi" }
[benchmark.risingwave.meta.memory_profiling]
enable = "true"
lg_prof_interval = 32
lg_prof_sample = 19
[benchmark.risingwave.compactor.memory_profiling]
enable = "true"
lg_prof_interval = 38
lg_prof_sample = 19
[benchmark.risingwave.compute.memory_profiling]
enable = "true"
lg_prof_interval = 40
lg_prof_sample = 19
[benchmark.risingwave.storage]
type = "s3"
[benchmark.risingwave.storage.s3]
region = "us-east-1"
access_key = ""
access_secret = ""
bucket = "ch-benchmark-pg-cdc-20231214-023805-19747"
data_directory = "hummock_001"
[benchmark.s3.bucket]
create_enabled = true
delete_enabled = true
[benchmark.etcd.resources]
cpu = { limit = "0", request = "0" }
mem = { limit = "4Gi", request = "4Gi" }
[benchmark.postgresql]
enabled = true
name = "benchmark-postgresql"
wal_level = "logical"
shared_preload_libraries = "decoderbufs"
[benchmark.postgresql.resources]
cpu = { limit = "0", request = "0" }
mem = { limit = "8Gi", request = "8Gi" }
[benchmark.postgresql.persistence]
enabled = true
size = "120Gi"

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20231227

Additional context

No response

[StrikeW]

Do we have a memory dump for the case just like our longevity test? The grafana metrics are normal to me, the node memory for CNs are all below the limit compute = { limit = "13Gi", request = "13Gi" }. cc @lmatz

[cyliu0]

Yes. We have the memory profiling enabled. You can check the same s3 bucket for the memory dump of this pipeline https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/143#018cb36c-0c27-48e2-b22f-94055d8cdd83

[lmatz]

The bucket for the OOM incident above seems gone, may need to run it again to generate a dump

@cyliu0 may need to test it again, link #13060

[cyliu0]

The reason might be the parallelism is too high when we use medium-arm-3cn-all-affinity testbed which uses a 32C64G node for all the CNs. https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/219

The q9 can pass with medium-3cn testbed which uses 3 different 8C16G nodes for CNs. https://buildkite.com/risingwave-test/ch-benchmark-pg-cdc/builds/220

This might be the deprecated of https://github.com/risingwavelabs/risingwave/issues/13060. Close this one