search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.02k stars 577 forks source link

nightly-20241106 sysbench perf degradation #19281

Open cyliu0 opened 1 day ago

cyliu0 commented 1 day ago

Describe the bug

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| sysbench-select-random-points-medium-1cn                      |        42742 | Negative   | sysbench-qps                                       | -92.41%             | -59.19%                     | -64.07%                       |
| sysbench-select-random-ranges-medium-1cn                      |        42744 | Negative   | sysbench-qps                                       | -91.41%             | -58.47%                     | -63.27%                       |
| nexmark-q105-blackhole-medium-1cn                             |        42752 | Negative   | avg-source-output-rows-per-second                  | -46.23%             | -23.84%                     | -29.06%                       |

Buildkite Job Grafana Metabase Sysbench

image

The nexmark q105 also drops. But it's not stable recently. Metabase Nexmark Q105

image

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20241106

Additional context

The only pull request for nightly-20241106 is https://github.com/risingwavelabs/risingwave/pull/19080 according to https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20241106

@Li0k PTAL

Li0k commented 5 hours ago

https://buildkite.com/risingwave-test/sysbench/builds/934#01930a58-97d3-44b1-888f-2d95513f874e

https://risingwave-labs.slack.com/files/U0436BKS2CF/F07V7TVD946/untitled?origin_team=T030LTU38S2&origin_channel=C034TRPKN1F

The system will hang when we set max_prefetch_block_number = 0, 100% reproduced.