search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.87k stars 569 forks source link

nightly-20240604 perf degradation #17108

Closed cyliu0 closed 2 months ago

cyliu0 commented 3 months ago

Describe the bug

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| nexmark-q16-blackhole-medium-1cn                              |        30726 | Negative   | avg-source-output-rows-per-second                  | -30.95%             | -12.08%                     | -6.43%                        |
| nexmark-q5-blackhole-watermark-medium-1cn                     |        30735 | Negative   | avg-source-output-rows-per-second                  | -31.01%             | -18.80%                     | -18.83%                       |
| nexmark-q17-blackhole-medium-1cn                              |        30738 | Negative   | avg-source-output-rows-per-second                  | -24.38%             | -10.98%                     | -9.65%                        |
| nexmark-q5-rewrite-blackhole-medium-1cn                       |        30740 | Negative   | avg-source-output-rows-per-second                  | -28.94%             | -15.20%                     | -16.78%                       |
| nexmark-q7-blackhole-watermark-medium-1cn                     |        30743 | Negative   | avg-source-output-rows-per-second                  | -26.24%             | -19.53%                     | -22.69%                       |
| nexmark-q8-blackhole-watermark-medium-1cn                     |        30748 | Negative   | avg-source-output-rows-per-second                  | -22.62%             | -14.41%                     | -12.77%                       |
| nexmark-q9-blackhole-watermark-medium-1cn                     |        30754 | Negative   | avg-source-output-rows-per-second                  | -24.97%             | -15.70%                     | -19.97%                       |
| sysbench-select-random-limits-medium-1cn                      |        30774 | Negative   | sysbench-qps                                       | -55.89%             | -19.84%                     | -32.98%                       |

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240604

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240604

lmatz commented 3 months ago

From https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240604, suspect the following: https://github.com/risingwavelabs/risingwave/pull/17018

cc: @Little-Wallace do you think it should affect the performance?

yuhao-su commented 3 months ago

Might be related #17138

cyliu0 commented 3 months ago

The SKUs are all back to normal for nightly-20240605. Any possible related fix PR in the commit history? https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240605