nightly-20240611 perf degradation

[cyliu0]

Describe the bug

+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| BENCHMARK NAME                                                | EXECUTION ID | STATUS     | KEY METRICS                                        | FLUCTUATION OF BEST | FLUCTUATION OF LAST 10 DAYS | FLUCTUATION OF LAST EXECUTION |
+---------------------------------------------------------------+--------------+------------+----------------------------------------------------+---------------------+-----------------------------+-------------------------------+
| nexmark-q4-blackhole-medium-1cn                               |        31360 | Negative   | avg-source-output-rows-per-second                  | -31.71%             | -17.57%                     | -16.73%                       |
| nexmark-q5-blackhole-medium-1cn                               |        31362 | Negative   | avg-source-output-rows-per-second                  | -41.01%             | -17.13%                     | -10.00%                       |
| nexmark-q5-rewrite-blackhole-medium-1cn                       |        31366 | Negative   | avg-source-output-rows-per-second                  | -30.04%             | -14.86%                     | -8.63%                        |
| nexmark-q7-blackhole-medium-1cn                               |        31372 | Negative   | avg-source-output-rows-per-second                  | -20.86%             | -13.77%                     | -3.59%                        |
| nexmark-q8-blackhole-medium-1cn                               |        31381 | Negative   | avg-source-output-rows-per-second                  | -32.22%             | -11.24%                     | -16.42%                       |
| nexmark-q9-blackhole-medium-1cn                               |        31385 | Negative   | avg-source-output-rows-per-second                  | -28.78%             | -16.15%                     | -17.88%                       |
| backfill-datagen-100m-rows-medium-arm-all-affinity            |        31388 | Negative   | backfill-batch-query-latency-per-100m-rows         | 33.40%              | 29.95%                      | 35.49%                        |
| nexmark-q3-no-condition-blackhole-medium-1cn                  |        31399 | Negative   | avg-source-output-rows-per-second                  | -29.81%             | -20.53%                     | -26.39%                       |
| sysbench-select-random-limits-medium-1cn                      |        31400 | Negative   | sysbench-qps                                       | -66.10%             | -32.02%                     | -1.00%                        |

http://metabase.risingwave-cloud.xyz/dashboard/241-nexmark-blackhole-1cn-anti-affinity-rw-avg-source-throughput?namespace=daily

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240611

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240611

[fuyufjh]

Today (nightly-20240612) the performance seems to be back to normal.

[cyliu0]

Today (nightly-20240612) the performance seems to be back to normal.

Yep, any related fix we can paste here as a reference and close this one?

[fuyufjh]

Yep, any related fix we can paste here as a reference and close this one?

I didn't find any direct fix in nightly-20240612 against anything in nightly-20240611. Is it an accidental jitter?