search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.88k stars 569 forks source link

feat(ci): Provide throughput metrics in benchmark process #9421

Open kwannoel opened 1 year ago

kwannoel commented 1 year ago
  1. See if the benchmark setup is optimal, throughput should match standalone bench on machine. This is important since benchmark is done in a docker container.
  2. Use this to more accurately detect when to stop profiling. We should provide an option the profile for the entire lifecycle of a query, rather than time-bound (as is the case now).
  3. Comparison with kube-bench throughput figures.
kwannoel commented 1 year ago

Mentioned by @BugenZhao https://github.com/risingwavelabs/risingwave/pull/9216#discussion_r1176293065

I guess there might be no need to care about the throughput seriously here. To make sure that we get an accurate result, the nexmark-bench pipeline from QA team is deployed on some dedicated servers and the performance can be much more stable. 🤔 If one wants to get a serious analysis of the performance changes, I presume there could be another workflow (or "GitHub PR tag" 😄) for doing this, reusing some infrastructure of the benchmark pipelines.

kwannoel commented 1 year ago

Something else to note: https://github.com/risingwavelabs/risingwave/pull/9216#discussion_r1176418475

For a workflow measuring throughput of rw, I think s3 bucket is still required. In that case the problem you mentioned of race condition still exists.

I will see how kube-bench does it, hopefully we can apply the same approach.

kwannoel commented 1 year ago

We can name s3 bucket folder as pr branch. See: https://github.com/risingwavelabs/risingwave/pull/9216#discussion_r1177214781

github-actions[bot] commented 1 year ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

github-actions[bot] commented 3 months ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄