search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.19k stars 5.84k forks source link

Occasional performance regression on benchmarksql #56617

Open owlsome2501 opened 2 weeks ago

owlsome2501 commented 2 weeks ago

Bug Report

image There is occasional performance regression on benchmarksql. releated issue: #55426

1. Minimal reproduce step (Required)

  1. Deploy a cluster with 3 TiDB and 3 Tikv.
  2. Run benchmark benchmarksql (benchmarksql-1k) multiple times until find the specific io or cpu usage pattern on grafana.

2. What did you expect to see? (Required)

No performance regression. QPS: 64039 (+6.4% compared to baseline 60193) image or QPS: 61259 (+1.77% compared to baseline 60193) image

3. What did you see instead (Required)

Performance regression . QPS: 55187 (-8.3% compared to baseline 60193) image or QPS: 54639 (-9.20% compared to baseline 60193) image

4. What is your TiDB version? (Required)

pingcap/tidb@26443dab89ba5b1a866de3c3e44e2f51468bc435 tikv/tikv@7c612787d0bfc57a339907ebcb26777dc296fd5f tikv/pd@098b802fcda75b5ae0527b0b50b95ba8b31403cf or pingcap/tidb@74034d4ac243b3c14dbf5f8a9edb92e740da4212 tikv/tikv@de06b38a118582ee01d16ab4d3582bd8fa5cb635 tikv/pd@76bc34aee22554353bd7ea9785d3772dcb7ab31b

owlsome2501 commented 2 weeks ago

/severity major /label affects-8.3 /label affects-8.4

ti-chi-bot[bot] commented 2 weeks ago

@owlsome2501: The label(s) affects-8.3 cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, needs-cherry-pick-release-8.1, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.4, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1.

In response to [this](https://github.com/pingcap/tidb/issues/56617#issuecomment-2408472667): >/severity major >/label affects-8.3 >/label affects-8.4 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
jebter commented 1 week ago

@owlsome2501 Can you confirm which modules introduced the issue?

owlsome2501 commented 1 week ago

Unable to determine the module. It may be hot spots or load balancing problem.