pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
36.32k stars 5.72k forks source link

statistics: support global singleflight for sync load (#52796) #53340

Closed ti-chi-bot closed 2 weeks ago

ti-chi-bot commented 2 weeks ago

This is an automated cherry-pick of #52796

What problem does this PR solve?

Issue Number: close #52797

Problem Summary:

What changed and how does it work?

before

whiteboard_exported_image

All items the query needs will be put into the global task channel( the default capacity is 1000). And workers will merge the same task with the singleflight. So performance depends on the number of tasks that can be concurrently executed. Under poor performance conditions, he will not be able to handle a sufficient number of tasks, resulting in slow processing. But there are still many same tasks in the channel. So they still need to check whether to sync load. And They will fill up this channel. Query can not insert the sync load task into it and load necessary stats.

after

whiteboard_exported_image (1)

So we need to refactor this code. We need first to let the query pass through singleflight to merge identical tasks. Then insert the task into the channel to make the worker syncload it.

So why is it that during the startup of Wetech, the queries are very unstable and there are severe timeouts with sync load? The problem lies right here.

Check List

Tests

Side effects

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None
hawkingrei commented 2 weeks ago

/retest

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 84.09091% with 7 lines in your changes are missing coverage. Please review.

Please upload report for BASE (release-7.5@19b7d4f). Learn more about missing BASE report.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## release-7.5 #53340 +/- ## ================================================ Coverage ? 71.9971% ================================================ Files ? 1411 Lines ? 409847 Branches ? 0 ================================================ Hits ? 295078 Misses ? 94911 Partials ? 19858 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb/pull/53340/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [unit](https://app.codecov.io/gh/pingcap/tidb/pull/53340/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `71.9971% <84.0909%> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#carryforward-flags-in-the-pull-request-comment) to find out more. | [Components](https://app.codecov.io/gh/pingcap/tidb/pull/53340/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [dumpling](https://app.codecov.io/gh/pingcap/tidb/pull/53340/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `53.9913% <0.0000%> (?)` | | | [parser](https://app.codecov.io/gh/pingcap/tidb/pull/53340/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `∅ <0.0000%> (?)` | | | [br](https://app.codecov.io/gh/pingcap/tidb/pull/53340/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `53.4659% <0.0000%> (?)` | |
ti-chi-bot[bot] commented 2 weeks ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, hi-rustin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/pingcap/tidb/blob/release-7.5/OWNERS)~~ [hawkingrei,hi-rustin] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
ti-chi-bot[bot] commented 2 weeks ago

[LGTM Timeline notifier]

Timeline: