search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
36.57k stars 5.76k forks source link

use goroutine pool to copr.(*copIteratorWorker).run #52887

Open hawkingrei opened 2 months ago

hawkingrei commented 2 months ago

Enhancement

github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).run will always do the runtime.newstack and it will take many time on it. so we can use goroutine pool to resolve it.

https://flamegraph.com/share/7adb104e-02f1-11ef-a542-36adf59c1176

image
you06 commented 1 month ago

Nice catch! This issue impacts the latency and cpu usage more when the coprocessor tasks are small.

I've test a read case with tiny coprocessor queries.

without goroutine pool with goroutine pool diff
93091qps 96475qps +3.2%

The newstack sections in framegraph are gone, and the cpu usage slightly went down in my test! See the total cpu usage of 3 tidb nodes in the metrics.

image

image

image

I'll work on a global goroutine pool in TiDB. The goroutines with short lifetime benefit a lot from it.