search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
36.85k stars 5.8k forks source link

Track memory usage in pessimistic locking phase #51217

Open ekexium opened 6 months ago

ekexium commented 6 months ago

Enhancement

part of #49643 image The memory consumed in InitCheckExistence in the graph is not tracked by the memory tracker.

RemHero commented 4 months ago

I'm interested in this. Can you assign it to me? Could you please give me some tips?

ekexium commented 4 months ago

Hi @RemHero, you can start from learning how other trackers track the memory usage of data structures, and apply the same way to this one. You can find the memory tracker defined in pkg/util/memory/tracker.go

RemHero commented 4 months ago

The InitCheckExistence is a method in tikv/client-go; refer to InitCheckExistence. There seems to be no way to get the memory consumption inside InitCheckExistence with Tracker? Could you please give me another hint? @ekexium Thanks

ekexium commented 4 months ago

@RemHero

  1. You need to figure out how to estimate the memory footprint of this part, as accurate as possible. mem_aware_map.go might help
  2. You need to record the memory usage via a tracker. It involves both TiDB repo and the client-go repo. You need to define a new tracker type, initialize it, and pass it to the function appropriately. One way worth consideration is passing a onMemChange closure to the structure.
RemHero commented 4 months ago

@ekexium Hello! I have submitted client-go PR and TiDB PR. It should be noted that the structure of LockCtx in client-go is changed, but the client-go version referenced in TiDB is v2. Even if PR in client-go is merged, it cannot make TiDB perceive the structural change of LockCtx.