cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.58k stars 3.71k forks source link

import: subject import work to elastic CPU admission #115423

Open sumeerbhola opened 7 months ago

sumeerbhola commented 7 months ago

(See the discussion in https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1701364715878499?thread_ts=1701013647.475009&cid=C01SRKWGHG8) importer.workloadReader.readFiles and bulk.BufferingAdder, neither of which integrate with AC, were consuming 31% of the cpu on this node, which affected other user-facing queries.

AC has an existing pacing mechanism in admission.Pacer that can be used to throttle long running elastic CPU work (currently used for ExportRequest, TTL reads, rangefeed/changefeed initial and catchup scans). This seems like the right mechanism to use here. It can cause arbitrary delays, so it won't necessarily play well with timeouts at a higher layer -- I don't have enough understanding of the code surrounding importer.ingestKvs and importer.workloadReader.readFiles to know the pitfalls.

Screenshot 2023-11-30 at 12 02 51 PM

@yuzefovich @dt

Jira issue: CRDB-34026

blathers-crl[bot] commented 7 months ago

cc @cockroachdb/disaster-recovery

blathers-crl[bot] commented 7 months ago

Hi @ajstorm, please add branch-* labels to identify which branch(es) this release-blocker affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.