backend/local: set pebble db max file limit - Githubissues

pingcap / tidb-lightning

This repository has been moved to https://github.com/pingcap/br

Apache License 2.0

142 stars 66 forks source link

backend/local: set pebble db max file limit #501

Closed glorv closed 3 years ago

glorv commented 3 years ago

What problem does this PR solve?

Set the maximum open file limit for pebble to avoid open files exceeds the system open files limit

What is changed and how it works?

set the estimated max open files limit to (sum of topN table files size) / tableConcurrency * 2
check the estimated open files limit with a default min value of 1024.
Set pebble maxOpenFiles to the estimatedOpenFiles, L0CompactionThreshold and L0StopWritesThreshold to estimatedOpenFiles/2.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Possible performance regression
- Since if there are more files in pebble than estimatedOpenFiles/2, pebble will start compacting L0 SST files and stop write, thus there may be performance regression in some extreme condition
Increased code complexity

Related changes

overvenus commented 3 years ago

LGTM