search

facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
27.85k stars 6.2k forks source link

Add CompactForTieringCollector to support automatically trigger compaction for tiering use case #12760

Closed jowlyzhang closed 2 weeks ago

jowlyzhang commented 3 weeks ago

This PR adds user property collector factory CompactForTieringCollectorFactory to support observe SST file and mark it as need compaction for fast tracking data to the proper tier.

A triggering ratio compaction_trigger_ratio_ can be configured to achieve the following: 1) Setting the ratio to be equal to or smaller than 0 disables this collector 2) Setting the ratio to be within (0, 1] will write the number of observed eligible entries into a user property and marks a file as need-compaction when aforementioned condition is met. 3) Setting the ratio to be higher than 1 can be used to just writes the user table property, and not mark any file as need compaction. For a column family that does not enable tiering feature, even if an effective configuration is provided, this collector is still disabled. For a file that is already on the last level, this collector is also disabled.

Test Plan: Added unit tests

facebook-github-bot commented 2 weeks ago

@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 weeks ago

@jowlyzhang merged this pull request in facebook/rocksdb@c73cf7a878ee217059b5c207f9ebf8507e9be6f1.