Open UlyanaAndrukhiv opened 2 months ago
@peterargue, I investigated the garbage collection
process for badger
and according to the badger/v2
documentation
Badger keeps values separately from the LSM tree. This means that the compaction operations that clean up the LSM tree do not touch the values at all. Values need to be cleaned up separately.
Badger relies on the client to perform garbage collection at a time of their choosing. It provides the following method, which can be invoked at an appropriate time (DB.RunValueLogGC())
Also I found a helpful comment from the author of Dgraph and Badger.
SST files constitute the LSM tree. Only when keys are discarded or become invalid in the SST files, are they removed from the value logs.
According to this the RunValueLogGC
method will consider a value log file for GC if only at least 50% (in our case) of its data are no longer useful such as those that have been deleted or overwritten.
Problem Definition
Garbage collection for tracker storage rarely runs successfully.
1) Tracker storage uses the default
ValueLogFileSize
option (1<<30 - 1 = ~10GB ) for creating database.https://github.com/onflow/flow-go/blob/b219b0333b8b4939ce0f3162d976b43013651b6b/module/executiondatasync/tracker/storage.go#L193
2) Setting
discardRatio
to 0.5, will rewrite db if 50% of its space can be discarded.https://github.com/onflow/flow-go/blob/b219b0333b8b4939ce0f3162d976b43013651b6b/module/executiondatasync/tracker/storage.go#L494
These parameters affects on execution data pruning and clutters the database.