[State Sync] Failed to run value log garbage collection for Execution Data Tracker

[UlyanaAndrukhiv]

Problem Definition

Garbage collection for tracker storage rarely runs successfully.

1) Tracker storage uses the default ValueLogFileSize option (1<<30 - 1 = ~10GB ) for creating database.

https://github.com/onflow/flow-go/blob/b219b0333b8b4939ce0f3162d976b43013651b6b/module/executiondatasync/tracker/storage.go#L193

2) Setting discardRatio to 0.5, will rewrite db if 50% of its space can be discarded.

https://github.com/onflow/flow-go/blob/b219b0333b8b4939ce0f3162d976b43013651b6b/module/executiondatasync/tracker/storage.go#L494

These parameters affects on execution data pruning and clutters the database.

[UlyanaAndrukhiv]

@peterargue, I investigated the garbage collectionprocess for badger and according to the badger/v2 documentation

Badger keeps values separately from the LSM tree. This means that the compaction operations that clean up the LSM tree do not touch the values at all. Values need to be cleaned up separately.

Badger relies on the client to perform garbage collection at a time of their choosing. It provides the following method, which can be invoked at an appropriate time (DB.RunValueLogGC())

Also I found a helpful comment from the author of Dgraph and Badger.

SST files constitute the LSM tree. Only when keys are discarded or become invalid in the SST files, are they removed from the value logs.

According to this the RunValueLogGC method will consider a value log file for GC if only at least 50% (in our case) of its data are no longer useful such as those that have been deleted or overwritten.