adds cleanup service to prune outdated assets, and configuration option to enable/disable it on storage-node startup
adds CLI command 'dev:cleanup' for performing pruning actions (ideally should not be used in production)
extends '/status' endpoint with additional info, e.g. sync, cleanup/pruning & serving buckets configurations
How the pruning of assets work
The cleanupService runs the data objects cleanup/pruning workflow. It removes all the locally stored data objects that the operator is no longer obliged to keep, because either the data object has been deleted from the runtime or it has been moved to some other bucket/s
PRECONDITIONS:
Since the cleanup uses the QueryNode to query the data obligations, the QueryNode processor should not lag more than MAXIMUM_QN_LAGGING_THRESHOLD blocks, otherwise the cleanup workflow would not be performed in order to avoid pruning assets based on outdated state.
If the asset being pruned from this storage node still exists in the runtime (i.e. its storage obligation has been moved), then at least "X" other storage nodes should hold the asset, otherwise, the cleanup workflow would not be performed, where "X" is defined by MINIMUM_REPLICATION_THRESHOLD
If the asset being pruned from this storage node is currently being downloaded by some external actors, then the cleanup action for this asset would be postponed
addresses #4813
This PR adds:
How the pruning of assets work
The
cleanupServiceruns the data objects cleanup/pruning workflow. It removes all the locally stored data objects that the operator is no longer obliged to keep, because either the data object has been deleted from the runtime or it has been moved to some other bucket/sPRECONDITIONS:
MAXIMUM_QN_LAGGING_THRESHOLDblocks, otherwise the cleanup workflow would not be performed in order to avoid pruning assets based on outdated state.MINIMUM_REPLICATION_THRESHOLD