implement transients directory quota allocation + automatic GC

[raulk]

Add a configuration option that enables users to limit the growth of the transients directory, along with a watermark at which transients will be reclaimed by internally calling GC.

type Config struct {
    ...
    TransientsQuota        uint64   // in bytes; GC active if non-zero
    TransientsGCWatermark  float64  // in percent; GC active if non-zero
}

GC triggering

GC will be triggered proactively on watermark breach.

But for more robustness, we should also trigger GC when an upgraded mount needs to download a transient whose size would make the transient directory exceed its allocated quota. This is the reactive path.

This path can be implemented by passing in a gater function to the upgrader, which the upgrader calls with the size of the mount, to get clearance to download that mount, and check out a reservation token.

• When the mount is downloaded, it returns the token to the gater in success, which lets the gater know that the reserved space has been truly used.
• If the mount fails to download, it returns the token in error, which informs the gater that the reserved space can be reallocated.

This pattern enables many concurrent mounts to compete for and share available space through a system of temporary reservation.

[raulk]

FYI @dirkmc @aarshkshah1992. This is the way I envision this working.

[aarshkshah1992]

@raulk Can we implement a GC based on LRU/LFU ? We can then re-use the DAGStore as a fixed size persistent cache of CAR files on Saturn/Retrieval Market L2 nodes. The requirement there is that home users will pre-allocate a fixed amount of space that we can use to persist CAR files which will then be served onwards to other CDN peers. The CAR files will be fetched using an "IPFSGatewayMount".

[raulk]

@aarshkshah1992 Yes, we can. Acquisitions and releases would need to notify the GC policy accordingly so it can internally track last access or access count. The key here will be to define a clean interface.