Repo Size Constraints

[jbenet]

It is a common problem already that the space of go-ipfs grows unbounded. We want to have the ability to limit the growth.

---

Complicating factors:

• pinned vs unpinned objects
• capacity should be configurable
• capacity should incorporate all repo contents
• adding (pinning) content beyond capacity

---

I've been thinking of a setup with two thresholds:

• gc threshold: if after this limit, all commands trigger a repo gc.
• local storage capacity: if adding an object to repo would increase capacity, command errors. (i.e. never exceed capacity)

[cryptix]

A threshold triggering repo gc sgtm. I'd be interested in ratio stats of pinned vs unpinned objects in general.

[jbenet]

From #1482, i think we should have one global maximum storage, and trigger GC when we past a watermark (say max - 1/5MB close). ideally we could also configure the GC watermark to bring it lower.

[rht]

node.repo.datastore.storage_gc_watermark ~ hard limit on pin size? Otherwise even GC after commands won't bring down the repo size.

[rht]

How about swapping the hard limit role? Instead of storage_gc_watermark + storage_max (hard limit), use pin_max (hard limit) + blurriness? blurriness/standard_dev/swappiness is like the amount of bluriness between local storage and peer storage. Default value is 100 (100% of pin_max). By now, the only concern for global size cap other than blockstore is leveldb store size.

Also, where to put the trigger check?

1. every 100 GetBlocks cycle
2. every 10 mins
3. every time a command is run