ipfs / go-datastore

key-value datastore interfaces
MIT License
228 stars 64 forks source link

Add LRUStore datastore wrapper #187

Closed gammazero closed 2 years ago

gammazero commented 2 years ago

Package lrustore provides a datastore wrapper that limits the number of items stored to the specified capacity. When at capacity, the oldest items are removed in LRU to make root for new items. Putting and getting items maintains LRU order.

This is useful for implementing an LRU-cache on top of a datastore to maintain limited storage size, or to have an LRU-cache for values that may be too large to store in memory.

Note that while the keys and values are persisted in the datastore, the LRU-ordered list of keys is kept only in memory. Keeping the keys in memory only was done for performance so as not to maintain a linked list in the datastore. This has two consequences:

Reviewers, please evaluate whether the above is preferable, or if the LRU-ordered list of keys also needs to be persisted.

welcome[bot] commented 2 years ago

Thank you for submitting this PR! A maintainer will be here shortly to review it. We are super grateful, but we are also overloaded! Help us by making sure that:

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment. Next steps:

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution. We are very grateful for your contribution!

BigLep commented 2 years ago

@gammazero : this isn't needed currently and it's not being needed right now, right? Assuming that's the case, we don't want to add it here for now to reduce maintenance. Do you want to stick it somewhere else for now and if it gets usage it moves over here later. Experimental/less maintained datastores can live in ipfs-shipyard or in the repo where they're used.

gammazero commented 2 years ago

I will keep this somewhere else. It was previously being used, but now is not, so agree that its better to have less to maintain.