search

peterbourgon / diskv

A disk-backed key-value store.
http://godoc.org/github.com/peterbourgon/diskv
MIT License
1.41k stars 102 forks source link

Cache expiration #23

Closed adampresley closed 7 years ago

adampresley commented 9 years ago

I've used this awesome library in a couple of projects now and I think it is a great tool. One thing I'm wondering about is a way to set a cache expiration policy. The policy would dictate when cache keys become stale and should expire and perhaps be erased. Have you considered any additional features to this library, or perhaps strategies to employ regarding this topic?

Thanks!

peterbourgon commented 9 years ago

Happy to hear it! Cache expiration solves the problem of data that changes behind the caching layer without the caching layer knowing about it. But with diskv, the only way to change the data is with a Write, which will always bust the cache. Are you looking to solve a particular use case?

adampresley commented 9 years ago

Right now I have two very different scenarios. The first is a service application that acts as a data provider for a CMS application, combining data from two different data sources. This one I'm thinking I want to "expire" certain cache keys based on a schedule of when an administrative user is most likely to made changes to content.

The second is an internal tool for a team I work on that allows us to manage certain aspects of our local development environments. I have a situation where a tool I wrote searches thousands of files, and I am caching the directories to be searched. The problem here is that at some point a directory may no longer exist, so here I'm thinking on expiring these cache keys on a daily basis.

peterbourgon commented 9 years ago

In both cases, it seems like you're setting up a diskv instance on top of a directory tree that's being mutated by other actors/processes. Is that right?

adampresley commented 9 years ago

In the first case I am using diskv to cache REST requests from a 3rd party provider that I in turn feed to my CMS system. My users can change information in this 3rd party system. At some point the data I'm caching needs to be refreshed.

In the second case the cache value is an array of directories. The actual directories can change, and at that point my cache is invalid because it references directories that no longer exist.

peterbourgon commented 9 years ago

Okay, thanks for the explanations. Let me think about this for a little bit.

peterbourgon commented 7 years ago

I don't think I'm going to do this :) Sorry!