search

peterbourgon / diskv

A disk-backed key-value store.
http://godoc.org/github.com/peterbourgon/diskv
MIT License
1.4k stars 102 forks source link

Handling large volumes of writes #42

Closed WinstonPrivacy closed 5 years ago

WinstonPrivacy commented 6 years ago

Hi there,

A limitation with the current implementation of diskv is that it doesn't work well for highly frequent updates (ie: hundreds or thousands per second). This can be problematic not only from a performance perspective, but it can seriously degrade the integrity of an SD card due to the amount of writes going on.

I've made some changes to my fork that allow a diskv client to write keys to memory only. The client can then call Persist() to flush these changes to disk. This would typically be done on a timed basis (ie: every 30-60 seconds) to minimize disk activity. The client is responsible for safely shutting down when using these routines, because keys in memory are not guaranteed to be backed by permanent storage.

I'd be interested in hearing some feedback about this. I'm not positive that these changes should go directly into diskv (though I haven't figured out how a client could manage it otherwise, or even if that would make any sense because it exposes the internals).

Anyway, we have been using this for months now in a production system and it has proven to be very solid.

peterbourgon commented 6 years ago

Hmm, I also think this may not be a natural fit, as even in the name of the project we imply a tight coupling to disk :) But I'd be happy to look at a PR, I can at least imagine a semi-elegant way this could work.

peterbourgon commented 5 years ago

Closing due to lack of activity. Feel free to re-open with a PR.