peterbourgon / diskv

A disk-backed key-value store.
http://godoc.org/github.com/peterbourgon/diskv
MIT License
1.4k stars 102 forks source link

Question about many tiny files vs block size #75

Closed xeoncross closed 2 years ago

xeoncross commented 2 years ago

This design creates a single file for every entry. If there are a lot of small entries, this wastes disk space as the default block size is often 4096 bytes or more. I was curious to know if there would be a way to combine the tiny writes into a single index and only break them out into their own files when they grew big enough (user input here?) to effectively be spread out over the file system.

peterbourgon commented 2 years ago

Nope! The notion that a diskv entry is 1-to-1 with a file on disk is a core property of the system.