holiman / billy

Very simple datastore
BSD 3-Clause "New" or "Revised" License
50 stars 7 forks source link

Bucket / item index space not the best split? #3

Closed karalabe closed 1 year ago

karalabe commented 1 year ago

Currently there can be 64K buckets and 16M items per bucket. If we'd like to use this as a storage backend for txs, 16M might be on the low end (:scream:) but 64K power-of-2 buckets is insane. Perhaps we could move 4 more bits over and have 8K buckets and 256M items?

karalabe commented 1 year ago

Even with linear buckets, 8K seems like never fillable

holiman commented 1 year ago

16M might be on the low end

Don't forget that you are also required to maintain the index (in memory) on the app-side too.. But sure, we can move 4 bits over.

holiman commented 1 year ago

64K power-of-2 buckets is insane

Yeah, my thought was to make that configurable, not only "allow" power-of-2

holiman commented 1 year ago

Perhaps we could move 4 more bits over and have 8K buckets and 256M items?

Moving 4 bites over makes it 4K, not 8K, buckets and 256M items