search

tidwall / buntdb

BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
MIT License
4.57k stars 289 forks source link

Performance on large datasets #27

Closed tonyalaribe closed 7 years ago

tonyalaribe commented 7 years ago

How does buntdb perform on data too large to fit into ram? would i have good performance with a dataset of over 5gb (mongodb dump), on a 500mb or 1gig ram server?

I'm currently evaluating buntdb over boltdb. As replacement for a system that currently runs on mongodb. I intend to use bleve search as well for indexing and search. Can I query with an array of keys? or get each key from the array individually?

tidwall commented 7 years ago

Hi Tony,

How does buntdb perform on data too large to fit into ram? would i have good performance with a dataset of over 5gb (mongodb dump), on a 500mb or 1gig ram server?

Assuming that the server has virtual memory, it'll start paging to disk when the all the ram is used up. This could cause the Go app to run very slowly, between 10-100x slower that reading directly from ram, depending on disk speed. If the server is not using virtual memory then the Go program will panic with an out of memory error.

I don't recommend that you do this. If memory is an issue then use an on-disk database like BoltDB or leveldb. For BuntDB you should have 1/4 more memory on the server than the working in-memory dataset to be safe. I suggest loading the data into a test buntdb program and see how much mem it uses.

Can I query with an array of keys? or get each key from the array individually?

You'll need to query each key individually.

tonyalaribe commented 7 years ago

Thanks Josh. You helped me clear my mind about what to use. And when buntdb is a good choice.

tidwall commented 7 years ago

You're welcome and best luck.