[Question] Key value mode?

[nictaylr]

What's the most high-performance way to use LiteDB as a Key-Value store? I would like something similar to the Key value store of Redis, but I still want to be using LiteDB.

I would like to know the most efficient and fastest way to store extremely large key value lookup tables (A Dictionary would not suffice, as the datasets are extremely large). I cannot load the entire table into memory, but I want to use LiteDB to access in a key-value way.

Thanks

[henon]

I would suggest using a Collection and store BsonDocuments where in doc["_id"] you assign your key and doc["value"] you store your value. May mbdavid correct me if there is a better way.

[ghost]

@mbdavid ?

[mbdavid]

Hi @nictaylr, to help you I need to know:

• What is you key? An string? Can be any other (like GUID)?
• How long is each value? More than 1 Mb
• Do you have massive deletes/updates?
• Your inserts are in batch or one-by-one?
• Is your app single/multi thread? single/multi process?

[nictaylr]

@mbdavid

• Key is a string for now.
• Values will likely be over 1MB
• Massive? If you mean large numbers of delete/updates, yes
• Batch inserts (Insert thousands of values at a time)
• Multi threaded, single process, uses a thread pool. (should I use ACID transactions?

Thanks

[mbdavid]

Using LiteDB, if your document (value) are over than 1Mb you must use FileStorage.

But you will have same problems with this option: you will need serialize your class by your own, insert/update/delete are made in chuncks (no transaction).

Another way is change document size const in your forked version. Than you can change max document size and still using a simple collection. You can use batch insertrs, but if each document use more than 1Mb your process will consume a lot of memory. All changes (data/index pages) stay in a Dictionary, in memory, during all transaction/batch. Only when write this pages will be clear. If you want speed, disable journal In you particular use, I beleave that is better to keep one instance of LiteDatabase per thread. But keep open this instance and not inside a small using statments. Each open/close instance will be open/close datafile (the slowest part of LiteDB)

This is the fast way using LiteDB. But is LiteDB the best solution for you? I really don't know. Store large documents are not an easy job. Consume a lot of memory.

[henon]

@mbdavid: and just for completeness sake, if you had small values?

[mbdavid]

@henon, basicaly my description was to "normal" documents size (less than 2000 bytes). More than this, LiteDB will use "extend" pages that works fine but is not best solution (in speed).

Big documents, more than 1Mb (in my first LiteDB version was 256kb!!) I do not recommend to use. It´s because each document will consume a lot of memory to read/write (any change on document needs write all document again). If you document is too big maybe it´s better redesign you model.

[nphamvn]

Where can I have a sample of use LiteDB as a Key-Value store, please? My value is very small, just some text.