[question] multiple small databases or one large database?

[ghost]

Is it recommended to use multiple small databases or one large database? Either way, I will be storing many (hundreds to thousands) of LiteCollection sized from 100MB to 1GB.

Which way will get me optimal performance, and is LiteDB affected in any way by the filesize?

Finally, what is the reason for recommending using a different database for each user? I thought that I could just store users and references to their data in a single database, which will be easier than setting up a different database for each user.

[mbdavid]

Hi @0xFireball,

I prefer multiple small databases. But it's just me, not because LiteDB. "Small" files are simple to copy, deploy, backup, delete, ....

LiteDB has no problems with big collections, it's quite fast. Of course, some operation can be slower when you have big collections, like create index or update all. It's very difficult because you will need change a lot of pages in memory before write on disk. It's lock you file during this process. But if you basic operation will be insert/update one/delete one/querying... no problems. LiteDB uses skip lists to index data, it's O(log n) complexity (100.000 or 1.000.000 documets are only 1 step between)

I recommended "one database per user" because I write LiteDB for that 😄. I was looking for an embedded database to write an account web application where each user had an database. Each user can encrypt your own datafile. Each user can read/write only your own data. When user unsubscribe, it's just delete 1 file in my server. It's all user data in a "self contained" single file. To setup was easy: each user (email) are hashed using SHA1 and it's datafile name.

I didnt found a no-sql database for that, so, I wrote LiteDB...

[sergiorykov]

@mbdavid Every file system has limitation on the number of files in folder where it's API is responsive enough. If you have 10000 users you will need to write a wrapper which will divide all those files on subfolders with several houndred files each.

Can you describe in brief business context where it'll be really useful? Or add to wiki super-duper-happy-use cases as NancyFX has?

Actually we use LiteDB in one of our products and have DB per Bounded Context - it really simplified maintenance processes.

[mbdavid]

Yes, you need create many subfolders to store all datafiles. You can use some like this:

Username: user1 SHA1: b3daa77b4c04a9551b8781d03191fe098f325e67 Database: C:\Storage\b\3\b3daa77b4c04a9551b8781d03191fe098f325e67.db

It's create 2 levels of folders based on first 2 chars from SHA1. Will have, at max, 36 folder with 36 subfolder each (1.296 folders). Using about 200 files per folder, you can have 260.000 users.

One database per user also are cool to limit disk size per user. You can create different plans with different storage sizes. Also, using per user datafile you will not have problems with write concurrency.

[sergiorykov]

Thank you for clarification!

[rkanani108]

@mbdavid / @sergiorykov Is there any sample code available that show how to create & manage database for individual users?