search

tekartik / sembast.dart

Simple io database
BSD 2-Clause "Simplified" License
786 stars 64 forks source link

loading data from file instead of memory #89

Closed MichaelSowah closed 4 years ago

MichaelSowah commented 5 years ago

First of all, I would like to appreciate you for such a wonderful package. now the only thing worrying me right now is the loading into memory thing. what informed that decision and would it be easy for just to switch to load from file or edit the code to do so. I am switching from one nosql package to this one for the extensive work done... as much my data can grow I prefer this to the sqflite. But this memory thing could be expensive for me since am doing mobile.

What will it take to have an option to read and write data to file instead of memory? And what are performance implications to load the data from file. And if you don't mind could you point me to where exactly so I can switch it to read from file. Thanks

alextekartik commented 5 years ago

Sembast was initially made as a tool to mock other database API for the dart VM (indexed_db, firestore). Some people started to use it for flutter.

I was looking for a plain file based, pure dart solution so I ended up with a simple append only file. When I was looking for existing javascript solution, I found NeDB (node) and lovefield (browser). I was surprised that they loaded everything in memory first...but it is actually the most efficient (read once from source and perform query in memory) and satisfy the needs of 99% of applications.

Yes that can be an issue for mobile although I heard about people with 100k records. Personnally I use sqflite for flutter and would recommend this if you expect 1 million records in your app.

Changing to read each time each record from the file would not be an easy task and I cannot easily point you to every change that would be needed. Read performance will be really bad. For each query with a filter, it will require reading all the records in a given store each time. If you have 100k records, it will mean performing 100k read each time so I don't even think it is worth trying without redesigning the library and the storage format.

I would love to develop a pure dart nosql database that could efficiently deal with 1 million records but that is not in the scope of sembast.

MichaelSowah commented 5 years ago

Thanks for the answer, anyway I don't think I will ever even load 10k records so I think I will do just fine then, I look forward to the pure dart nosql database that could efficiently deal with 1 million records in future, once great work with your packages