search

mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.53k stars 1.24k forks source link

Data protection, journaling and fsync in Linux. #1251

Open pauldotknopf opened 5 years ago

pauldotknopf commented 5 years ago

How does this project deal with fsync, page-cache and data integrity?

For example, in SQLite, it uses fsync when it writes to the actual database file to ensure that in Linux, that dirty pages are flushed to disk entirely. If let Linux flush dirty pages on it's own, it can do so non-sequentially, leading to file corruption.

omfgicbf commented 5 years ago

I'm experiencing issues with this under Linux. (#1245, #1048, #851)

LiteDB supports journalling, and adding the "flush" option to the connection string adds a call to Flush(true) to the underlying FileStream, which should result in a call to fsync() under Mono on Linux.

However, I still get corrupted files when testing power-fail scenarios, which means either the journaling is not working properly, or something else is going on.

Unfortunately, there has been very little recent feedback on these issues as I believe the author is busy with some personal commitments and with the upcoming release of version 5.

I believe there is a repair method available in more recent commits, however, these have not been released to as a package yet.

It is unclear if these issues will be addressed with version 5, however, it will take time before it is considered stable enough that people are comfortable to switch a production workload to a new major version.

I'm hoping there is a plan to maintain version 4 until this is the case.

pauldotknopf commented 5 years ago

Thank you kindly for the update!

mbdavid commented 5 years ago

Hi guys,

I have not had much time to respond github issues/questions because all time that I have to I spend working on v5. I always count on the collaboration of the community, so thanks @omfgicbf . I'm not linux user but all I use is FileStream classes and methods. Flush is a connection string option to always flush to disk after any change. This will decrease performance.

In v5 there this options are not present anymore because I implement a second LOG file that always flush after write on disk (but pages are written in disk in page queue using a second thread). Also, this problem about corrupting datafile must be eliminated all persist layer was re-write using WAL concepts. I'm still working on this but results are great. You can test v5 using master branch and there are support to Upgrade v4 datafile. First beta will be released soon.