[BUG] LiteDB Does Not Flush Data To Disk When Database is Encrypted

[9ee1]

Version LiteDB => 5.0.8 OS => Windows 10 .NET => .NET Core 3.1

Describe the bug Creating an encrypted database, using a password in the connection string, does not flush data to disk if the number of documents that is written is minimal and you have multiple LiteDatabase handles open for the same database file in shared mode.

For example, if you attempt to commit a single document, sometimes it will flush it to disk and sometimes it won't. If you attempt to commit 1000 documents in a single transaction, it will flush to disk. The behavior is sporadic.

In both cases, LiteDB does not throw an exception and reports the number of documents committed.

Code to Reproduce

[Fact]
public void TestWithPassword() {
    var mapper = new BsonMapper();
    mapper.Entity<MyDocument>()
        .Field(p => p.CreateDate, "create_date")
        .Field(p => p.Value, "value")
        .Ctor(d => {
            var cCreateDate = d["create_date"].AsDateTime.ToUniversalTime();
            var cValue = d["value"].AsString;
            var cMyDocument = new MyDocument(cValue, cCreateDate);
            return cMyDocument;
        });

    using var context = new LiteRepository(new ConnectionString {
        Collation = Collation.Default,
        Connection = ConnectionType.Shared,
        Filename = "FILE/PATH",
        InitialSize = 0,
        Password = "Password", // Comment this out and it works.
        ReadOnly = false,
        Upgrade = false
    }, mapper);

    // ...
    //
    // Comment this out and it works.
    using var context2 = new LiteRepository(new ConnectionString {
        Collation = Collation.Default,
        Connection = ConnectionType.Shared,
        Filename = "FILE/PATH",
        InitialSize = 0,
        Password = "Password",
        ReadOnly = false,
        Upgrade = false
    }, mapper);

    context.Database.GetCollection<MyDocument>().EnsureIndex("value_ix", p => p.Value);

    // ...
    //
    // Comment this out and it works.
    context2.Database.GetCollection<MyDocument>().EnsureIndex("value_ix", p => p.Value);

    for (var i = 0; i < 1000; i++) {
        var iString = i.ToString();
        var document = new MyDocument(iString, DateTime.UtcNow);
        context.Insert(document);

        var existingDocument = context.FirstOrDefault<MyDocument>(e => e.Value == iString);
        existingDocument.Should().NotBeNull(); // Fails.
    }
}

Expected behavior Data is flushed to disk.

Screenshots/Stacktrace N/A

Additional context This behavior does not occur if the database is not encrypted.

[lbnascimento]

@9ee1 I couldn't reproduce the issue. I don't know if this has anything to do with this problem, but I believe you're using the return value of LiteCollection<T>.Insert(T) incorrectly: this method returns the Id of the inserted value, and apparently you assume that it returns the number of inserted documents.

[9ee1]

Thanks for your response @lbnascimento. You're right my code sample is inaccurate. I can update it. But the behavior is there. I verified manually by opening up the DB in LiteDB Studio and there is no data there even after the API indicates the document was committed. It is sporadic so it does not happen on every commit.

Let me know if you need me to provide an updated code sample.

[lbnascimento]

@9ee1 Yes, an updated code sample would be appreciated. I tried running your example dozens of times, both in debug and release mode, checking the datafile with LiteDB Studio every time and could not reproduce the issue once.

[9ee1]

@lbnascimento - I did a little more testing and it seems the problem occurs only under this scenario:

1. Create a custom mapper
2. Create a LiteDatabase/LiteRepository - Either one works, it won't matter. Specify a connection string with a password for encryption. Assign the aforementioned mapper
3. Create a second LiteDatabase/LiteRepository - Assign the aforementioned mapper
4. Apply the same index on both database handles
5. Commit a document using 1 database handle. Success
6. Retrieve document using same handle. Failure. Document does not exist

This is a pretty unconventional use case I think of having 2 database handles in the same process. The reason my application works like that today is because I register a factory to create a LiteDatabase instance in a DI container and its injected in all my dependencies when they are resolved. So its quite possible for 2 application threads (think web application) to request a LiteDatabase instance and get a new one that they dispose of when the thread terminates.

Interestingly enough, if you create an unencrypted LiteDatabase instance, the problem does not occur. Here is a code sample to replicate. I also updated my original comment to include this sample:

[Fact]
public void TestWithPassword() {
    var mapper = new BsonMapper();
    mapper.Entity<MyDocument>()
        .Field(p => p.CreateDate, "create_date")
        .Field(p => p.Value, "value")
        .Ctor(d => {
            var cCreateDate = d["create_date"].AsDateTime.ToUniversalTime();
            var cValue = d["value"].AsString;
            var cMyDocument = new MyDocument(cValue, cCreateDate);
            return cMyDocument;
        });

    using var context = new LiteRepository(new ConnectionString {
        Collation = Collation.Default,
        Connection = ConnectionType.Shared,
        Filename = "FILE/PATH",
        InitialSize = 0,
        Password = "Password", // Comment this out and it works.
        ReadOnly = false,
        Upgrade = false
    }, mapper);

    // ...
    //
    // Comment this out and it works.
    using var context2 = new LiteRepository(new ConnectionString {
        Collation = Collation.Default,
        Connection = ConnectionType.Shared,
        Filename = "FILE/PATH",
        InitialSize = 0,
        Password = "Password",
        ReadOnly = false,
        Upgrade = false
    }, mapper);

    context.Database.GetCollection<MyDocument>().EnsureIndex("value_ix", p => p.Value);

    // ...
    //
    // Comment this out and it works.
    context2.Database.GetCollection<MyDocument>().EnsureIndex("value_ix", p => p.Value);

    for (var i = 0; i < 1000; i++) {
        var iString = i.ToString();
        var document = new MyDocument(iString, DateTime.UtcNow);
        context.Insert(document);

        var existingDocument = context.FirstOrDefault<MyDocument>(e => e.Value == iString);
        existingDocument.Should().NotBeNull(); // Fails.
    }
}