mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.52k stars 1.24k forks source link

[BUG] ILiteCollection<T>.Count() throws LiteDB ENSURE: request page must be less or equals lastest page in data file #2401

Open AlainGhawi opened 9 months ago

AlainGhawi commented 9 months ago

Version LiteDB Version: 5.0.17 OS: Windows 11 .NET 7 version

Describe the bug

When I am inserting in an empty database a high number of records (11500 to be exact), I am facing an issue when calling at a later stage ILiteCollection.Count() method to trim the DB (I need the count) returning the following System.Exception: LiteDB ENSURE: request page must be less or equals lastest page in data file

Please note that if I rebuild the database it works fine again

Code to Reproduce

Insert Code:

public async Task<bool> StoreTelemetryDataAsync(List<TelemetryDataModel> data, CancellationToken token = default)
{
    DateTime currentTimeUtc = DateTime.UtcNow;
    try
    {
        using IAutoReleaseLock scopeLock = await AcquireLockAsync(token);

        try
        {
            m_context.Database.BeginTrans();

            ILiteCollection<TelemetryDataModel> col = m_context.Database.GetCollection<TelemetryDataModel>("telemetryData");

            foreach (TelemetryDataModel model in data)
            {
                bool isExist = col.FindOne(Query.EQ(nameof(TelemetryDataModel.Timestamp), new BsonValue(model.Timestamp))) is not null;
                if (isExist)
                {
                    Logger.LogDebug("Skip storing telemetry data with timestamp '{time}' because it already exists", model.Timestamp);
                    continue;
                }

                while (!token.IsCancellationRequested)
                {
                    try
                    {
                        col.Insert(model);
                        break;
                    }
                    catch (LiteException e) when (e.ErrorCode == LiteException.INDEX_DUPLICATE_KEY)
                    {
                        // On duplicate key error, increment key until it works.
                        model.Id += 1;
                    }
                }
            }
            m_context.Database.Commit();
            m_context.Database.Checkpoint();
            return true;
        }
        catch (LiteException e) when (e.ErrorCode == LiteException.FILE_SIZE_EXCEEDED)
        {
            Logger.LogWarning(e, "File size limit has been reached: {ex}", e.Message);
            m_context.Database.Rollback();
            throw;
        }
        catch (Exception)
        {
            m_context.Database.Rollback();
            throw;
        }
    }
    catch (Exception e) when (!token.IsCancellationRequested && e is not OperationCanceledException)
    {
        HandleLiteDbException(e);
        throw new Exception($"Error while storing telemetry data from time '{currentTimeUtc}' into database", e);
    }
}

Trim Code:

public async Task<bool> TrimDatabaseAsync(double percentageToTrim = 20.0, CancellationToken token = default)
{
        try
        {
            m_context.Database.BeginTrans();

            ILiteCollection<TelemetryDataModel> telemetryDataCol = m_context.Database.GetCollection<TelemetryDataModel>(IotMessagesConstants.Database.Collections.TelemetryData);

            //The exception is thrown here
            int telemetryDataCount = telemetryDataCol.Count();

            //Additional code cannot be shared
                m_context.Database.Commit();

    // Checkpoint to write all log file committed changes to the database file
    m_context.Database.Checkpoint();

    // Rebuilds and defragments the database structure to reclaim some disk space from the db file size
    m_context.Database.Rebuild();

    return true;
}
catch
{
    m_context.Database.Rollback();
    throw;
}
}

Expected behavior

The expected behavior is that the insert of a high number of records does not corrupt the database

Stacktrace

LiteDB ENSURE: request page must be less or equals lastest page in data file System.Exception: LiteDB ENSURE: request page must be less or equals lastest page in data file at LiteDB.Constants.ENSURE(Boolean conditional, String message) at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion) at LiteDB.Engine.IndexService.GetNode(PageAddress address) at LiteDB.Engine.IndexService.Find(CollectionIndex index, BsonValue value, Boolean sibling, Int32 order) at LiteDB.Engine.IndexEquals.Execute(IndexService indexer, CollectionIndex index)+MoveNext() at LiteDB.LinqExtensions.<>cDisplayClass20`2.<g|0>d.MoveNext() at LiteDB.Engine.BasePipe.LoadDocument(IEnumerable1 nodes)+MoveNext() at LiteDB.Engine.QueryPipe.Select(IEnumerable1 source, BsonExpression select)+MoveNext() at LiteDB.Engine.QueryExecutor.<>c__DisplayClass10_0.<g__RunQuery|0>d.MoveNext() at LiteDB.BsonDataReader..ctor(IEnumerable1 values, String collection) at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan) at LiteDB.Engine.QueryExecutor.ExecuteQuery() at LiteDB.Engine.LiteEngine.Query(String collection, Query query) at LiteDB.LiteQueryable1.ExecuteReader() at LiteDB.LiteQueryable1.ToDocuments()+MoveNext() at System.Linq.Enumerable.SelectEnumerableIterator2.MoveNext() at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable1 source, Boolean& found) at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source) at LiteDB.LiteCollection`1.FindOne(BsonExpression predicate)

Additional context

The context is to store a high number of telemetry data and try to trim the database when it has a size of 95% of the max size of the database.