Version
LiteDB Version: 5.0.17
OS: Windows 11
.NET 7 version
Describe the bug
When I am inserting in an empty database a high number of records (11500 to be exact),
I am facing an issue when calling at a later stage ILiteCollection.Count() method to trim the DB (I need the count) returning the following System.Exception: LiteDB ENSURE: request page must be less or equals lastest page in data file
Please note that if I rebuild the database it works fine again
Code to Reproduce
Insert Code:
public async Task<bool> StoreTelemetryDataAsync(List<TelemetryDataModel> data, CancellationToken token = default)
{
DateTime currentTimeUtc = DateTime.UtcNow;
try
{
using IAutoReleaseLock scopeLock = await AcquireLockAsync(token);
try
{
m_context.Database.BeginTrans();
ILiteCollection<TelemetryDataModel> col = m_context.Database.GetCollection<TelemetryDataModel>("telemetryData");
foreach (TelemetryDataModel model in data)
{
bool isExist = col.FindOne(Query.EQ(nameof(TelemetryDataModel.Timestamp), new BsonValue(model.Timestamp))) is not null;
if (isExist)
{
Logger.LogDebug("Skip storing telemetry data with timestamp '{time}' because it already exists", model.Timestamp);
continue;
}
while (!token.IsCancellationRequested)
{
try
{
col.Insert(model);
break;
}
catch (LiteException e) when (e.ErrorCode == LiteException.INDEX_DUPLICATE_KEY)
{
// On duplicate key error, increment key until it works.
model.Id += 1;
}
}
}
m_context.Database.Commit();
m_context.Database.Checkpoint();
return true;
}
catch (LiteException e) when (e.ErrorCode == LiteException.FILE_SIZE_EXCEEDED)
{
Logger.LogWarning(e, "File size limit has been reached: {ex}", e.Message);
m_context.Database.Rollback();
throw;
}
catch (Exception)
{
m_context.Database.Rollback();
throw;
}
}
catch (Exception e) when (!token.IsCancellationRequested && e is not OperationCanceledException)
{
HandleLiteDbException(e);
throw new Exception($"Error while storing telemetry data from time '{currentTimeUtc}' into database", e);
}
}
Trim Code:
public async Task<bool> TrimDatabaseAsync(double percentageToTrim = 20.0, CancellationToken token = default)
{
try
{
m_context.Database.BeginTrans();
ILiteCollection<TelemetryDataModel> telemetryDataCol = m_context.Database.GetCollection<TelemetryDataModel>(IotMessagesConstants.Database.Collections.TelemetryData);
//The exception is thrown here
int telemetryDataCount = telemetryDataCol.Count();
//Additional code cannot be shared
m_context.Database.Commit();
// Checkpoint to write all log file committed changes to the database file
m_context.Database.Checkpoint();
// Rebuilds and defragments the database structure to reclaim some disk space from the db file size
m_context.Database.Rebuild();
return true;
}
catch
{
m_context.Database.Rollback();
throw;
}
}
Expected behavior
The expected behavior is that the insert of a high number of records does not corrupt the database
Stacktrace
LiteDB ENSURE: request page must be less or equals lastest page in data file System.Exception: LiteDB ENSURE: request page must be less or equals lastest page in data file at LiteDB.Constants.ENSURE(Boolean conditional, String message) at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion) at LiteDB.Engine.IndexService.GetNode(PageAddress address) at LiteDB.Engine.IndexService.Find(CollectionIndex index, BsonValue value, Boolean sibling, Int32 order) at LiteDB.Engine.IndexEquals.Execute(IndexService indexer, CollectionIndex index)+MoveNext() at LiteDB.LinqExtensions.<>cDisplayClass20`2.<g|0>d.MoveNext() at LiteDB.Engine.BasePipe.LoadDocument(IEnumerable1 nodes)+MoveNext() at LiteDB.Engine.QueryPipe.Select(IEnumerable1 source, BsonExpression select)+MoveNext() at LiteDB.Engine.QueryExecutor.<>c__DisplayClass10_0.<g__RunQuery|0>d.MoveNext() at LiteDB.BsonDataReader..ctor(IEnumerable1 values, String collection) at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan) at LiteDB.Engine.QueryExecutor.ExecuteQuery() at LiteDB.Engine.LiteEngine.Query(String collection, Query query) at LiteDB.LiteQueryable1.ExecuteReader() at LiteDB.LiteQueryable1.ToDocuments()+MoveNext() at System.Linq.Enumerable.SelectEnumerableIterator2.MoveNext() at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable1 source, Boolean& found) at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source) at LiteDB.LiteCollection`1.FindOne(BsonExpression predicate)
Additional context
The context is to store a high number of telemetry data and try to trim the database when it has a size of 95% of the max size of the database.
Version LiteDB Version: 5.0.17 OS: Windows 11 .NET 7 version
Describe the bug
When I am inserting in an empty database a high number of records (11500 to be exact), I am facing an issue when calling at a later stage ILiteCollection.Count() method to trim the DB (I need the count) returning the following System.Exception: LiteDB ENSURE: request page must be less or equals lastest page in data file
Please note that if I rebuild the database it works fine again
Code to Reproduce
Insert Code:
Trim Code:
Expected behavior
The expected behavior is that the insert of a high number of records does not corrupt the database
Stacktrace
LiteDB ENSURE: request page must be less or equals lastest page in data file System.Exception: LiteDB ENSURE: request page must be less or equals lastest page in data file at LiteDB.Constants.ENSURE(Boolean conditional, String message) at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion) at LiteDB.Engine.IndexService.GetNode(PageAddress address) at LiteDB.Engine.IndexService.Find(CollectionIndex index, BsonValue value, Boolean sibling, Int32 order) at LiteDB.Engine.IndexEquals.Execute(IndexService indexer, CollectionIndex index)+MoveNext() at LiteDB.LinqExtensions.<>cDisplayClass20`2.<g |0>d.MoveNext() at LiteDB.Engine.BasePipe.LoadDocument(IEnumerableg__RunQuery|0>d.MoveNext() at LiteDB.BsonDataReader..ctor(IEnumerable
1 nodes)+MoveNext() at LiteDB.Engine.QueryPipe.Select(IEnumerable
1 source, BsonExpression select)+MoveNext() at LiteDB.Engine.QueryExecutor.<>c__DisplayClass10_0.<1 values, String collection) at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan) at LiteDB.Engine.QueryExecutor.ExecuteQuery() at LiteDB.Engine.LiteEngine.Query(String collection, Query query) at LiteDB.LiteQueryable
1.ExecuteReader() at LiteDB.LiteQueryable1.ToDocuments()+MoveNext() at System.Linq.Enumerable.SelectEnumerableIterator
2.MoveNext() at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable1 source, Boolean& found) at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable
1 source) at LiteDB.LiteCollection`1.FindOne(BsonExpression predicate)Additional context
The context is to store a high number of telemetry data and try to trim the database when it has a size of 95% of the max size of the database.