[BUG] Getting into a rare and unrecoverable state after 5.0.21

[hosadoya]

Version Which LiteDB version/OS/.NET framework version are you using. (REQUIRED) 5.0.21/Windows/.net 6

Describe the bug A clear and concise description of what the bug is. After updating the NuGet to 5.0.21 version, getting randomly into an unrecoverable state. Here are some of the logs in sequence:

The very first and only unique error:

LiteDB.LiteException: This transaction are invalid state
   at LiteDB.Engine.QueryExecutor.<>c__DisplayClass12_0.<<ExecuteQuery>g__RunQuery|2>d.MoveNext()
   at LiteDB.Utils.Extensions.EnumerableExtensions.OnDispose[T](IEnumerable`1 source, Action onDispose)+MoveNext()
   at LiteDB.Utils.Extensions.EnumerableExtensions.OnDispose[T](IEnumerable`1 source, Action onDispose)+MoveNext()
   at LiteDB.BsonDataReader..ctor(IEnumerable`1 values, String collection, EngineState state)
   at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
   at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
   at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

After this getting 94 of those errors from different threads:

LiteDB.LiteException: pages in memory store must be non-shared
   at LiteDB.BsonDataReader.Read()
   at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

After that getting thousands of those errors:

LiteDB.LiteException: Maximum number of transactions reached
   at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
   at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
   at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
   at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

This already happened 3 times. The only way to recover was to restart the process.

Code to Reproduce Write a small snippet to isolate your bug and could be possible to our team test. (REQUIRED)

This seems to be related to the recent OnDispose change as it is where the issue starts. No repro as this is very random error happening few times per week. All we got is detail logs shown above.

Expected behavior A clear and concise description of what you expected to happen.

No error/corruption.

Screenshots/Stacktrace If applicable, add screenshots/stacktrace

NA

Additional context Add any other context about the problem here.

App is running many (max 20) concurrent tasks in parallel which CRUD into the LiteDB in varying ways.

[flier268]

@hosadoya You can downgrade to 5.0.17 https://github.com/mlockett42/litedb-async/issues/34

[RichardVogelij]

5.0.21 / .net core 7.

This happens also to us about twice a week in a usage where many concurrent tasks CRUD the LiteDB database.

Downgrading is not an option as 5.0.21 fixes a (for us) common occurrence of "empty page must be defined as empty type" cases.

I regretfully do not have the issue reproducible - the cause seems random, but it seems to be more prevalent when inserting some data into an audit-log table in a background task.

I have not seen the first occurrence of the "This transaction are invalid state" - but I am not 100% certain it did not occur. I do however see a lot of the "pages in memory store must be non-shared" starting all of a sudden.

When this happens the entire LiteDB connection becomes unusable, first only getting "pages in memory store must be non-shared" and eventually getting nothing but errors complaining about "Maximum number of transactions reached " on all subsequent queries.

Thankfully no data seems to be corrupted and restarting the app fixes the issue.

Since there seems to be no solution yet I'm thinking about detecting this particular issue and destroying+recreating the LiteDB connection as temporary work around - but would like to urge the devs of LiteDB to take this one seriously;

Is there any other suggestion other than downgrading perhaps?