mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.62k stars 1.25k forks source link

[BUG] System.OutOfMemoryException - LiteDatabase in memory #2541

Open ednsinf opened 2 months ago

ednsinf commented 2 months ago

Bug not resolved. When I use LiteDatabase in memory and use bulkinsert and insert in 1k to 1k rows (using transaction our not) and shows memory erros always. Please verify thanks. at System.IO.MemoryStream.set_Capacity(Int32 value) at System.IO.MemoryStream.EnsureCapacity(Int32 value) at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at LiteDB.Engine.ConcurrentStream.Write(Byte[] buffer, Int32 offset, Int32 count) at LiteDB.Engine.DiskService.WriteLogDisk(IEnumerable`1 pages) at LiteDB.Engine.TransactionService.PersistDirtyPages(Boolean commit) at LiteDB.Engine.TransactionService.Commit() at LiteDB.Engine.LiteEngine.CommitAndReleaseTransaction(TransactionService transaction) at LiteDB.Engine.LiteEngine.Commit()

JKamsker commented 2 months ago

Can you provide a repro?

ednsinf commented 2 months ago

This error occurs here LiteDB\Engine\Engine\Transaction.cs -> AutoTransaction line 93 image

ednsinf commented 2 months ago

More details: at System.IO.MemoryStream.set_Capacity(Int32 value) at System.IO.MemoryStream.EnsureCapacity(Int32 value) at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at LiteDB.Engine.ConcurrentStream.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\LiteDB\Engine\Disk\Streams\ConcurrentStream.cs:line 75 at LiteDB.Engine.DiskService.WriteDataDisk(IEnumerable1 pages) in C:\LiteDB\Engine\Disk\DiskService.cs:line 304 at LiteDB.Engine.WalIndexService.CheckpointInternal() in C:\LiteDB\Engine\Services\WalIndexService.cs:line 349 at LiteDB.Engine.WalIndexService.TryCheckpoint() in C:\LiteDB\Engine\Services\WalIndexService.cs:line 293 at LiteDB.Engine.LiteEngine.CommitAndReleaseTransaction(TransactionService transaction) in C:\Users\e_dns\Downloads\LiteDB\LiteDB\Engine\Engine\Transaction.cs:line 118 at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn) in C:\LiteDB\Engine\Engine\Transaction.cs:line 91

ednsinf commented 2 months ago

Sugested solution to fix Memory Alocation error and inprove performance: File: LiteDB\Engine\Disk\StreamFactory\StreamFactory.cs - line 33

public Stream GetStream(bool canWrite, bool sequencial)
 {
     const int bufferSize = 81920; // 80 KB    

     if (_password == null)
     {
         using (var bs = new BufferedStream(_stream, bufferSize))
         {
             return new ConcurrentStream(bs, canWrite);
         }
     }
     else
     {
         using (var bs = new BufferedStream(_stream, bufferSize))
         {
             return new AesStream(_password, new ConcurrentStream(bs, canWrite));
         }
     }
 }
JKamsker commented 2 months ago

You are always more than welcome to open a pr including the fix and tests