search

mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.52k stars 1.24k forks source link

[QUESTION] Will the database often corrupted after bulk insert data #1680

Closed mwkldeveloper closed 4 years ago

mwkldeveloper commented 4 years ago

db version 5.0.7 Raspberry pi raspbian .net core 3.1

When i try insert a record it throw:

LiteDB.LiteException: Invalid Index on 203. Full zero: False. Page Type: Empty. Prev/Next: 4294967295/442. UniqueID: 4. ShareCounter: -1.

GW-FUB commented 4 years ago

I got the same problem. I'm inserting some hundreds of datasets into the database and call DeleteAll() & Insert() when reloading.

The first time inserting into the empty database works great. Also reading. But when updating or sometimes also reading data the database gets corrupted. This happens in an unregular pattern (at least I haven't found any so far).

Here is one of my exceptions:

LiteDB.LiteException: Invalid Data on 15. Full zero: False. Page Type: Index. Prev/Next: 4294967295/4294967295. UniqueID: 4497. ShareCounter: 1. 
  at LiteDB.Engine.DataPage..ctor (LiteDB.Engine.PageBuffer buffer) [0x00017] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BasePage.ReadPage[T] (LiteDB.Engine.PageBuffer buffer) [0x000b7] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.Snapshot.ReadPage[T] (System.UInt32 pageID, LiteDB.Engine.FileOrigin& origin, System.Int64& position, System.Int32& walVersion) [0x000c2] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.Snapshot.GetPage[T] (System.UInt32 pageID, LiteDB.Engine.FileOrigin& origin, System.Int64& position, System.Int32& walVersion) [0x0003c] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.Snapshot.GetPage[T] (System.UInt32 pageID) [0x00000] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.DataService+<Read>d__5.MoveNext () [0x00020] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BufferReader.MoveFordward (System.Int32 count) [0x00041] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BufferReader.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x0004f] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BufferReader.ReadNumber[T] (System.Func`3[T1,T2,TResult] convert, System.Int32 size) [0x0004a] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BufferReader.ReadInt32 () [0x00000] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BufferReader.ReadDocument (System.Collections.Generic.HashSet`1[T] fields) [0x00000] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.DatafileLookup.Load (LiteDB.Engine.PageAddress rawId) [0x00018] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.DatafileLookup.Load (LiteDB.Engine.IndexNode node) [0x00007] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.BasePipe+<LoadDocument>d__6.MoveNext () [0x0004a] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.QueryPipe+<Select>d__2.MoveNext () [0x000dd] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.Engine.QueryExecutor+<>c__DisplayClass10_0+<<ExecuteQuery>g__RunQuery|0>d.MoveNext () [0x002fa] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.BsonDataReader.Read () [0x00023] in <805b6841a9694a12b73a9471c608085e>:0 
  at LiteDB.LiteQueryable`1+<ToDocuments>d__26[T].MoveNext () [0x0005f] in <805b6841a9694a12b73a9471c608085e>:0 
  at System.Linq.Enumerable+SelectEnumerableIterator`2[TSource,TResult].ToList () [0x0002d] in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/src/Xamarin.iOS/external/corefx/src/System.Linq/src/System/Linq/Select.cs:161 
  at System.Linq.Enumerable.ToList[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x0000e] in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/src/Xamarin.iOS/external/corefx/src/System.Linq/src/System/Linq/ToCollection.cs:30 
[Stacktrace of my code is left out ...]
lbnascimento commented 4 years ago

@mwkldeveloper @GW-FUB Could you provide your datafiles?

GW-FUB commented 4 years ago

@lbnascimento Unfortunately not. I made a downgrade to v4 and all of the old files are gone now.

jjxtra commented 4 years ago

Do you get this error on Ubuntu or Windows?

mwkldeveloper commented 4 years ago

After upgrade LiteDb to v5.0.8, the problem seems fixed.

meeety commented 3 years ago

It looks like I got the same issue on 5.0.8, I just posted a new comment under #1940

ccornici commented 2 years ago

Same here, I had about 100 items which I deleted via smth like DELETE mycollectionname from LiteDB Studio, and now I constantly get this error in code: LiteDB.LiteException: 'Invalid Data on 304. Full zero: False. Page Type: Empty. Prev/Next: 4294967295/317. UniqueID: 4. ShareCounter: 1. '