search

mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.6k stars 1.25k forks source link

LiteDB.LiteException: 'Document size exceed 2037500 limit' in v5 #1295

Open tom-richtr opened 5 years ago

tom-richtr commented 5 years ago

There is currently a limit on document size. Would it be possible to disable or change it? It is better to keep the size of documents as small as possible but in our scenario we have bigger documents which are not changed or read so much. This was already addressed in issue #645 and the restriction was not enforced. Is it possible to keep it same in v5 or set the limit by user settings?

mbdavid commented 5 years ago

Hi @tom-richtr, you are the second one who talk about this. Current v5 version has a complete different way to store pages and now I need control document segments. I implement a single byte to control this segments (0-255). Each page contains 8159 free bytes... so, why is limited in 2Mb document.

I will think about in use 2 bytes to to this (and extend to 500mb document size)... let's me check if is possible

Jacknq commented 5 years ago

Wasnt the document size unlimited in v4?

mbdavid commented 5 years ago

@Jack85, yes, in v4 is unlimited because use another technique to store big documents (use Extend Pages).

Jacknq commented 5 years ago

I think it would be cool in v5 also. To have it limitless and possibly work with chunks, to be backwatds compatible..

IggaF commented 4 years ago

Hi @mbdavid, we still seeing this limit exception while using v5-rc. We need to store documents that occasionally have single string value of more than 2mb in size, which can't be divided/reduced etc... We can compress that value, but then we won't be able to use LiteDB Studio for debug - which we really like :)

Have you tried increasing it to 2 bytes? Or maybe you can suggest to store these documents differently? Also, once they are written - they are used as read-only

And thanks for LiteDb - its really good!

Tviljan commented 4 years ago

I would hope for same limits as mongodb, 16Mb

mbdavid commented 4 years ago

Hi @IggaF and @Tviljan, this limitation will remove soon. I change how engine stores splitted document. Will be no limit for document size (but still recommend to keep "small as possible")

aktxyz commented 4 years ago

great !!! how soon is soon :)

PeterHagen commented 4 years ago

Our production site seems to have broken due to this issue. So this update is really welcome!

mbdavid commented 4 years ago

Hi @PeterHagen, it's on 5.1 roadmap :)

Dhillu commented 4 years ago

@mbdavid Any ETA on v5.1 release ?

nightroman commented 4 years ago

@Dhillu This was already addressed in this release: https://github.com/mbdavid/LiteDB/releases/tag/v5.0.5

The issue may be closed, IMHO. ~16mb is on par with MongoDB, at least. Unless, of course, there is a plan to increase this limit in v5.1.

Dhillu commented 4 years ago

@nightroman yeah but still it is throwing this error which is marked as done for the future release v5.1.

nightroman commented 4 years ago

@Dhillu If your document size is less than ~16mb and it cannot be saved in v5.0.5 then it is probably a bug. You may describe and submit it separately, this may bring some more attention to your issue.

Dhillu commented 4 years ago

@nightroman I am getting the same error #1295

"LiteDB.LiteException: 'Document size exceed 2037500 limit' in v5" Any suggestions on fixing this ?

nightroman commented 4 years ago

Any suggestions on fixing this ?

No. If your document is larger than 16mb than maybe (it is up to developers to answer) it is not going to be fixed. Maybe it is the permanent limit. Like in MongoDB.

nightroman commented 4 years ago

"LiteDB.LiteException: 'Document size exceed 2037500 limit' in v5"

If you get this error then you are not using the latest LiteDB v5..0.5 where the limit is increased to about 16mb. What is your LiteDB version?

Dhillu commented 4 years ago

v5.0.5 @nightroman

nightroman commented 4 years ago

v5.0.5 @nightroman

Then you cannot get the same error. What is your error exactly? Like I said, describe your case as a new issue, provide the details. It will be easier for developers (or observers like me) to help you.

Fungusware commented 4 years ago

"LiteDB.LiteException: 'Document size exceed 2037500 limit' in v5"

Is occurring in 5.0.8 Is there any way to open the DB once this limit is hit.

lbnascimento commented 4 years ago

@Fungusware I'm not sure how you're getting this error in v5.0.8, since the document size limit was increased from 2MB to 16MB in v5.0.5. Are you sure you're running v5.0.8?

Fungusware commented 4 years ago

Fair call, the error my users were getting was the 2Mb limit, in my testing it was the 16Mb

The issue though is that we cannot upgrade to v5 as many of the existing v4 databases have these large documents in. Is 5.1 removing the limit?

lbnascimento commented 4 years ago

@Fungusware The limit was raised to 16MB in v5.0.5 and is currently on par with other document-oriented databases like MongoDB. Using documents even larger than that would severely hurt performance.

hatzhang commented 1 year ago

Having the same issue, our winform client use LiteDB to persistent memory caches when it exits, and re-cache those when bootup, and we are using the InsertBulk api. It's good for us to just ignore those documents which has a size exceeds the maxium limit. Can we make this a configurable options, or maybe add a new parameter to InsertBulk to indicate whether to throw exception, or just ignore.