Question : Database with DatabaseOpenFlags.DuplicatesSort how to store data bigger than 512 bytes ?

[valeriob]

Hi, i'm having hard time to find docs about it, but it looks like that when database is opened with DatabaseOpenFlags.DuplicatesSort there is a limit on the data value that can be stored, at last i reach an exception when the value is more thatn 512 bytes.

Thank you

[AlgorithmsAreCool]

~Hmm, typically you should only get this error if the key size if the problem. Not the value size.~

~How large are your keys?~ (i'll take a peek at other causes in the mean time)

[AlgorithmsAreCool]

Actually, nvm. I see some code in LMDB that can cause this, i'm trying to interpret it

[AlgorithmsAreCool]

So, I've confirmed that when running under DUPSORT the max size of a value is limited to the max key size (normally 511 bytes). It is an internal LMDB thing 🤷‍♀️.

See also:

https://github.com/LMDB/lmdb/blob/4b6154340c27d03592b8824646a3bc4eb7ab61f5/libraries/liblmdb/mdb.c#L7599 https://github.com/mozilla/rkv/issues/49#issue-334590741

Recommended workaround is to have a 2nd table/database that is not DUPSORT.

Database A: DUPSORT Key : KeyA Value : KeyB

Database 2 : Not DUPSORT Key : KeyB Value : BigValue

[AlgorithmsAreCool]

I apparently also made a note about this in my other issue. I guess I forgot about it 🙃.

https://github.com/CoreyKaylor/Lightning.NET/issues/130

[valeriob]

Thank you very much for your confirmation, i think i've been burned by this in the past but apparenty i forgot too 😄 I guess i'll need to make my key longer, it's probably for the best since i guess that with dupsort there will be some cpu spent on that data, still for time series data it was usefull.

[AlgorithmsAreCool]

Let me know if you have any other questions

[valeriob]

Thank you very much :D