search

jnwatson / py-lmdb

Universal Python binding for the LMDB 'Lightning' Database
http://lmdb.readthedocs.io/
Other
614 stars 96 forks source link

`lmdb.BadValsizeError` when using database with `dupsort=True` #310

Closed edmondchuc closed 2 years ago

edmondchuc commented 2 years ago

Affected Operating Systems

Affected py-lmdb Version

1.2.1

py-lmdb Installation Method

pip install lmdb

Using bundled or distribution-provided LMDB library?

Bundled

Distribution name and LMDB library version

(0, 9, 29)

Machine "free -m" output

              total        used        free      shared  buff/cache   available
Mem:          12727         121        8624         364        3981       11928
Swap:          4096           0        4096

Other important machine info

Windows WSL2 Ubuntu 20.04

Describe Your Problem

The value's size during txn.put() is limited to around ~500 bytes when using databases with dupsort=True and lmdb.BadValsizeError is raised.

It works fine when performing txn.put() on the main database or on databases where dupsort=False.

Errors/exceptions Encountered

Size of key: 34
Size of value: 960
Traceback (most recent call last):
  File "/home/edmond/projects/rdflib-lmdb/lmdb_max_data_size.py", line 23, in <module>
    txn.put(key, value, db=db)
lmdb.BadValsizeError: mdb_put: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size

Describe What You Expected To Happen

I expected the transaction to commit successfully like it does on the main database.

Describe What Happened Instead

lmdb.BadValsizeError is raised.

jnwatson commented 2 years ago

In dupsort=True DBs, the value size is limited to the maximum key size (512 as built). This is a limitation of the upstream lmdb library.