Cant write string values longer than 996 characters

[eramirez51]

I love this library, but unfortunately, if I write very long string values, the values gets replaced to -2 int value. Could you please help?

Here is my lmdb setting.

        let lmdbStore = open({
            path: lmdbDir,
            compression: true,
            commitDelay: 10,
            dupSort: false,
            syncBatchThreshold: 100000000
        });

[kriszyp]

Do you have any more details on how you are writing the strings? I have tested string values up to 50MB without issues. There may be issues with strings over 100MB, but that may be due to fundamental constraints of V8, not sure, but I could investigate.

[eramirez51]

Thank you for your reply.

It seems related to how I open the lmdb reader from the client.

Here is my writer

const { open } = require('lmdb-store');

//If you remove one character in the `value` below, the bug disappears
const value = "1230000265545|1230000287520|1230000049329|1230000323011|1230000327724|1230000342441|1230000261597|1230000134169|1230000130074|1230000144751|1230000160197|1230000151405|1230000162760|1230000344523|1230000344523|12300002655451230000265545|1230000287520|1230000049329|1230000323011|1230000327724|1230000342441|1230000261597|1230000134169|1230000130074|1230000144751|1230000160197|1230000151405|1230000162760|1230000344523|1230000344523|12300002655451230000265545|1230000287520|1230000049329|1230000323011|1230000327724|1230000342441|1230000261597|1230000134169|1230000130074|1230000144751|1230000160197|1230000151405|1230000162760|1230000344523|1230000344523|12300002655451230000265545|1230000287520|1230000049329|1230000323011|1230000327724|1230000342441|1230000261597|1230000134169|1230000130074|1230000144751|1230000160197|1230000151405|1230000162760|1230000344523|1230000344523|12300002655451230000265545|1230000287520|1230000049329|1230000323011|1230000327724|1230000327724|1230000327724|1234567"
const key = "test"

const lmdbStore = open({
    path: ".",
    compression: true,
    commitDelay: 10,
    dupSort: false,
    syncBatchThreshold: 100000000
});

lmdbStore.put(key,value)

Here is my reader (in a different app (API server))

const { open } = require('lmdb-store');

const key = "test"

const lmdbStore = open({path: "."});

console.log(lmdbStore.get(key))
//outputs `-2`

Looking deeper, if I change my reader to below, there is no problem regardless of value size.

const { open } = require('lmdb-store');

const key = "test"
const lmdbStore = open({
    path: ".",
    compression: true,
    commitDelay: 10,
    dupSort: false,
    syncBatchThreshold: 100000000
});
console.log(lmdbStore.get(key))

My environment

"lmdb-store": "^1.1.11",

node --version:
 v14.5.0

uname -a
Darwin dev-adminnoMacBook-Pro.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64

[kriszyp]

Oh, I see. I think the issue is that you must be consistent with your use of the compression setting. I believe you are enabling compression in one process, but in the other you don't have compression enabled, so it doesn't attempt to decompress the file, and so you are getting compressed data that hasn't been decompressed. This is triggered at this particular string size because the default threshold for performing compression is 1000 bytes (only compresses if 1000 bytes or more). So just make sure you are being consistent with your compression setting.

[eramirez51]

Oh. That makes sense. Thanks for your help.