kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
486 stars 39 forks source link

how to close lmdb-store instances #27

Closed saasira closed 3 years ago

saasira commented 3 years ago

i apologize for submitting this as an issue but could not find any option to close the opened lmdb store instances anywhere in the docs

i understand it is safe to terminate lmdb database without closing but why not close explicitly for the added comfort of guaranteed safety?

node-lmdb has the close function but i am surprise lmdb-store does not expose that function through its api

excuse me if i am mistaken in saying close function is missing in lmdb-store api

kriszyp commented 3 years ago

There actually was a close() method on stores, but I added docs and a test for it (and made sure only the root store closes the env). Let me know if you have any issues with it.

saasira commented 3 years ago

Hi Kris,

thanks for the quick response

i have another question about configuration; should i raise those as separate issues or is it okay to ask here?

anyway, i am putting the question here, if you suggest to raise separate ticket i will do so later.

i did not see an option to set MDB_NORDAHEAD when opening database; as per documentation this helps reduce unnecessary reads when the usage pattern is mostly random [which i am interested in]

thanks, Samba

kriszyp commented 3 years ago

Yes, it's perfectly fine to ask here. Setting MDB_NORDAHEAD should be available with noReadAhead: true flag. I will add that to the documentation so that is clear.

kylebernhardy commented 3 years ago

In regards to closing an environment I am experiencing an error if I execute a close after a query. It seems to be a timing issue for the read transaction to close before the environment does. If I wrap the close in a setTimeout even with a time of 0 there is no error. I have provided both in the code below:

const fs = require('fs');
const path = require('path');
const lmdb = require('lmdb-store');

function createEnv(name){
    let environment_path = path.join(__dirname, 'data', name);
    try {
        fs.rmdirSync(environment_path, {recursive:true})
        fs.mkdirSync(environment_path);
    }catch(e){
        console.error(e);
    }
    let env_init = {
        "path": environment_path,
        "mapSize": 1024*1024,
        "maxReaders": 100,
        sharedStructureKey: Symbol.for('structures')
    };
    let env = lmdb.open(env_init);
    return env;
}

let env = createEnv('test');

let id_dbi = env.openDB('id', {
    create: true,
    dupSort: false,
    useVersions: true});

let promise;
let objs = [];
for(let x = 0, length = 100000; x<length; x++){
    objs.push({id:'d'+x, timestamp: Date.now()});
}

for(let x = 0, length = objs.length; x<length; x++){
    let obj = objs[x];
    promise = id_dbi.ifNoExists(x, ()=>{
        id_dbi.put(obj.id, obj);
    });
}

promise.then(d=>{
    let rez = [];
    for (let { key, value } of id_dbi.getRange({  })) {
        rez.push(value);
    }
    //this commented block will stop the error
    /*setTimeout(()=>{
        env.close();
    }, 0);*/

    //this causes the error
    env.close();
});
kriszyp commented 3 years ago

Should have fixes for these in 0.9.12. Let me know if there are issues.

kylebernhardy commented 3 years ago

The fix works great! Thank you Kris.