search

Venemo / node-lmdb

Node.js binding for lmdb
MIT License
361 stars 72 forks source link

readme

node-lmdb

This is a node.js binding for LMDB, an extremely fast and lightweight transactional key-value store database.

Donate

About

About this module

The aim of this node module is to provide bindings so that people can use LMDB from their node applications, aiming for a simple and clean API which is on par with the LMDB API but tries to apply javascript patterns and naming conventions as much as possible to make users feel familiar about it.

We support zero-copy retrieval of string and binary values. Binary values are operated on via the Node.js Buffer API.

About LMDB

Here are the main highlights of LMDB, for more, visit https://www.symas.com/lmdb :)

Supported platforms

License info

The node-lmdb code is licensed to you under the terms of the MIT license. LMDB itself is licensed under its own OpenLDAP public license (which is similarly permissive).

Usage

Introduction

Step 0: require the module

Just like with any other node module, the first step is to require() the module.

var lmdb = require('node-lmdb');

Step 1: create an environment

Env represents a database environment. You can create one with the new operator and after that, you must open it before you can use it. open() accepts an object literal in which you can specify the configuration options for the environment.

var env = new lmdb.Env();
env.open({
    path: __dirname + "/mydata",
    mapSize: 2*1024*1024*1024, // maximum database size
    maxDbs: 3
});

Close the environment when you no longer need it.

env.close();

Step 2: open one or more databases

An environment (Env) can contain one or more databases. Open a database with env.openDbi() which takes an object literal with which you can configure your database.

var dbi = env.openDbi({
    name: "myPrettyDatabase",
    create: true // will create if database did not exist
})

Close the database when you no longer need it.

dbi.close();

Step 3: use transactions

The basic unit of work in LMDB is a transaction, which is called Txn for short. Here is how you operate with your data. Every piece of data in LMDB is referred to by a key. You can use the methods getString(), getBinary(), getNumber() and getBoolean() to retrieve something, putString(), putBinary(), putNumber() and putBoolean() to store something and del() to delete something.

IMPORTANT: always close your transactions with abort() or commit() when you are done with them.

var txn = env.beginTxn();
var value = txn.getString(dbi, 1);

console.log(value);

if (value === null) {
    txn.putString(dbi, 1, "Hello world!");
}
else {
    txn.del(dbi, 1);
}

txn.putString(dbi, 2, "Yes, it's this simple!");
txn.commit();

Asynchronous batched operations

You can batch together a set of operations to be processed asynchronously with node-lmdb. Committing multiple operations at once can improve performance, and performing a batch of operations and using sync transactions (slower, but maintains crash-proof integrity) can be efficiently delegated to an asynchronous thread. In addition, writes can be defined as conditional by specifying the required value to match in order for the operation to be performed, to allow for deterministic atomic writes based on prior state. The batchWrite method accepts an array of write operation requests, where each operation is an object or array. If it is an object, the supported properties are:

If the write operation is a specified with an array, the supported elements are:

When batchWrite is called, node-ldmb will asynchronously create a new write transaction, execute all the operations in the provided array, except for any conditional writes where the condition failed, and commit the transaction, if there were no errors. For conditional writes, if the condition did not match, the write will be skipped, but the transaction will still be committed. However, if any errors occur, the transaction will be aborted. This entire transaction will be created by node-lmdb and executed in a separate thread. The callback function will be called once the transaction is finished. It is possible for an explicit write transaction in the main JS thread to block or be blocked by the asynchronous transaction. For example:

env.batchWrite([
    [dbi, key1, Buffer.from("Hello")], // put in key 1
    [dbi, key2, Buffer.from("World")], // put in key 2
    [dbi, key3], // delete any entry from key 3 (can also use null as value to indicate delete)
    [dbi, key4, valuePlusOne, oldValue] // you could atomically increment by specifying the require previous state
], options, (error, results) => {
    if (error) {
        console.error(error);
    } else {
        // operations finished and transaction was committed
        let didWriteToKey4Succeed = results[3] === 0
    }
})

The callback function will be either be called with an error in the first argument, or an array in the second argument with the results of the operations. The array will be the same length as the array of write operations, with one to one correspondence by position, and each value in the result array will be: 0 - Operation successfully written 1 - Condition not met (only can happen if a condition was provided) 2 - Attempt to delete non-existent key (only can happen if ignoreNotFound enabled)

The options include all the flags from put options, and this optional property:

Basic concepts

LMDB has four different entities:

Here is how you use LMDB in a typical scenario:

Example iteration over a database with a Cursor:

var cursor = new lmdb.Cursor(txn, dbi);

for (var found = cursor.goToFirst(); found !== null; found = cursor.goToNext()) {
    // Here 'found' contains the key, and you can get the data with eg. getCurrentString/getCurrentBinary etc.
    // ...
}

The cursor goTo methods (goToFirst, goToNext, etc.) will return the current key. When an item is not found, null is returned. Beware that the key itself could be a falsy JavaScript value, so you need to explicitly check against null with the !== operator in your loops.

Data Types in node-lmdb

LMDB is very simple and fast. Using node-lmdb provides close to the native C API functionally, but expressed via a natural javascript API. To make simple things simple, node-lmdb defaults to presenting keys and values in LMDB as strings. For convenience number, boolean and Buffer values are also supported.

The simplest way to store complex data types (such as objects) is to use JSON.stringify before putting it into the database and JSON.parse when you retrieve the data.

For more complex use cases access to keys and values as binary (node.js Buffer type) is provided. In LMDB itself keys (with one exception) and values are simply binary sequences of bytes. You can retrieve a key or value from an LMDB database as binary even if it was written as a string. The same does not apply in reverse! Using binary access also allows interoperation with LMDB databases created by, or shared with applications that use data serialisation formats other than UTF-16 strings (including, in particular, strings using other encodings such as UTF-8).
See our chapter Working with strings for more details.

Keys

When using a cursor keys are read from the database and it is necessary to specify how the keys should be returned. The most direct mapping from LMDB C API is as a node.js Buffer (binary), however it is often more convenient to return the key as a string, so that is the default.

You can specify the key type when you open a database:

dbi = env.openDbi({
    // ... etc.
    keyIsBuffer: true
});

When working with transactions, you can override the key type passed to openDbi by providing options to put, get and del functions.
For example:

var buffer = new Buffer('48656c6c6f2c20776f726c6421', 'hex');
var key = new Buffer('key2');
txn.putBinary(dbi, key, buffer, { keyIsBuffer: true });
var data = txn.getBinary(dbi, key, { keyIsBuffer: true });
data.should.deep.equal(buffer);
txn.del(dbi, key, { keyIsBuffer: true });

Finally, when working with cursors, you can override the key type by passing similar options as the 3rd argument of the Cursor constructor:

cursor = new lmdb.Cursor(txn, dbi, { keyIsBuffer: true });

Examples

You can find some in the source tree. There are some basic examples and I intend to create some advanced ones too.

The basic examples we currently have:

Advanced examples:

Caveats

Unsafe Get Methods

Because of the nature of LMDB, the data returned by txn.getStringUnsafe(), txn.getBinaryUnsafe(), cursor.getCurrentStringUnsafe() and cursor.getCurrentBinaryUnsafe() is only valid until the next put operation or the end of the transaction. Also, with Node 14+, you must detach the buffer after using it, by calling env.detachBuffer(buffer). This must be done before accessing the same entry again (or V8 will crash). If you need to use the data later, you can use the txn.getBinary(), txn.getString(), cursor.getCurrentBinary() and cursor.getCurrentString() methods. For most usage, the optimisation (no copy) gain from using the unsafe methods is so small as to be negligible - the Unsafe methods should be avoided.

Working with strings

Strings can come from many different places and can have many different encodings. In the JavaScript world (and therefore the node.js world) strings are encoded in UTF-16, so every string stored with node-lmdb is also encoded in UTF-16 internally. This means that the string API (getString, putString, etc.) will only work with UTF-16 encoded strings.

If you only use strings that come from JavaScript code or other code that is a “good node citizen”, you never have to worry about encoding.

How to use other encodings

This has come up many times in discussions, so here is a way to use other encodings supported by node.js. You can use Buffers with node-lmdb, which are a very friendly way to work with binary data. They also come in handy when you store strings in your database with encodings other than UTF-16.

You can, for example, read a UTF-8 string as a buffer, and then use Buffer's toString method and specify the encoding:

// Get stored data as Buffer
var buf = txn.getBinary(dbi, key);
// Use the Buffer toString API to convert from UTF-8 to a JavaScript string
var str = buf.toString('utf8');

Useful links:

Storing UTF-16 strings as Buffers

While node.js doesn't require the UTF-16 strings to be zero-terminated, node-lmdb automatically and transparently zero-terminates every string internally. As a user, this shouldn't concern you, but if you want to write a string using the Buffer API and read it as a string, you are in for a nasty surprise.

However, it will work correctly if you manually add the terminating zero to your buffer.

Conceptually, something like this will work:

// The string we want to store using a buffer
var expectedString = 'Hello world!';

// node-lmdb internally stores a terminating zero, so we need to manually emulate that here
// NOTE: this would NEVER work without 'utf16le'!
var buf = Buffer.from(expectedString + '\0', 'utf16le');

// Store data as binary
txn.putBinary(dbi, key, buf);

// Retrieve same data as string and check
var data3 = txn.getString(dbi, key);

// At this point, data3 is equal to expectedString
Build Options

A few LMDB options are available at build time, and can be specified with options with npm install (which can be specified in your package.json install script): npm install --use_vl32=true: This will enable LMDB's VL32 mode, when running on 32-bit architecture, which adds support for large (multi-GB) databases on 32-bit architecture. npm install --use_fixed_size=true: This will enable LMDB's fixed-size option, when running on Windows, which causes Windows to allocate the full file size needed for the memory-mapped allocation size. The default behavior of dynamically growing file size as the allocated memory map, while convenient, uses a non-standard Windows API and can cause significant performance degradation, but using the fixed size option ensures much more stable/better performance on Windows (consider using lmdb-store on top of node-lmdb for automated memory-map growth).

On MacOS, there is a default limit of 10 robust locked semaphores, which imposes a limit on the number of open write transactions (if you have over 10 db environments with a write transaction). If you need more concurrent write transactions, you can increase your maximum undoable semaphore count by setting kern.sysv.semmnu on your local computer. Or you can build with POSIX semaphores, using npm install --use_posix_semaphores=true. However POSIX semaphores are not robust semaphores, which means that if you are running multiple processes and one crashes in the midst of transaction, it may block other processes from starting a transaction on that environment. Or try to minimize overlapping transactions and/or reduce the number of db environments (and use more databases within each environment).

Limitations of node-lmdb

Contributing

If you find problems with this module, open an issue on GitHub. Also feel free to send me pull requests. Contributions are more than welcome! :)

Building node-lmdb

LMDB is bundled in node-lmdb so you can simply build this module using node-gyp.

# Install node-gyp globally (needs admin permissions)
npm -g install node-gyp

# Clone node-lmdb
git clone git@github.com:Venemo/node-lmdb.git

# Go to node-lmdb directory
cd node-lmdb

# At first, you need to download all dependencies
npm install

# Once you have all the dependencies, the build is this simple
node-gyp configure
node-gyp build

Building node-lmdb on Windows

Windows isn't such a great platform for native node addons, but it can be made to work. See this very informative thread: https://github.com/nodejs/node-gyp/issues/629

  1. Install latest .NET Framework (v4.6.2 at the time of writing)
  2. Install latest node.js (v7.9.0 at the time of writing).
  3. This is Windows. Reboot.
  4. Now open a node.js command prompt as administrator and run the following commands.
    NOTE: these commands WILL take a LOT of time. Please be patient.
npm -g install windows-build-tools
npm -g install node-gyp
npm -g install mocha
npm config set msvs_version 2015 --global

After this, close the command prompt and open a new one (so that changes to PATH and whatever else can take proper effect). At this point you should have all the necessary junk for Windows to be able to handle the build. (You won't need to run node as administrator anymore.) Note that windows-build-tools will silently fail to install if you don't have the .NET Framework installed on your machine.

  1. Add python2 to PATH. Note that windows-build-tools installed python2 (v2.7.x) for you already, so easiest is to use "Change installation" in the Control Panel and select "Change" and then "Add python.exe to PATH".
  2. This is Windows. Reboot again just to be sure.

Congrats! Now you can work with native node.js modules.

When you are building node-lmdb for the first time, you need to install node-lmdb's dependencies with npm install:

cd node-lmdb
npm install

Note that npm install will also attempt to build the module. However once you got all the dependencies, you only need to do the following for a build:

cd node-lmdb
node-gyp configure
node-gyp build

Managing the LMDB dependency

# Adding upstream LMDB as remote
git remote add lmdb https://git.openldap.org/openldap/openldap.git
# Fetch new remote
git fetch lmdb
# Adding the subtree (when it's not there yet)
git subtree add  --prefix=dependencies/lmdb lmdb mdb.master --squash
# Updating the subtree (when already added)
git subtree pull --prefix=dependencies/lmdb lmdb mdb.master --squash

Developer FAQ

How fast is this stuff?

LMDB is one of the fastest databases on the planet, because it's in-process and zero-copy, which means it runs within your app, and not somewhere else, so it doesn't push your data through sockets and can retrieve your data without copying it in memory.

We don't have any benchmarks for node-lmdb but you can enjoy a detailed benchmark of LMDB here: http://symas.com/mdb/microbench/ obviously, the V8 wrapper will have some negative impact on performance, but I wouldn't expect a significant difference.

Why is the code so ugly?

Unfortunately, writing C++ addons to Node.js (and V8) requires a special pattern (as described in their docs) which most developers might find ugly. Fortunately, we've done this work for you so you can enjoy LMDB without the need to code C++.

How does this module work?

It glues together LMDB and Node.js with a native Node.js addon that wraps the LMDB C API.

Zero-copy is implemented for string and binary values via a V8 custom external string resource and the Node.js Buffer class.

How did you do it?

These are the places I got my knowledge when developing node-lmdb:

Acknowledgements

Below you can find a list of people who have contributed (in alphabetical order). Big thank you to everybody!
(NOTE: if you think your name should be here, but isn't, please contact the author.)

Support

node-lmdb is licensed to you under the terms of the MIT license, which means it comes with no warranty by default.

However,

You can also consider donating to support node-lmdb development:

Donate