How to troubleshoot LMDB ?

[kk00ss]

org.fusesource.lmdbjni.LMDBException: Invalid argument

After 1-10 successful transactions with large number of writes to 8 different databases it fails on database.put in particular database. Sometimes the problem is with document of size 210KB, first one, but at other times it works fine with that document and fails on smaller ones. Could anyone please point me in a right direction ? Issue can be reproduced my project and about any large document. I'm wondering does anyone uses LMDB via lmdbjni in production ? Thanks

[krisskross]

This is actually LMDB native that return a EINVAL. Hard to know exactly what the problem might be. As you can see from C code there are a few cases where this can happen, like incorrect flags, invalid transactions etc. Hope this helps.

int
mdb_put(MDB_txn *txn, MDB_dbi dbi,
    MDB_val *key, MDB_val *data, unsigned int flags)
{
    MDB_cursor mc;
    MDB_xcursor mx;
    int rc;

    if (!key || !data || !TXN_DBI_EXIST(txn, dbi, DB_USRVALID))
        return EINVAL;

    if (flags & ~(MDB_NOOVERWRITE|MDB_NODUPDATA|MDB_RESERVE|MDB_APPEND|MDB_APPENDDUP))
        return EINVAL;

    if (txn->mt_flags & (MDB_TXN_RDONLY|MDB_TXN_BLOCKED))
        return (txn->mt_flags & MDB_TXN_RDONLY) ? EACCES : MDB_BAD_TXN;

    mdb_cursor_init(&mc, txn, dbi, &mx);
    mc.mc_next = txn->mt_cursors[dbi];
    txn->mt_cursors[dbi] = &mc;
    rc = mdb_cursor_put(&mc, key, data, flags);
    txn->mt_cursors[dbi] = mc.mc_next;
    return rc;
}

[kk00ss]

Well, that sorta helps, thank you. I've inserted code that prints a list of present keys, before storing new data, there is no overwrite happening. But now it fails after a few commits while attempting to create cursor for write transaction - and I think that means it's something with transaction itself, do you know if there are any limits on single transaction size ? After creating cursor throws exception - another attempt to do that gives different exception: org.fusesource.lmdbjni.LMDBException: Native object has been freed. for transaction. After removing the code for storing largest chunks of data - I see MDB_BAD_TXN: Transaction must abort, has a child, or is invalid. How could transaction become invalid, I'm not using child transactions.

[krisskross]

Can you provoke the error consistently every time? If not, then it might be a concurrency issue. For example, you may not share transactions between threads. Or maybe you commit/abort the transaction by mistake at some point?

Try isolate the problem, remove code, keep only minimum needed. Then provoke the error again. If you can't reproduce the error from there then move backward by adding more and more code back until you hit the problem.

[kk00ss]

Thanks, error happens inconsistently (I've removed all multi-threading) on some of transactions, sometimes it's totally fine to add particular document, and sometimes it fails. I'll try another kv-storage to see if issue will persist.

[krisskross]

Me and Ben are doing a rewrite of LMDB using JNR-FFI which is almost finished now. You could try that if you want?

https://github.com/lmdbjava/lmdbjava

[kk00ss]

You should take a look at mdbx from this guys https://github.com/ReOpen/ReOpenLDAP It's a rewrite of LMDB from Russian company, they use it in production. It only supports linux. I think you mentioned rewrite of LMDB driver, not database itself ? BoltDB is the most famous LMDB rewrite and it is considerably slower, but pure GO, and guys claim it's code is much cleaner.

[krisskross]

Poor choice of words. This is not a rewrite of LMDB only the Java and JNI code around it. It should be more robust since its using libffi (not hawtjni) https://sourceware.org/libffi.

[kk00ss]

Just for history, If someone will google similar problem. Found cause - closing database after closing environment, I'm wandering why it could work for a few attempts. LMDB could be strange sometimes, but it's awesome. On my rig it's 2x MapDB on write speed, and write speed is its "weak spot".

[unoexperto]

@kk00ss Could you please advise why I'm getting same error while doing this ?

val parentTxn = env.createReadTransaction()
val txn = env.createTransaction(txnParent, true) // exception occurs here

exception stack-trace

Exception in thread "main" org.fusesource.lmdbjni.LMDBException: Invalid argument
    at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:44)
    at org.fusesource.lmdbjni.Env.createTransaction(Env.java:451)

DB is open in write mode.

[kk00ss]

Hi, there's another driver you would be better off using. It's from the same team. As this one.

On Jun 19, 2017 18:54, "unoexperto" notifications@github.com wrote:

@kk00ss https://github.com/kk00ss Could you please advise why I'm getting same error while doing this ?

val parentTxn = env.createReadTransaction() val txn = env.createTransaction(txnParent, true) // exception occurs here

exception stack-trace

Exception in thread "main" org.fusesource.lmdbjni.LMDBException: Invalid argument at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:44) at org.fusesource.lmdbjni.Env.createTransaction(Env.java:451)

DB is open in write mode.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deephacks/lmdbjni/issues/72#issuecomment-309483608, or mute the thread https://github.com/notifications/unsubscribe-auth/ADz6Eksf--23HPTAQz_XZYT8YN9dxrxSks5sFpm2gaJpZM4JBKK7 .

[unoexperto]

@kk00ss I'd love to use it!! In fact I wrote first implementation using lmdbjava which is much faster than lmdbjni but it doesn't support custom comparator that I need :( Here is the issue: https://github.com/lmdbjava/lmdbjava/issues/56

Could you please suggest what to do ?

[kk00ss]

I would go for implementing that comparator. Should be easier then getting this driver to work.

On Jun 19, 2017 19:35, "unoexperto" notifications@github.com wrote:

@kk00ss https://github.com/kk00ss I'd love to use it!! In fact I wrote first implementation using lmdbjava which is much faster than lmdbjni but it doesn't support custom comparator that I need :( Here is the issue: lmdbjava/lmdbjava#56 https://github.com/lmdbjava/lmdbjava/issues/56

Could you please suggest what to do ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deephacks/lmdbjni/issues/72#issuecomment-309495459, or mute the thread https://github.com/notifications/unsubscribe-auth/ADz6Ekv4FHOEUUuDKE6srAKn1Xhq8piLks5sFqM0gaJpZM4JBKK7 .

[krisskross]

@unoexperto That error code is reported by LMDB and its hard to tell what you're doing wrong from the snippet you posted.

[krisskross]

Regarding the comparator, we have not yet tried to implement it for lmdbjava but is supect that its possible using JNR. You could try implementing it yourself and post a PR.

[unoexperto]

@krisskross @kk00ss I'll be happy to contribute but I looked into it and couldn't find example of passing java callback to https://github.com/jnr/jnr-ffi Does it support it ?

[krisskross]

Not sure. Haven't looked at the API. Ask them if you don't find it.

[benalexau]

In another (closed source) project I have JNR-FFI handling a callback. Here's a (slightly redacted) method signature I put into the Library class, ApiErrorCode process(String filename, Callback callback). The callback interface itself:

import jnr.ffi.Pointer;
import jnr.ffi.annotations.Delegate;

  public interface Callback {
    @Delegate
    CallbackReturn next(Pointer sys, Pointer msg);
  }

Feel free to contribute a PR for the LmdbJava ticket 56 and I'll be happy to review.

[benalexau]

Feel free to contribute a PR for the LmdbJava ticket 56 and I'll be happy to review.

FYI LmdbJava 0.6.0 and above now has mdb_set_compare support for user-defined comparators.

[Johny-Kirth]

INFO: org.fusesource.lmdbjni.LMDBException: Invalid argument Am getting this exception while setting maxreaders in LMDBJNI...

Env env = new Env(); env.open(LMDB_DIR); env.setMaxReaders(150); db = env.openDatabase("sub");

Env is created and initialized in the given path.. But I cannot set maxreaders in LMDBJNI as exception arises. If I skip setting maxreaders, then also am getting exception

org.fusesource.lmdbjni.LMDBException:MDB_READERS_FULL : Environment maxreaders limit reached at org.fusesource.lmdbjni.Util.checkErrorCode at org.fusesource.lmdbjni.Env.createTransaction at org.fusesource.lmdbjni.Env.createReadTransaction at org.fusesource.lmdbjni.Database.get(Database.java:170)

Kindly resolve this issue as this is banging my head for more than a week... Since am new to java programming... Thanks in advance

[benalexau]

@Johny-Kirth perhaps try https://github.com/lmdbjava/lmdbjava. It has an up-to-date tutorial and this may make it easier for you to get started.