MDB_MAP_FULL error using db::Transaction to commit

[kgl-prml]

I wrote a program to transfer my csv file to lmdb format using db:DB and db::Transaction defined in db.hpp and db_lmdb.hpp. And I encountered a strange error. I doubt if there exists a bug in the db:Transaction definition. convert_csv_data.cpp.txt

In db_lmdb.cpp line 72: int put_rc = mdb_put(mdb_txn, mdb_dbi, &mdb_key, &mdb_data, 0);

And line 84: MDB_CHECK(mdb_txn_commit(mdb_txn));

My result shows that mdb_put succeed (i.e. put_rc = 0) and mdb_txn_commit failed and throws a MDB_MAP_FULL error. （i.e. error code is -30792). So in the void LMDBTransaction::Commit(), it thinks the map size is enough (i.e. put_rc=0), however mdb_txn_commit failed because of MDB_MAP_FULL. Furthermore, the current LMDB map size is 2M only.

Part of the log information is listed as follows:

I0507 15:26:59.594712 123979 convert_csv_data.cpp:119] Processed label 66000 files.
I0507 15:26:59.598665 123979 db_lmdb.cpp:85] Put rc: 0
I0507 15:26:59.598675 123979 db_lmdb.cpp:88] Commit rc: -30792
F0507 15:26:59.598695 123979 db_lmdb.hpp:15] Check failed: mdb_status == 0 (-30792 vs. 0) MDB_MAP_FULL: Environment mapsize limit reached

The convert_csv_data.cpp wrote by myself is in the attachment.

[lukeyeager]

It looks like this is related to my pull request: https://github.com/BVLC/caffe/pull/3731

Can you edit your post to add ```cpp for preformatting? It's hard to tell what you edited.

// like this
int main() {}

[kgl-prml]

OK. Sorry for that. My modifications are as follows: (From line 82 in src/caffe/util/db_lmdb.cpp). And I have created a pull request #4115.

int commit_rc = 0;
if (!out_of_memory) {
    // Commit the transaction
    commit_rc = mdb_txn_commit(mdb_txn);
    //MDB_CHECK(commit_rc);
    if(commit_rc == MDB_MAP_FULL){
       out_of_memory = true;
    } else {
       MDB_CHECK(commit_rc);
       mdb_dbi_close(mdb_env_, mdb_dbi);
       keys.clear();
       values.clear();
    }
} 
if(out_of_memory){
    // Double the map size and retry
    if(commit_rc != MDB_MAP_FULL){
       mdb_txn_abort(mdb_txn);
    }
    mdb_dbi_close(mdb_env_, mdb_dbi);
    DoubleMapSize();
    Commit();
}

[lukeyeager]

Thanks for the bug report @kgl-prml. I had never encountered a MDB_MAP_FULL error coming from mdb_txn_commit(). Can you try this branch and see if it solves your problem? If so, I'll create a pull request to get it added to master.

https://github.com/BVLC/caffe/compare/master...lukeyeager:bvlc/fix-lmdb-pr

[lukeyeager]

And I have created a pull request #4115.

Oh sorry, I hadn't seen that yet.