mdb_txn_commit failed

[PkuRainBow]

Hi, I have build the project successfully and want to generate the lmdb file using the convert_imageset_and_flow.cpp project. But I have tried for multiple times and meet the exactly the same error:

The error is located at(line #458 convert_imageset_and_flow.cpp) 👍

if (++count % 100 == 0) {
  // Commit txn
  if (db_backend == "leveldb") {  // leveldb
    db->Write(leveldb::WriteOptions(), batch);
    delete batch;
    batch = new leveldb::WriteBatch();
  } else if (db_backend == "lmdb") {  // lmdb
      CHECK_EQ(mdb_txn_commit(mdb_txn), MDB_SUCCESS)
        << "mdb_txn_commit failed";
    CHECK_EQ(mdb_txn_begin(mdb_env, NULL, 0, &mdb_txn), MDB_SUCCESS)
        << "mdb_txn_begin failed";
  } else {
    LOG(FATAL) << "Unknown db backend " << db_backend;
  }
  LOG(ERROR) << "Processed " << count << " files.";

I have tried to increase the initial database size like this,

  // Open db
  //...
  } else if (db_backend == "lmdb") {  // lmdb
    LOG(INFO) << "Opening lmdb " << argv[arg_offset+2];
    CHECK_EQ(mkdir(argv[arg_offset+2], 0744), 0)
        << "mkdir " << argv[arg_offset+2] << "failed";
    CHECK_EQ(mdb_env_create(&mdb_env), MDB_SUCCESS) << "mdb_env_create failed";
    CHECK_EQ(mdb_env_set_mapsize(mdb_env, 109951162777600), MDB_SUCCESS)  // increase the size
        << "mdb_env_set_mapsize failed";
    CHECK_EQ(mdb_env_open(mdb_env, argv[arg_offset+2], 0, 0664), MDB_SUCCESS)
    //...

But I still the same error. Could anyone help me?

[PkuRainBow]

@nikolausmayer

[nikolausmayer]

Hi,

does the description in this thread match your problem case? https://groups.google.com/d/msg/caffe-users/GFaBKPeUvbw/Gf8mKra2BQAJ

You already tried a larger initial size. Could you also try a smaller transaction size? Perhaps % 10 instead of % 100 in if (++count % 100 == 0).

I don't have LMDB experience and probably cannot be of much help here.

[nikolausmayer]

(closed due to inactivity)