Cannot open db (lmdb)

[boaerosuke]

It seems that it is not able to read the mnist database file that is downloaded. I checked the installation of lmdb and leveldb installed and in place but i am not able to fix it.

Any ideas?

Traceback (most recent call last): File "tutorial.py", line 224, in <module> workspace.RunNetOnce(train_model.param_init_net) File "/Users/davin/Caffe2Workspace/caffe2/build/caffe2/python/workspace.py", line 183, in RunNetOnce StringifyProto(net), File "/Users/davin/Caffe2Workspace/caffe2/build/caffe2/python/workspace.py", line 166, in CallWithExceptionIntercept return func(*args, **kwargs) RuntimeError: [enforce fail at db.h:190] db_. Cannot open db: /Users/davin/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb of type lmdb Error from operator: output: "dbreader_/Users/davin/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "/Users/davin/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" }

[neha26shah]

Even I am facing the same issue. Earlier I tried to use the tutorial files and it failed. Later on I created my own db with https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/lmdb_create_example.py

and it still fails.

[dlahmad]

I'm also faceing the same issue.

Am 11.08.2017 11:49 nachm. schrieb "Neha Shah" notifications@github.com:

Even I am facing the same issue. Earlier I tried to use the tutorial files and it failed. Later on I created my own db with https://github.com/caffe2/caffe2/blob/master/caffe2/ python/examples/lmdb_create_example.py

and it still fails.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/caffe2/caffe2/issues/1074#issuecomment-321926796, or mute the thread https://github.com/notifications/unsubscribe-auth/AU93K8FSXFZCaIu0kb55wxTbxbujLwU4ks5sXMxzgaJpZM4Oz-4D .

[boaerosuke]

I made the tutorial on another computer and it worked. I think the makefile is not able to find my lmdb packages. I figured it out in the logs of the build file. But I couldn't fix it

[jwatte]

I see this same error, on Ubuntu 16.04, running on an NVIDIA Jetson TX2. This is in the MNIST.ipynb example, and the error is:

Traceback for operator 0 in network mnist_train_init /usr/local/lib/python2.7/caffe2/python/model_helper.py:431

:5 :6 /usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py:2882 /usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py:2822 /usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py:2718 /usr/local/lib/python2.7/dist-packages/ipykernel/zmqshell.py:533 /usr/local/lib/python2.7/dist-packages/ipykernel/ipkernel.py:196 /usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py:399 /usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py:235 /usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py:283 /usr/local/lib/python2.7/dist-packages/tornado/stack_context.py:277 /usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py:414 /usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py:472 /usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py:440 /usr/local/lib/python2.7/dist-packages/tornado/stack_context.py:277 /usr/local/lib/python2.7/dist-packages/tornado/ioloop.py:888 /usr/local/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py:177 /usr/local/lib/python2.7/dist-packages/ipykernel/kernelapp.py:477 /usr/local/lib/python2.7/dist-packages/traitlets/config/application.py:658 /usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:16 /usr/lib/python2.7/runpy.py:72 /usr/lib/python2.7/runpy.py:174 --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) in () 1 # The parameter initialization network only needs to be run once. ----> 2 workspace.RunNetOnce(train_model.param_init_net) ... RuntimeError: [enforce fail at lmdb.cc:20] mdb_status == MDB_SUCCESS. 12 vs 0. Cannot allocate memory Error from operator: output: "dbreader_/home/nvidia/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "/home/nvidia/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" } The databases seem to be in the paths in question: nvidia@tegra-ubuntu:~/caffe2_notebooks/tutorial_data/mnist$ ls -lR /home/nvidia/caffe2_notebooks/tutorial_data/mnist/ /home/nvidia/caffe2_notebooks/tutorial_data/mnist/: total 8 drwxr-xr-x 2 nvidia nvidia 4096 Jun 28 11:52 mnist-test-nchw-lmdb drwxr-xr-x 2 nvidia nvidia 4096 Jun 28 11:52 mnist-train-nchw-lmdb /home/nvidia/caffe2_notebooks/tutorial_data/mnist/mnist-test-nchw-lmdb: total 13448 -rwxr-xr-x 1 nvidia nvidia 13762560 Jun 28 11:52 data.mdb -rwxr-xr-x 1 nvidia nvidia 8192 Jun 28 11:52 lock.mdb /home/nvidia/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb: total 80412 -rwxr-xr-x 1 nvidia nvidia 82333696 Jun 28 11:53 data.mdb -rwxr-xr-x 1 nvidia nvidia 8192 Jun 28 11:52 lock.mdb Deleting the lock.mdb files does not help.

[liyemei]

@boaerosuke you should compile lmdb ,the caffe2 :DUSE_LMDB=OFF .

[kangdongh]

@jwatte I solved this problem by modifying const MAP_SIZE from 1TB to 64GB in caffe2/db/lmdb.cc file and recompiling caffe2.

However, caffe2 allocates same lmdb space (1TB) for each read_db and I don't know why

I'm using Jetson TX2, so I guess that the whole virtual memory space is limited for lmdb :(

[liyemei]

@boaerosuke yes,the caffe2 makefile or CMakeLists.txt did not find lmdb resource on our working machine ,so you should add lmdb.cc 、header file of the lmdb and some dependencies files, such as the windows lib file:lmdb.lib。 In the mac os may be ：.dylib。I also solved the problem by add lmdb.cc and recompiling caffe2

[boaerosuke]

@liyemei oh this makes sense. I´ll try that! Thanks 👍

[subokita]

Just in case if you're stuck or confused, here's how I resolved my issue,

source activate ml      # my conda-env is called ml, just rename to your env
conda install lmdb      # if it hasn't been installed 
cd ~/anaconda/envs/ml  # go to your anaconda install dir and proceed to the environment dir

Ensure that the following files are there

~/anaconda/envs/ml/include/lmdb.h
~/anaconda/envs/ml/lib/liblmdb.a  
~/anaconda/envs/ml/lib/liblmdb.so

Proceed to your caffe2/build directory and run

cmake -DUSE_CUDA=OFF -DUSE_LMDB=OFF        # I don't have CUDA btw

Modify the CMakeCache.txt so that

//Path to a file.
LMDB_INCLUDE_DIR:PATH=/Users/your_username/anaconda/envs/ml/include/

//Path to a library.
LMDB_LIBRARIES:FILEPATH=/Users/your_username/anaconda/envs/ml/lib/liblmdb.so

Then re-run sudo make install, CMake should print out the following:

--   USE_LMDB              : ON
--     LMDB version        : 0.9.21

It seems like FindLMDB.cmake failed to do its job in this particular case

[liyemei]

@boaerosuke make sure your lmdb files like this. //Path to a file. LMDB_INCLUDE_DIR:PATH=/Users/your_username/anaconda/envs/ml/include/

//Path to a library. LMDB_LIBRARIES:FILEPATH=/Users/your_username/anaconda/envs/ml/lib/liblmdb.so

[G-ram]

On Mac, I simply had to turn the option on -DUSE_LMDB=ON when running CMake.

[arakhmati]

I followed the instructions above and linked the lmdb files from anaconda, however that didn't resolve my issue. Then, I realised that cmake was linking default python while I needed to use python from anaconda.

To fix it, I modified the CMakeCache.txt to be as follows:

//No help, variable specified on the command line.
PYTHON_EXECUTABLE:FILEPATH=/home/ubuntu/anaconda3/bin/python

//Path to a file.
PYTHON_INCLUDE_DIR:PATH=/home/ubuntu/anaconda3/include/python3.6m

//Path to a library.
PYTHON_LIBRARY:FILEPATH=/home/ubuntu/anaconda3/lib/libpython3.so

[LMerCy]

@liyemei where should i add LMDB_INCLUDE_DIR:PATH and LMDB_LIBRARIES

[GonChen]

I faced the same issue. This issue is due to cmake is not found lmdb installed. Installing the liblmdb-dev and re-cmake & make caffe2 solved my issue. sudo apt-get install liblmdb-dev

[varelal]

I was able to fix the problem by following the Custom Anaconda install. Here is the link. https://caffe2.ai/docs/getting-started.html?platform=mac&configuration=compile The only change i did was cmake -DCMAKE_PREFIX_PATH=~/anaconda2/envs/my_caffe2_env -DCMAKE_INSTALL_PREFIX=~/anaconda2/envs/my_caffe2_env -DUSE_LMDB=ON ..

[CarlosYeverino]

Any suggestion to fix this issue in windows 10? I think I have a similar issue https://github.com/caffe2/tutorials/issues/7

[yjtan118]

On Windows 10, i have managed to get this to run following some steps from the suggestions above:

• making sure lmdb is installed via conda (conda install lmdb)
• making sure the conda environment files are present in the WINDOWS path variables
• change -DUSE_LMDB=ON in the build_windows.bat
• run the build_windows.bat

Forgot to mention, I'm using VS 2017 64-bit compiler.

[MRJTM]

when i use the old source package , I encourtered this problem but when I choose to clone the latest version of source code and compile it follow the install instructions of caffe2 offical ,then this problem disappeared. And before I do cmake, I first conda install lmdb

[MRJTM]

And When I compile with python2.7 this problem appears but it disappears with python3.6 I didn't change any other configuration , it seem that this problem only happen with python2.7

[johnknelsonintific]

Hi all, I'm currently dealing with this same issue for the same tutorial. I was hoping to follow @subokita fix steps, but I'm not finding an explicit caffe2/build directory. Is this because I've installed caffe2 via Anaconda pre-built binaries(I'm not compiling myself, and I haven't dealt with cmake yet although maybe I should make sure I can) into a specific environment (I've installed lmbd to this environment as well)? The caffe2 I've installed is also not more than a month or two old, so maybe the installation has changed since that fix?

Also sought to update my conda install since @MRJTM mentioned success after recompiling with latest source, but I apparently have the latest caffe2 installation via Anaconda. I am using Python 3.6.5 and conda 4.5.8.

Any other Anaconda users seeing this error?

[campo23]

@johnknelsonintific same error here, i have literally no idea how to fix this problem

[CarlosYeverino]

Related issue stated in pytorch/pytorch#9736

I had to switch to ubuntu to fix this issue.