lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

wanji commented 9 years ago

The following code produces lmdb.BadRSlotError in the second iteration (idx==1):

import os
import numpy as np
from dbarray import DBArray

os.system("rm -fr test.db")

for idx in range(2):
    print "###################################"

    # Create random array
    nrows = 1
    ncols = 5
    dtype = np.float32
    arr = np.random.random((nrows, ncols))
    arr = np.require(arr, dtype)
    print arr

    """ `DBArray` from scratch
    """
    # Create `DBArray`
    dba1 = DBArray('test.db')
    # Initialize
    dba1.set_shape((nrows, ncols))
    dba1.set_dtype(dtype)
    # Set rows
    dba1[:] = arr[:]
    # Convert to ndarray
    print dba1.tondarray()

    """ Open exsiting `DBArray`
    """
    # Open another DB
    dba2 = DBArray('test.db')
    print dba2.tondarray()

Error message:

###################################
[[ 0.86903572  0.26334628  0.59591901  0.5937745   0.51802516]]
[[ 0.86903572  0.26334628  0.59591901  0.5937745   0.51802516]]
[[ 0.86903572  0.26334628  0.59591901  0.5937745   0.51802516]]
###################################
[[ 0.42391387  0.74991614  0.7317664   0.62704509  0.33670583]]
Traceback (most recent call last):
  File "bug1.py", line 28, in <module>
    print dba1.tondarray()
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 353, in tondarray
    return self.get_rows(range(self.nrows))
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 229, in get_rows
    resarr[i, :] = self.get_row(v_rid[i])
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 257, in get_row
    self._storage.get(pack(PACK_NUM_TYPE, rid)))
  File "/usr/local/lib/python2.7/dist-packages/dbarray/storage.py", line 103, in get
    with self.env.begin() as txt:
lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

wanji commented 9 years ago

The error occurs while create a new DBArray and assign it to a variable already holds the same DBArray.

Assign None to the variable before the new DBArray, the error disappeared:

...
    # Create `DBArray`
    dba1 = None
    dba1 = DBArray('test.db')
...
    # Open another DB
    dba2 = None
    dba2 = DBArray('test.db')

And now the output is correct:

###################################
[[ 0.58020157  0.8281129   0.42305169  0.92618233  0.48898369]]
[[ 0.58020157  0.8281129   0.42305169  0.92618233  0.48898369]]
[[ 0.58020157  0.8281129   0.42305169  0.92618233  0.48898369]]
###################################
[[ 0.93871009  0.52787697  0.11251965  0.07693885  0.38463211]]
[[ 0.93871009  0.52787697  0.11251965  0.07693885  0.38463211]]
[[ 0.93871009  0.52787697  0.11251965  0.07693885  0.38463211]]

The assignment of None forces the destructor of existing object be called before constructor of the new object.

wanji commented 9 years ago

Delete dba1 before access dba2, similar error occurs again:

...
    # Create `DBArray`
    dba1 = None
    dba1 = DBArray('test.db')
...
    # Open another DB
    dba2 = None
    dba2 = DBArray('test.db')
    del dba1
    print dba2.tondarray()
...

###################################
[[ 0.83253026  0.06177803  0.53406161  0.32942179  0.70239663]]
[[ 0.83253026  0.06177803  0.53406161  0.32942179  0.70239663]]
Traceback (most recent call last):
  File "bug2.py", line 37, in <module>
    print dba2.tondarray()
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 353, in tondarray
    return self.get_rows(range(self.nrows))
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 229, in get_rows
    resarr[i, :] = self.get_row(v_rid[i])
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 257, in get_row
    self._storage.get(pack(PACK_NUM_TYPE, rid)))
  File "/usr/local/lib/python2.7/dist-packages/dbarray/storage.py", line 102, in get
    with self.env.begin() as txt:
lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

Now we can confirm that the reason for error is that we cannot close a database while there are other object relies on the same database.

wanji commented 9 years ago

When used with cc4c, the following error raies:

=========================
Traceback (most recent call last):
  File "convnet2.py", line 685, in <module>
    model.start()
  File "convnet2.py", line 290, in start
    self.train()
  File "convnet2.py", line 306, in train
    self.module_do_write_features()
  File "convnet2.py", line 247, in module_do_write_features
    self.next_dump = self.get_next_batch(None, train=False, dump=True)
  File "convnet2.py", line 378, in get_next_batch
    batch_data = dp.get_next_batch(batchnum)
  File "/home/wanji/dcbir/experiments/cc4c/pyutil/data/provider.py", line 178, in get_next_batch
    (sel_image, sel_label, v_sel_idx) = self.select_data(batchnum)
  File "/home/wanji/dcbir/experiments/cc4c/pyutil/data/provider.py", line 243, in select_data
    sel_image = self.dba[first:last, :].T
  File "/home/wanji/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/dbarray/dbarray.py", line 148, in __getitem__
    rows = self.get_rows(v_rid)
  File "/home/wanji/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/dbarray/dbarray.py", line 229, in get_rows
    resarr[i, :] = self.get_row(v_rid[i])
  File "/home/wanji/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/dbarray/dbarray.py", line 257, in get_row
    self._storage.get(pack(PACK_NUM_TYPE, rid)))
  File "/home/wanji/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/dbarray/storage.py", line 106, in get
    with self.env.begin() as txt:
lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

This error raises randomly.

wanji / dbarray

lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot #1