search

twmht / python-rocksdb

Python bindings for RocksDB
BSD 3-Clause "New" or "Revised" License
276 stars 90 forks source link

Hard crash on DB.close() #58

Open oersted opened 5 years ago

oersted commented 5 years ago

I experience a hard crash when closing a DB.

I have experimented with a number of diverse settings. No matter the load or the contents of the database it always seems to happen.

This is the case both in master with Python 3 and in Python 2 with the fix from #40. I also tested with both librocksdb-dev 5.17 (from Debian stretch-backports) and the facebook/rocksdb master (>6.1.2).

At one point I got a detailed stack trace and memory map but I can't seem to recover them. I will edit this post if I can produce them later on. I do know that the error was "double free or corruption".

Closing a DB is a critical feature for my use case. I need to be able to reliably populate DBs in subprocesses and transfer the control to the master process afterwards. This is not possible unless the database is properly closed before the transfer. I have tried the del db; gc.collect() strategy from #10 and the unit tests, but, as expected, del is not deterministic and the file lock is not freed in most cases. In any case, if del run it would likely also crash in the close() method.

I believe #46 might be experiencing the same issue.

Please, let me know if this is the case for everyone or if it is an environment issue.

twmht commented 5 years ago

@oersted

Did you try pybind11 branch (https://github.com/twmht/python-rocksdb/tree/pybind11)? It should provide most of the features including stable close function.

And also let me know if there are any features you want but not in the branch.

oersted commented 5 years ago

Thank you, yes I did take a look at pybind11, I was not sure how stable it was. I am currently using the merge operations heavily and I can't find them in the docs at first sight. Are they implemented?

I have also experimented with plain get-put instead of merge, and it seems to work well, so I might try pybind11 anyways.

twmht commented 5 years ago

Yup, merge operators are still missing features in pybind11 branch. But I will add it as soon as possible.

oersted commented 5 years ago

@twmht How is progress on merge operators in pybind11? I have managed to get around this until now, but it is becoming a more important performance bottleneck.

twmht commented 5 years ago

@oersted

yes. It's almost done. Just need some more unit test. The update will be as soon as possible.

oersted commented 5 years ago

@twmht Thanks :)

oersted commented 5 years ago

Hello again @twmht. I thought we figured out how to do this properly, but it looks like it is not fully reliable, weird stuff happens when closing the DB at large scale.

Anyways, I would really appreciate it if you gave a look at finalizing the Merge API for pybind11. I have 2 developers here that can help you if needed, this is fairly critical to our work now.

maxibor commented 3 years ago

Any update on this @twmht @oersted ? Being able to close the database would be lovely :)

binh-vu commented 3 years ago

@maxibor I found that @iFA88 created a fork (https://github.com/iFA88/python-rocksdb) that fixed this issue. Original thread: #46. I repost it here to increase visibility.