facebook / rocksdb

A library that provides an embeddable, persistent key-value store for fast storage.
http://rocksdb.org
GNU General Public License v2.0
28.74k stars 6.34k forks source link

[question] Using custom threading library #10682

Open mdorier opened 2 years ago

mdorier commented 2 years ago

I'm using RocksDB in a program that relies on a custom threading library. It's works fine because RocksDB spawns and manage its own POSIX threads, and because I make sure I access RocksDB from only one of my custom threads. Now, I could get more performance if I could make RocksDB use my threads and mutex instead of its own. I have seen various abstract classes like ThreadPool in the RocksDB headers. Is it possible to inject a custom threading layer into RocksDB via some of its abstract classes? Essentially I would need to make RocksDB not just use my threads, but also custom mutex/lock/condition variable classes that my threading library uses, in place of the POSIX ones. How would you go about doing that?

mrambacher commented 2 years ago

My guess is that this is not a trivial operation. If I were you, I would attack it as follows: 1- Find the places that use a thread or mutex (and related classes) and replace those with variants in the "port" namespace (redirection for most) 2- You may also need to do something for thread-local variables. 3- Write your own version of the port classes and interject them 4- You may need to implement your own Env class for the Thread-related functions

Note that #1 and #2 would be things to contribute back as PRs to RocksDB whereas #3/#4 are more likely proprietary code.

Note that the port namespace would allow you to pick your own packages at compile-time. To do something like this at run-time would be more complicated.

Good luck!!

/ Mark

On Thu, Sep 15, 2022 at 10:41 AM Matthieu Dorier @.***> wrote:

I'm using RocksDB in a program that relies on a custom threading library. It's works fine because RocksDB spawns and manage its own POSIX threads, on one hand, and because I make sure I access RocksDB from only one of my custom threads. Now, I could get more performance if I could make RocksDB use my threads and mutex instead of its own. I have seen various abstract classes like ThreadPool in the RocksDB header. Is it possible to inject a custom threading later into RocksDB via some of its abstract classes? Essentially I would need to make RocksDB not just use my threads, but also custom mutex/locks/condition variables classes that my threading library uses, in place of the POSIX ones. How would you go about doing that?

— Reply to this email directly, view it on GitHub https://github.com/facebook/rocksdb/issues/10682, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABCTBCMW6KPO757PERDK4LV6MYTDANCNFSM6AAAAAAQNO6AFU . You are receiving this because you are subscribed to this thread.Message ID: @.***>