search

estraier / tkrzw

a set of implementations of DBM
Apache License 2.0
164 stars 20 forks source link

great work! any production users cases studies? who is using this? #37

Open hiqsociety opened 2 years ago

hiqsociety commented 2 years ago

great work! any production users cases studies? who is using this?

tieugene commented 1 year ago

great work! any production users cases studies? who is using this?

I use for BTC blockchain exploring. ~2 billions keys. I choose it after long and hard tests: https://github.com/tieugene/kvtest. Try yourself.

kostya commented 1 year ago

And what is your results?

tieugene commented 1 year ago

And what is your results?

As for my profile tkrzw is the best. My profile:

As for 2^30 records on my hardware (kps == kilorecords per second):

You can test it today with kvtest above.

derekbsnider commented 1 year ago

I too am interested, especially in how tkrzw compares to previous libraries.

More specifically, I've been using QDBM for well over 15 years for a specific purpose, whereby I need to maintain and update B+ Tree database files which are being read over NFS, and QDBM Villa does a very good job of being able to update these files in place on one server, while they are being read over NFS by multiple other servers, without the need for file locking.

Neither Tokyo Cabinet, not Kyoto Cabinet were able to do as good a job with this task as QDBM, and find it rather sad that QDBM is basically abandonware.

As a side note, for anyone who still has any interest in QDBM, the latest stable (working) version was qdbm-1.8.74. All versions after that, up to and including version 1.8.78 are buggy, and lead to file corruption.

EmmanuelOga commented 9 months ago

+1, I would like to know more about this too!

https://github.com/estraier/tkrzw-rpc also looks pretty unique, and I wonder if it is being used for any production systems.

amirouche commented 9 months ago

@derekbsnider Thanks for noticing QDBM. I am maintaining a list of similar libraries, I rent https://okvs.dev, and I was missing QDBM. Do you configure QDBM specifically for use over NFS? How many writers they are at a given time? It may be relying on a file per-transaction, and do compaction from time to time, somewhat, somehow like SQLite's LSM extensions. By the way, SQLite's LSM extensions can be built as shared library .so and be used embedded in most language that can bind to C / C++.

Note: AFAICT database files over NFS unix networked filesystem are a very nice, but forgotten art. The current approach is to spin a service with a dedicated API (aka. stored procedures), or a generic API (6NF, tuple store, pattern matching, datalog, or a small scheme).

marioroy commented 9 months ago

I gave Tkrzw a spin at Perl Monks.

  1. Exploring and learning Tkrzw. https://perlmonks.org/?node_id=11153424
  2. Sharding, managed by the C++ library. https://perlmonks.org/?node_id=11153432
  3. Sharding, managed by the application (many HashDBMs). https://perlmonks.org/?node_id=11153433
derekbsnider commented 9 months ago

@derekbsnider Thanks for noticing QDBM. I am maintaining a list of similar libraries, I rent https://okvs.dev, and I was missing QDBM. Do you configure QDBM specifically for use over NFS? How many writers they are at a given time? It may be relying on a file per-transaction, and do compaction from time to time, somewhat, somehow like SQLite's LSM extensions. By the way, SQLite's LSM extensions can be built as shared library .so and be used embedded in most language that can bind to C / C++.

The only options I build it with are: --enable-pthread --enable-zlib --disable-debug (Remember, the version to use is 1.8.74).

I don't keep the files open. The server cluster daemons that handle the writes and updates manages millions of .qdb files, and opens writes and closes them for each transaction (not the most efficient, but guarantees that the NFS readers get the latest view). Reads over NFS are successful and will either show the state pre or post transaction without any need for locking. QDBM supports local locking, but I do not use the locking feature. I open without locking options.

EmmanuelOga commented 8 months ago

For those looking for something similar, the closest actively developed project imho is https://github.com/tikv/tikv. https://github.com/apple/foundationdb is also there but it is way harder to use, comparatively.