Performance benchmarks and comparison to existing DBs

[dumblob]

I wonder whether you made some performance benchmarks to compare 0-db to existing DBs. I couldn't find any.

Could you do some very rough benchmarking, comparing the results e.g. to SQLite (yes, it's apples to oranges, but SQLite is a very good baseline to understand the advantages of 0-db) and publish the results here in this repo?

[maxux]

There is no official benchmarks. I wrote this tool https://github.com/maxux/redis-benchmark to benchmark zdb and compare it to others redis-like. I don't see how you could compare it to sqlite, like you said, you want to compare orange and apple. It would make more sense to compare it to lmdb, rocksdb or that kind of database. But even, they don't support « sequential mode » we support and which is one of the best advantage of zdb, concurrence doesn't have that at all.

If you want to compare the same kind-of-workload (key-value store), doing sql insertion to sqlite will be a lot slower by design anyway.

[dumblob]

If you want to compare the same kind-of-workload (key-value store), doing sql insertion to sqlite will be a lot slower by design anyway.

Let the numbers speak. Could you publish your measurements at least for the Redis vs 0-db comparison?

Btw. I'm afraid key-value insertion into SQLite could be (much) faster than 0-db.

So despite this comparison being apples to oranges, I'd like to know how it performs in an app running locally. I.e. answering the question "should I use local 0-db when deploying on ThreeFold appliances or rather use SQLite". This is a rather important decision IMHO.

[maxux]

I'll write some test to compare with SQLite and will publish result I get. But I can already tell you on a MacBook Air M1 (which have nvme storage), zdb can handle 5 GB/s write burst (it depends on payload size obviously).

SQLite with an index on the key will dramatically slow down with the amount of key increasing. But again it's not really easy to compare since in user-key mode, 0-db keep an index in memory, SQLite use on disk index. Speed is not the only numbers to take in account.

[dumblob]

I'm usually using

create table if not exists memo (
  rid integer primary key autoincrement not null,  -- not superfluous as SQLite always has this column
  key blob not null,
  value blob not null,
  unique ( key ) );

for key-value approximation in SQLite. The unique clause instructs SQLite to create index. You can also test it without unique to see what happens without index.

Btw. don't forget to set some hundreds of MBytes as cache (pragma cache_size = -300 000; - yes, it's a negative number in KBytes).

[maxux]

I finally had time to implement the sqlite benchmark side (https://github.com/maxux/redis-benchmark/tree/master/compare) and I admit sqlite is quite fast. The current workflow compare zdb in network mode (client/server) and the network stack is obviously a bottleneck. I'll write a new test using zdb as library like sqlite to get a better overview of the real performance.

Thus, when inserting 524,288 keys (4 KB payload each time), sqlite reach 236 MB/s (60637 keys/s) and zdb 172 MB/s (44133 keys/s) in single threaded version. When using 4 threads writers for zdb, it reachs 281 MB/s. I didn't tried sqlite in multithreaded version yet.

Reading speed is dramatically faster for sqlite (661 MB/s vs 200 MB/s for zdb, but network stack again is probably the bottleneck). All theses tests are made on my Linux laptop with NVMe underneath.

I'll come back with a better rewrite of the benchmark to get rid of the network stack to get more real values.

[dumblob]

Nice, thanks! I think the numbers are a good start for zdb actually. Will stay tuned to see how it goes without the network stack.

Btw. you might get some inspiration (or even use 1:1 the code) from the zero-copy capability of nng network library (I have only the best experience with this gem).