压测一下rocksdb folly集成对multiget的收益

[wanghenshui]

https://github.com/facebook/rocksdb/wiki/RocksDB-Contribution-Guide#folly-integration

http://rocksdb.org/blog/2022/10/07/asynchronous-io-in-rocksdb.html

[wanghenshui]

TEST_TMPDIR=/data/tmp/folly2 ./db_bench --benchmarks=fillrandom,multireadrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 Set seed to 1675609423217093 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Sun Feb 5 23:03:43 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 100 bytes each (50 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 11062.6 MB (estimated) FileSize: 6294.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled DB path: [/data/tmp/folly2/dbbench] fillrandom : 1.610 micros/op 621123 ops/sec 160.999 seconds 100000000 operations; 68.7 MB/s entries_per_batch = 1 DB path: [/data/tmp/folly2/dbbench] ... finished 89500000 ops
... finished 89600000 ops

multireadrandom : 12.162 micros/op 82222 ops/sec 1216.209 seconds 100000000 operations; 5.7 MB/s (63213817 of 100000000 found)

[wanghenshui]

TEST_TMPDIR=/data/tmp/folly2 ./db_bench --benchmarks=fillrandom,multireadrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 Set seed to 1675611744336339 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Sun Feb 5 23:42:24 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 100 bytes each (50 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 11062.6 MB (estimated) FileSize: 6294.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled DB path: [/data/tmp/folly2/dbbench] ... finished 81100000 ops
fillrandom : 1.457 micros/op 686420 ops/sec 145.683 seconds 100000000 operations; 75.9 MB/s entries_per_batch = 1 DB path: [/data/tmp/folly2/dbbench] ... finished 9200000 ops
... finished 22500000 ops
multireadrandom : 11.795 micros/op 84782 ops/sec 1179.493 seconds 100000000 operations; 5.9 MB/s (63214562 of 100000000 found)

[wanghenshui]

-multiread_batched (Use the new MultiGet API) type: bool default: false
-multiread_stride (Stride length for the keys in a MultiGet batch)
-num_multi_db (Number of DBs used in the benchmark. 0 means single DB.)
-optimize_multiget_for_io (When set true, RocksDB does asynchronous reads
  for SST files in multiple levels for MultiGet.) type: bool default: true
    --num_multi_db

他马的，测错了。

[wanghenshui]

TEST_TMPDIR=/data/tmp/folly4 ./db_bench --benchmarks=fillrandom,readrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 -async_io=true Set seed to 1676038034574712 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Fri Feb 10 22:07:14 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 100 bytes each (50 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 11062.6 MB (estimated) FileSize: 6294.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags

Integrated BlobDB: blob cache disabled DB path: [/data/tmp/folly4/dbbench] fillrandom : 1.456 micros/op 687009 ops/sec 145.558 seconds 100000000 operations; 76.0 MB/s DB path: [/data/tmp/folly4/dbbench] ... finished 5700000 ops

... finished 5800000 ops
readrandom : 4.994 micros/op 200240 ops/sec 499.400 seconds 100000000 operations; 14.0 MB/s (63223972 of 100000000 found)

[wanghenshui]

w@w-msi:~/code/rocksdb/build$ grep "folly::" /data/tmp/folly6/dbbench/LOG 2023/02/10-22:39:15.859545 149684 DMutex implementation: folly::DistributedMutex

[wanghenshui]

TEST_TMPDIR=/data/tmp/folly9 ./db_bench --benchmarks=fillrandom,multireadrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 Set seed to 1676093396813298 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Sat Feb 11 13:29:56 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 100 bytes each (50 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 11062.6 MB (estimated) FileSize: 6294.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled DB path: [/data/tmp/folly9/dbbench] fillrandom : 1.439 micros/op 695016 ops/sec 143.882 seconds 100000000 operations; 76.9 MB/s entries_per_batch = 1 DB path: [/data/tmp/folly9/dbbench] multireadrandom : 7.097 micros/op 140910 ops/sec 709.672 seconds 100000000 operations; 9.9 MB/s (63206634 of 100000000 found)

TEST_TMPDIR=/data/tmp/folly9 ./db_bench --benchmarks=fillrandom,multireadrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 -async_io=true Set seed to 1676092298052831 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Sat Feb 11 13:11:38 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 100 bytes each (50 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 11062.6 MB (estimated) FileSize: 6294.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled DB path: [/data/tmp/folly9/dbbench] fillrandom : 1.512 micros/op 661526 ops/sec 151.165 seconds 100000000 operations; 73.2 MB/s entries_per_batch = 1 DB path: [/data/tmp/folly9/dbbench] ... finished 40000000 ops

multireadrandom : 9.084 micros/op 110078 ops/sec 908.445 seconds 100000000 operations; 7.7 MB/s (63214706 of 100000000 found)

小key，开了反倒性能下降

[wanghenshui]

TEST_TMPDIR=/data/tmp/folly9 ./db_bench --benchmarks=fillrandom,multireadrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 -value_size=1000 Set seed to 1676095181678966 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Sat Feb 11 13:59:41 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (500 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 49209.6 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled DB path: [/data/tmp/folly9/dbbench] fillrandom : 7.725 micros/op 129447 ops/sec 772.516 seconds 100000000 operations; 125.4 MB/s entries_per_batch = 1 DB path: [/data/tmp/folly9/dbbench] ... finished 74500000 ops

... finished 74600000 ops
multireadrandom : 46.229 micros/op 21631 ops/sec 4622.913 seconds 100000000 operations; 13.2 MB/s (63209916 of 100000000 found)

[wanghenshui]

TEST_TMPDIR=/data/tmp/folly9 ./db_bench --benchmarks=multireadrandom -max_background_jobs=12 -num=100000000 -target_file_size_base=33554432 -value_size=1000 -async_io=true -use_existing_db=true Set seed to 1676100871673090 because --seed was 0 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled RocksDB: version 7.9.0 Date: Sat Feb 11 15:34:32 2023 CPU: 32 * AMD Ryzen 9 7950X 16-Core Processor CPUCache: 1024 KB Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (500 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 49209.6 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1

entries_per_batch = 1 DB path: [/data/tmp/folly9/dbbench] multireadrandom : 44.799 micros/op 22321 ops/sec 4479.911 seconds 100000000 operations; 13.7 MB/s (63210391 of 100000000 found)

[wanghenshui]

[wanghenshui]

w@w-msi:~$ fio --randrepeat=1 --ioengine=sync --direct=1 --gtod_reduce=1 --name=test --filename=/data/tmp/fio_test_file --bs=4k --iodepth=64 --size=4G --readwrite=randread --numjobs=32 --group_reporting test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=64 ... fio-3.30 Starting 32 processes Jobs: 32 (f=32): [r(32)][100.0%][r=962MiB/s][r=246k IOPS][eta 00m:00s] test: (groupid=0, jobs=32): err= 0: pid=11567: Sat Feb 11 19:17:06 2023 read: IOPS=244k, BW=953MiB/s (999MB/s)(128GiB/137532msec) bw ( KiB/s): min=22408, max=1006104, per=100.00%, avg=976605.28, stdev=2430.25, samples=8768 iops : min= 5602, max=251526, avg=244151.32, stdev=607.56, samples=8768 cpu : usr=0.26%, sys=1.54%, ctx=33554635, majf=5, minf=346 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=33554432,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs): READ: bw=953MiB/s (999MB/s), 953MiB/s-953MiB/s (999MB/s-999MB/s), io=128GiB (137GB), run=137532-137532msec

Disk stats (read/write): nvme1n1: ios=33537154/0, merge=0/0, ticks=4317401/0, in_queue=4317401, util=99.94%

[wanghenshui]

w@w-msi:~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/data/tmp/fio_test_file2 --bs=4k --iodepth=64 --size=4G --readwrite=randread test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.30 Starting 1 process

Jobs: 1 (f=1): [r(1)][100.0%][r=971MiB/s][r=248k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=11667: Sat Feb 11 19:19:47 2023 read: IOPS=247k, BW=965MiB/s (1012MB/s)(4096MiB/4246msec) bw ( KiB/s): min=981088, max=996264, per=100.00%, avg=990337.00, stdev=5501.80, samples=8 iops : min=245272, max=249066, avg=247584.25, stdev=1375.45, samples=8 cpu : usr=12.82%, sys=43.18%, ctx=338888, majf=0, minf=71 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs): READ: bw=965MiB/s (1012MB/s), 965MiB/s-965MiB/s (1012MB/s-1012MB/s), io=4096MiB (4295MB), run=4246-4246msec

Disk stats (read/write): nvme1n1: ios=1024690/0, merge=0/0, ticks=260540/0, in_queue=260539, util=97.69%

[wanghenshui]

dd if=/data/tmp/test of=/data/tmp/test2 bs=64k 记录了1250000+0 的读入 记录了1250000+0 的写出 81920000000字节（82 GB，76 GiB）已复制，112.668 s，727 MB/s

[wanghenshui]

[wanghenshui]

NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh bulkload,readrandom

s_sec   mb_sec  lsm_sz  blob_sz c_wgb   w_amp   c_mbps  c_wsecs c_csecs b_rgb   b_wgb   usec_op p50 p99 p99.9   p99.99  pmax    uptime  stall%  Nstall  u_cpu   s_cpu   rss test    date    version job_id  githash
1536598 615.5   18GB    0GB 17.6    0.9 276.0   203 199 0   0   0.7 0.5 1   1   1139    41244   65  43.2    64  0.2 0.0 NA  bulkload    2023-02-11T21:58:57 8.0.0       a72f591825
1480156 374.8   11GB    0GB 0.0 NA  0.0 0   0   0   0   21.6    2.6 327 696 1563    414661  2162    0.0 0   12.4    2.6 16.4    readrandom.t32  2023-02-11T22:02:17 8.0.0       a72f591825

[wanghenshui]

1546717 619.5   177GB   0GB 177.2   0.9 280.7   2028    1964    0   0   0.6 0.5 1   1   1151    35774   646 46.8    651 2.2 0.1 NA  bulkload    2023-02-11T22:55:03 8.0.0       a72f591825
867544  219.7   107GB   0GB 0.0 NA  0.0 0   0   0   0   36.9    2.6 408 960 3881    333915  3689    0.0 0   12.5    4.7 17.6    readrandom.t32  2023-02-12T14:15:06 8.0.0       a72f591825

[wanghenshui]

1334 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh multireadrandom 1335 cat /tmp/report.tsv 1336 vim /tmp/report.tsv 1337 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh multireadrandom 1338 ASYNC_IO=1 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh multireadrandom 1339 ASYNC_IO=1 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh readrandom 1340 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh readrandom 1341 vim /tmp/report.tsv 1342 ls 1343 cp -r ../folly_build_tmp/ /tmp/fbcode_builder_getdeps-ZhomeZwZcodeZrocksdbZthird-partyZfollyZbuildZfbcode_builder 1344 cp -r ../../folly_build_tmp/ /tmp/fbcode_builder_getdeps-ZhomeZwZcodeZrocksdbZthird-partyZfollyZbuildZfbcode_builder 1345 ./db_bench_folly 1346 cp benchmark.sh benchmark_folly.sh 1347 vim benchmark_folly.sh 1348 mv db_bench db_bench_no_folly 1349 cp db_bench_folly db_bench 1350 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh readrandom 1351 ASYNC_IO=1 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh readrandom 1352 ASYNC_IO=1 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh multireadrandom 1353 history 1354 NUM_THREADS=32 NUM_KEYS=100000000 DB_DIR=/data/tmp/ben WAL_DIR=/data/tmp/wal ./benchmark.sh multireadrandom 1355 history