search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
4.99k stars 534 forks source link

help, I tried to use rocksdb but failed #236

Closed shcw closed 2 years ago

shcw commented 2 years ago

Note: for reported bugs, please fill in the following details. bug reports without detailed steps on how to reproduce will be automatically closed.

Dragonboat version

v3.3.5

Expected behavior

success exec

Actual behavior

I tried to use rocksdb but failed

panic: failed to create NodeHost: mkdir : no such file or directory

[root@localhost dragonboat-example] CGO_LDFLAGS="-L/usr/local/lib -lrocksdb" CGO_CFLAGS="-I/usr/local/include"  go run ./helloworld/...

node address: localhost:63001
2022-03-29 18:50:10.813376 I | dragonboat: go version: go1.17.8, linux/amd64
2022-03-29 18:50:10.813425 I | dragonboat: dragonboat version: 3.3.1 (Rel)
2022-03-29 18:50:10.813481 I | config: using default EngineConfig
2022-03-29 18:50:10.813522 I | config: using default LogDBConfig
2022-03-29 18:50:10.813576 I | dragonboat: DeploymentID set to 1
2022-03-29 18:50:10.813843 C | dragonboat: failed to create NodeHost: mkdir : no such file or directory
panic: mkdir : no such file or directory [recovered]
    panic: failed to create NodeHost: mkdir : no such file or directory

goroutine 1 [running]:
github.com/lni/goutils/logutil/capnslog.(*PackageLogger).Panicf(0x2aaad3dd0438, {0xbf6947, 0xc000208048}, {0xc000204570, 0xc000200300, 0xc0000f5798})
    /data/go/pkg/mod/github.com/lni/goutils@v1.3.0/logutil/capnslog/pkg_logger.go:88 +0xbb
github.com/lni/dragonboat/v3/logger.(*capnsLog).Panicf(0xc000200300, {0xbf6947, 0x41aee7}, {0xc000204570, 0xaf0bc0, 0x1})
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/logger/capnslogger.go:74 +0x26
github.com/lni/dragonboat/v3/logger.(*dragonboatLogger).Panicf(0xc000242000, {0xbf6947, 0x1d}, {0xc000204570, 0x1, 0x1})
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/logger/logger.go:132 +0x57
github.com/lni/dragonboat/v3.NewNodeHost.func2()
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/nodehost.go:345 +0xa9
panic({0xb34840, 0xc0002006c0})
    /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/lni/dragonboat/v3/internal/logdb/kv/rocksdb.openRocksDB({0x10, 0x10, 0x2, 0x2, 0x0, 0x8000000, 0x4, 0x8, 0x11, 0x18, ...}, ...)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/internal/logdb/kv/rocksdb/kv_rocksdb.go:184 +0x33d
github.com/lni/dragonboat/v3/internal/logdb/kv/rocksdb.NewKVStore({0x10, 0x10, 0x2, 0x2, 0x0, 0x8000000, 0x4, 0x8, 0x11, 0x18, ...}, ...)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/internal/logdb/kv/rocksdb/kv_rocksdb.go:48 +0x110
github.com/lni/dragonboat/v3/internal/logdb.openRDB({0x10, 0x10, 0x2, 0x2, 0x0, 0x8000000, 0x4, 0x8, 0x11, 0x18, ...}, ...)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/internal/logdb/db.go:79 +0x8a
github.com/lni/dragonboat/v3/internal/logdb.OpenShardedDB({0x0, {0x0, 0x0}, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0, {0x0, ...}, ...}, ...)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/internal/logdb/sharded.go:79 +0x519
github.com/lni/dragonboat/v3/internal/logdb.NewLogDB({0x0, {0x0, 0x0}, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0, {0x0, ...}, ...}, ...)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/internal/logdb/logdb.go:142 +0x17c
github.com/lni/dragonboat/v3/plugin/rocksdb.(*Factory).Create(...)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/plugin/rocksdb/rocksdb.go:35
github.com/lni/dragonboat/v3/config.(*defaultLogDB).Name(0xc000208058)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/config/config.go:604 +0x230
github.com/lni/dragonboat/v3.(*NodeHost).createLogDB(0xc000242000)
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/nodehost.go:1709 +0x136
github.com/lni/dragonboat/v3.NewNodeHost({0x0, {0xc00021c000, 0x22}, {0xc00021c000, 0x22}, 0xc8, {0xbec31a, 0xf}, 0x0, {0x0, ...}, ...})
    /data/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.1/nodehost.go:350 +0x854
main.main()
    /data/package/dragonboat-example/helloworld/main.go:237 +0x49f
exit status 2

Steps to reproduce the behavior

  1. pull code 
    git clone https://github.com/lni/dragonboat-example.git
  2. Modify the file dragonboat-example/helloworld/main.go 
    nhc := config.NodeHostConfig{
WALDir: datadir,
NodeHostDir: datadir,
RTTMillisecond: 200,
RaftAddress: nodeAddr,
LogDBFactory:   (&rocksdb.Factory{}).Create,
}
  3. excuting an order 
    CGO_LDFLAGS="-L/usr/local/lib -lrocksdb" CGO_CFLAGS="-I/usr/local/include"  go run ./helloworld/...

PS: Extended Information

[root@localhost dragonboat-example]# ll /usr/local/lib
lrwxrwxrwx 1 root root        19 Mar 29 17:35 librocksdb.so -> librocksdb.so.6.4.6
lrwxrwxrwx 1 root root        19 Mar 29 17:35 librocksdb.so.6 -> librocksdb.so.6.4.6
lrwxrwxrwx 1 root root        19 Mar 29 17:35 librocksdb.so.6.4 -> librocksdb.so.6.4.6
-rwxr-xr-x 1 root root 145008080 Mar 29 17:35 librocksdb.so.6.4.6

[root@localhost dragonboat-example]# ll /usr/local/include/rocksdb/
-rw-r--r-- 1 root root 30128 Mar 29 17:35 advanced_options.h
-rw-r--r-- 1 root root 11035 Mar 29 17:35 cache.h
-rw-r--r-- 1 root root 85052 Mar 29 17:35 c.h
-rw-r--r-- 1 root root  2588 Mar 29 17:35 cleanable.h
-rw-r--r-- 1 root root  8623 Mar 29 17:35 compaction_filter.h
-rw-r--r-- 1 root root  3204 Mar 29 17:35 compaction_job_stats.h
-rw-r--r-- 1 root root  4436 Mar 29 17:35 comparator.h
-rw-r--r-- 1 root root  1599 Mar 29 17:35 concurrent_task_limiter.h
-rw-r--r-- 1 root root 16351 Mar 29 17:35 convenience.h
-rw-r--r-- 1 root root   361 Mar 29 17:35 db_bench_tool.h
-rw-r--r-- 1 root root  1192 Mar 29 17:35 db_dump_tool.h
-rw-r--r-- 1 root root 66999 Mar 29 17:35 db.h
-rw-r--r-- 1 root root  7625 Mar 29 17:35 env_encryption.h
-rw-r--r-- 1 root root 61586 Mar 29 17:35 env.h
-rw-r--r-- 1 root root  1149 Mar 29 17:35 experimental.h
-rw-r--r-- 1 root root  6089 Mar 29 17:35 filter_policy.h
-rw-r--r-- 1 root root  1947 Mar 29 17:35 flush_block_policy.h
-rw-r--r-- 1 root root  1616 Mar 29 17:35 iostats_context.h
-rw-r--r-- 1 root root  4918 Mar 29 17:35 iterator.h
-rw-r--r-- 1 root root  1229 Mar 29 17:35 ldb_tool.h
-rw-r--r-- 1 root root 17891 Mar 29 17:35 listener.h
-rw-r--r-- 1 root root  3183 Mar 29 17:35 memory_allocator.h
-rw-r--r-- 1 root root 14990 Mar 29 17:35 memtablerep.h
-rw-r--r-- 1 root root 12091 Mar 29 17:35 merge_operator.h
-rw-r--r-- 1 root root  4011 Mar 29 17:35 metadata.h
-rw-r--r-- 1 root root 65892 Mar 29 17:35 options.h
-rw-r--r-- 1 root root  9897 Mar 29 17:35 perf_context.h
-rw-r--r-- 1 root root  1229 Mar 29 17:35 perf_level.h
-rw-r--r-- 1 root root  2314 Mar 29 17:35 persistent_cache.h
-rw-r--r-- 1 root root  5749 Mar 29 17:35 rate_limiter.h
-rw-r--r-- 1 root root  7460 Mar 29 17:35 slice.h
-rw-r--r-- 1 root root  4449 Mar 29 17:35 slice_transform.h
-rw-r--r-- 1 root root  1268 Mar 29 17:35 snapshot.h
-rw-r--r-- 1 root root   492 Mar 29 17:35 sst_dump_tool.h
-rw-r--r-- 1 root root  5068 Mar 29 17:35 sst_file_manager.h
-rw-r--r-- 1 root root  1245 Mar 29 17:35 sst_file_reader.h
-rw-r--r-- 1 root root  5400 Mar 29 17:35 sst_file_writer.h
-rw-r--r-- 1 root root 19483 Mar 29 17:35 statistics.h
-rw-r--r-- 1 root root  2534 Mar 29 17:35 stats_history.h
-rw-r--r-- 1 root root 12625 Mar 29 17:35 status.h
-rw-r--r-- 1 root root 27740 Mar 29 17:35 table.h
-rw-r--r-- 1 root root 10116 Mar 29 17:35 table_properties.h
-rw-r--r-- 1 root root  2005 Mar 29 17:35 threadpool.h
-rw-r--r-- 1 root root  6402 Mar 29 17:35 thread_status.h
-rw-r--r-- 1 root root  1713 Mar 29 17:35 trace_reader_writer.h
-rw-r--r-- 1 root root  3791 Mar 29 17:35 transaction_log.h
-rw-r--r-- 1 root root  1448 Mar 29 17:35 types.h
-rw-r--r-- 1 root root  3384 Mar 29 17:35 universal_compaction.h
drwxr-xr-x 3 root root  4096 Mar 29 17:35 utilities
-rw-r--r-- 1 root root   641 Mar 29 17:35 version.h
-rw-r--r-- 1 root root  4395 Mar 29 17:35 wal_filter.h
-rw-r--r-- 1 root root  5409 Mar 29 17:35 write_batch_base.h
-rw-r--r-- 1 root root 13854 Mar 29 17:35 write_batch.h
-rw-r--r-- 1 root root  3362 Mar 29 17:35 write_buffer_manager.h
xkeyideal commented 2 years ago

Give up rocksdb, use pebbledb.

Dragonboat use pebbledb as raft log store, so you no need use two different kv db in your project.

shcw commented 2 years ago

Give up rocksdb, use pebbledb.

Dragonboat use pebbledb as raft log store, so you no need use two different kv db in your project.

@xkeyideal because I took over an old code and upgraded it now, I want to reuse the original kv db

lni commented 2 years ago

Please use Pebble, RocksDB is no longer supported.

If you have a dragonboat based system that already has some RocksDB data in it, Pebble is fully compatible with that.

kevburnsjr commented 2 years ago

Please use Pebble, RocksDB is no longer supported.

If you have a dragonboat based system that already has some RocksDB data in it, Pebble is fully compatible with that.

The repository still contains dozens of references to RocksDB. https://github.com/lni/dragonboat/search?q=rocksdb

@lni If I put together a PR to remove all dangling references to RocksDB, is that something you might consider merging?

If not all the code then at least the docs? https://github.com/lni/dragonboat/blob/master/docs/storage.md#rocksdb

Switch to RocksDB only involves one extra line of code, just set the LogDBFactory field of your config.NodeHostConfig to rocksdb.Factory available in the github.com/lni/dragonboat/v3/plugin/rocksdb.

shcw commented 2 years ago

Please use Pebble, RocksDB is no longer supported.

If you have a dragonboat based system that already has some RocksDB data in it, Pebble is fully compatible with that.

@lni Thank you very much :)

lni commented 2 years ago

@lni If I put together a PR to remove all dangling references to RocksDB, is that something you might consider merging?

@kevburnsjr thanks for looking into this, yes, please send in the PR and I will be happy to merge it.

Please feel free to let me know if you need anything else from me.