search

mosuka / blast

Blast is a full text search and indexing server, written in Go, built on top of Bleve.
Apache License 2.0
1.08k stars 76 forks source link

Badger instead of RocksDB #17

Closed Scapal closed 5 years ago

Scapal commented 6 years ago

Have you considered using Badger K/V store instead of RocksDB in order to keep CGO dependencies minimal ?

https://github.com/dgraph-io/badger

ghost commented 6 years ago

BLast uses bleve to do its magic.

I agree that removing cgo is nice and badgers perf and scaling is much easier to get working with raft IMHO.

here is the issue relating to using badger. https://github.com/blevesearch/bleve/issues/591

It looks possible now. It was not possible before.

mosuka commented 6 years ago

Hi @Scapal

I agree with you to keep minimize CGO dependencies. Currently, Blast focuses on server implementation for using Bleve on distributed environment. I will support Badger when Bleve merges Badger's code.

Thanks.

ghost commented 5 years ago

@mosuka The blocker for using Badger seems to have been lifted

see: https://github.com/blevesearch/bleve/issues/591 at the bottom @manishrjain changed Badger to support transactions with multiples readers !

EDIT. It looks like Scorch is the only way forward to use bleve. KV ( BoltDB) is no longer supported. I am not sue yet how you did your Raft implementation yet, but is it dependent on boltdb ?

mosuka commented 5 years ago

@gedw99 Blast has two storages. One is Bleve's index. The other one is KVS for storing raw data before indexing. The latter currently uses BoltDB, but it is not affected by Bleve. In the future I would like to migrate to Badger.

ghost commented 5 years ago

Hay all, there is a new badger bleve repo here: https://github.com/alash3al/bbadger

Its called "bbadger" !!

It would be great if everyone interested in this can help with examples, PR's and general dog fooding. Its also very easy to use.

@mosuka This should address the need for badger that you mentioned.

@Scapal This removes the CGO aspects. I agree with you about the CGO as its a huge slowdown for golang to use CGO

mosuka commented 5 years ago

Hi @Scapal @gedw99 ,

Fixed to support Bagder.

For example, build Badger to be available for index storage as follows:

$ make BUILD_TAGS=badger build

Then, you can start Blast as follows:

$ ./bin/blast-index start --node-id=index1 --data-dir=/tmp/blast/index1 --bind-addr=:6060 --grpc-addr=:5050 --http-addr=:8080 --index-storage-type=badger

Please clone the master branch.

mosuka commented 5 years ago

Sorry, it still have a bug. Please wait.

mosuka commented 5 years ago

Fixed some Bader plugin bugs. But this feature is considered experimental.