timshannon / badgerhold

BadgerHold is an embeddable NoSQL store for querying Go types built on Badger
MIT License
515 stars 52 forks source link

Is the Badger stream framework supported? #45

Closed davedbase closed 3 years ago

davedbase commented 3 years ago

I was wondering if Bagderhold can support stream (https://dgraph.io/docs/badger/get-started/#stream) if not is it a major enhancement or minor and easy to adapt?

timshannon commented 3 years ago

I'm not sure how it would apply to Badgerhold which is a querying layer on top of Badger. You can easily use this stream functionality with Badgerhold by simply accessing the underlying Badger database: https://godoc.org/github.com/timshannon/badgerhold#Store.Badger

store, err := badgerhold.Open(opt)
if err != nil {
    t.Fatalf("Error opening %s: %s", opt.Dir, err)
}

stream := store.Badger().NewStream()
// db.NewStreamAt(readTs) for managed mode.

// -- Optional settings
stream.NumGo = 16                     // Set number of goroutines to use for iteration.
stream.Prefix = []byte("some-prefix") // Leave nil for iteration over the whole DB.
stream.LogPrefix = "Badger.Streaming" // For identifying stream logs. Outputs to Logger.

// ChooseKey is called concurrently for every key. If left nil, assumes true by default.
stream.ChooseKey = func(item *badger.Item) bool {
  return bytes.HasSuffix(item.Key(), []byte("er"))
}

All of the index, type, and encoding information that Badgerhold uses is stored in the badger database, so that will all get streamed along with your data.

davedbase commented 3 years ago

Gotcha. Thanks.