Feature Request: Support for batch processing

[PD-Pramila]

Are there any plans to add batch insertion in badgerhold?

[timshannon]

No plans, but I'm open to sane PRs.

You can easily do batch inserts yourself by directly accessing the Badger DB:

db := store.Badger()
wb := db.NewWriteBatch()
defer wb.Cancel()

for i := 0; i < N; i++ {
  err := wb.Set(key(i), value(i), 0) // Will create txns as needed.
  handle(err)
}
handle(wb.Flush()) // Wait for all txns to finish.

[PD-Pramila]

Thanks for the reply. I tried that :). The thing is, we want to use the gob feature and other indexing things provided by Badgerhold, which are done while creating and committing Txn. We want to use mix of batch and txn insertions. So, if I use above code directly then there will be different types of data saved, right?

What I understood so far is that, if I have to use the batch of badger DB, then I have to write wrapper for gob and indexing in my code, same as badgerhold DB.

[timshannon]

Ah, in that case, if you want to make use of the indexing, then yeah, you'd either need to duplicate what I'm doing in BadgerHold.

[timshannon]

Looking further at how WriteBatch is implemented, I'm not sure of a good way to allow this. WriteBatches don't support iterators, and iterators are needed to lookup indexes, so the only way I'd be able to support indexing and WriteBatches is to manage the indexes outside of the WriteBatch transaction, which would mean dirty reads / possibly corrupted data.