Add BadgerDB driver - Githubissues

k3s-io / kine

Run Kubernetes on MySQL, Postgres, sqlite, dqlite, not etcd.

Apache License 2.0

1.56k stars 233 forks source link

Add BadgerDB driver #16

Open narqo opened 4 years ago

narqo commented 4 years ago

BadgerDB is an embeddable, fast, pure-Go key-value database. At least from a brief look, it feels that it might fit well as an alternative to sqlite:

it's pure Go, meaning it doesn't (necessarily) require cgo, which looks attractive in cases similar to described in #14
it supports concurrent ACID transactions (I haven't dig deep into the implementation of logstructured.Log, so can't say if that's important for a kine's driver).

See "Design" secsion of BadgerDB's README.

Note, BadgerDB v2 uses optional zstd compression, which currently requires cgo. Refer to https://github.com/dgraph-io/badger/issues/1162 on switching to pure Go zstd.

Nevertheless, I don't think zstd level of compression is required for kine/k3s cases.

ibuildthecloud commented 4 years ago

This sounds interesting. Some things to consider first. Kine exposes an etcd style abstracted on (right now) SQL backends. Even though etcd is a key value store, it's actually hard to reproduce the behavior of etcd on an arbitrary k/v (at least I haven't figured it out yet). The reason is that Kubernetes is very tied to the way etcd works with revision counters. There is a basic assumption that a historical record of changes is available and that one can query them for a short period of time. So internally what you want out of a data store is to be able to store a log of changes and then to be able to query that log easily. So in SQL it's quite easy to do. I haven't thought of a simple approach in a k/v store yet but also haven't given a lot of thought to it. Obviously it's possible, just might be a bit more work than one would expect.

Also, this code base is very poorly documented, sorry.

narqo commented 4 years ago

I've spent some time reading about etcd's data-model; indeed it's a bit more complex than I expected after looking at sqlite's driver but still doable. Will try to sketch up a PR.

carlosedp commented 4 years ago

What about using bbolt directly (https://github.com/etcd-io/bbolt) as an embedded DB? Dunno if it would work... just a thought :)

emolitor commented 3 years ago

Minor note, the latest version of BadgerDB, from v3.2103.1 onward, is now using a pure go zstd dependency.