evgmik
commented
3 years ago (the comment was updated 2021-02-07 with correct numbers)
Storage efficiency of badger db for different versions.
I modified the script posted https://github.com/dgraph-io/badger/issues/686#issue-399905584
The script generates 100,000 key:value pairs with each key and each value being 10bytes.
package main
import (
"github.com/dgraph-io/badger"
"log"
"fmt"
)
func main() {
opts := badger.DefaultOptions("/tmp/badger")
db, err := badger.Open(opts)
if err != nil {
log.Fatal(err)
}
defer db.Close()
wb := db.NewWriteBatch()
defer wb.Cancel()
for i := 0; i < 100000; i++ {
key := fmt.Sprintf("key_%06d", i)
val := fmt.Sprintf("val_%06d", i)
err := wb.Set([]byte(key), []byte(val)) // Will create txns as needed.
if err != nil {
log.Fatal(err)
}
}
wb.Flush()
fmt.Println("Bulk upload done")
}The script generates 10,000 key:value pairs with each key and each value being 10bytes. So the minimal storage space is 2MB = 2,000,000 bytes.
With badger v1.6.2 1.3s total execution time. I got factor 4 more HDD space allocation.
$ l /tmp/badger
total 8.3M
4.8M 2021-02-06 21:00 000000.vlog
3.5M 2021-02-06 21:00 000002.sst
116 2021-02-06 21:00 MANIFESTWith badger latest version (not sure about this) 0.6s total execution time. I got factor of 0.7 more HDD space allocation.
$ l /tmp/badger
total 1.4M
1.4M 2021-02-06 21:32 000001.sst
20 2021-02-06 21:32 000001.vlog
1.0M 2021-02-06 21:32 DISCARD
28 2021-02-06 21:32 KEYREGISTRY
30 2021-02-06 21:32 MANIFEST
Note that DISCARD occupies only 4kB (I guess it empty but preallocated), this is why ls shows 1MB
So badger DB overhead is reasonable with newer version. In fact it is "negative", i.e. DB takes less space than the raw data. I am guessing this is due to compression which is new feature in badger since v2.0.0. Also new version is about factor of 2 faster (at least with this simple script)
sahib
For the
brigdatabase we are currently usingbadgerv1.6.2 and there are thoughts to move to a more modern verstion 3.2011.1 (at the time of writing). But first we should think isbadgeris the right DB.badgeruse different storage for key and value, which allows to do manipulation with keys very fast.badgeris the fastest according to their benchmark. But this kind of DB benefit the most when keys are short and values are large. This is technically not the case for thebrigstructure, our keys are hashes which are around 50 bytes, and keys are quite short or comparable. On one test instance the raw (absolute minimum without compression) size of the keys is 9151 B and values 16568. So a typical value twice larger than the key. More importantly, the values are small 1-2kB. Sobadgerwhich was designed for big data brings us no benefits. In fact we have huge data write size amplification due tobadgerinternal structure. In above case you would need 9151+16568 = 25719 of raw storage, yetbadger v1.6.2 dbtakes 517kB (all it stores about 100 commits of the same filename with a different content). So we see write amplification of 20 times. The raw size could be even smaller, if we use more compact values encoding. Also we see, that a simplebrig touch newfiletakes 3-4kB of DB storage.Sure, we all have GB of HDD but there is clearly an inefficiency.