After enabled badger, the vlog file up to 700M one day and 4GB one week

[lucasjinreal]

This is too big for my tiny server.

I want keep the lefted message to restore after shutdown server, but this size if far out of my expectations.

Why it is so big? And how to reduce it? (it might about badger but stores what messages left to users who use it, so post the issue here)

[mochi-co]

Hi @lucasjinreal - did you check https://dgraph.io/docs/badger/get-started/#memory-usage ? You should be able to apply these configurations in the Badger hook options. Maybe something like this:

// make sure to import the extra badger packages:
import badgerdb "github.com/dgraph-io/badger"
import "github.com/timshannon/badgerhold"

// some complex nesting of different structs called 'Options'
err := server.AddHook(new(badger.Hook), &badger.Options{
  Path: badgerPath,
  Options: &badgerhold.Options{
    Options: badgerdb.Options{
        ValueLogFileSize: 256000,
    },
  },
})

This will give access to all the options in https://github.com/dgraph-io/badger/blob/main/options.go

Let me know if it works, if so we can update the example file.

[lucasjinreal]

@mochi-co thanks for reply, should I update mochi lib?

[mochi-co]

@lucasjinreal Try updating your main.go with your badger hook and see if it fixes the issue :)

[lucasjinreal]

I got no such field in Options.

also, why does vlog file has relation to the memory usage?

[mochi-co]

You might be missing an import - see this example:

// SPDX-License-Identifier: MIT
// SPDX-FileCopyrightText: 2022 mochi-mqtt, mochi-co
// SPDX-FileContributor: mochi-co

package main

import (
    "log"
    "os"
    "os/signal"
    "syscall"

    mqtt "github.com/mochi-mqtt/server/v2"
    "github.com/mochi-mqtt/server/v2/hooks/auth"
    "github.com/mochi-mqtt/server/v2/hooks/storage/badger"
    "github.com/mochi-mqtt/server/v2/listeners"

    badgerdb "github.com/dgraph-io/badger"
    "github.com/timshannon/badgerhold"
)

func main() {
    badgerPath := ".badger"
    defer os.RemoveAll(badgerPath) // remove the example badger files at the end

    sigs := make(chan os.Signal, 1)
    done := make(chan bool, 1)
    signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
    go func() {
        <-sigs
        done <- true
    }()

    server := mqtt.New(nil)
    _ = server.AddHook(new(auth.AllowHook), nil)

    err := server.AddHook(new(badger.Hook), &badger.Options{
        Path: badgerPath,
        Options: &badgerhold.Options{
            Options: badgerdb.Options{
                ValueLogFileSize: 256000,
            },
        },
    })
    if err != nil {
        log.Fatal(err)
    }

    tcp := listeners.NewTCP("t1", ":1883", nil)
    err = server.AddListener(tcp)
    if err != nil {
        log.Fatal(err)
    }

    go func() {
        err := server.Serve()
        if err != nil {
            log.Fatal(err)
        }
    }()

    <-done
    server.Log.Warn("caught signal, stopping...")
    _ = server.Close()
    server.Log.Info("main.go finished")
}

As for vlog memory - I have no idea I'm afraid, I've never used badger in production :) Someone else here may have better ideas

[werbenhu]

@lucasjinreal Take a look at the configuration modifications and the addition of garbage collection described in #370 to see if they would be helpful for your use of BadgerDB.

[lucasjinreal]

@werbenhu Hi, for a user, I just want use it out-of-box.

I didn't do any further config either have a very large throughoutput on my server, the size should be small default.

But looked at the thread you linked, it's likely a existed issue and unable to fix.

So, if I want restore message simply (do't let me dive into so complicated cache machnsim) would that only be redis?

[werbenhu]

@lucasjinreal You can now use Pebble and Badger/v4 as persistent databases. Can this issue be closed now?

[lucasjinreal]

Yes