Getting slow

[betim]

Hi,

When I start fresh with a new db.Dir I get: INFO: Replay took: 67.932µs

But after inserting/deleting 1 million structs even though file stays small, I'm getting INFO: Replay took: 16.604332ms -- and it keeps getting worse.

Snippet:

    sID := ksuid.New().String()
    err := db.Store.DeleteMatching(&Session{},
        badgerhold.Where("Origin").Eq(origin).And("User.Email").Eq(u.Email))

    if err != nil {
        return err
    }

    err = db.Store.Insert(sID, &Session{
        Origin: origin,
        Since:  time.Now(),
        User:   u,
    })

    if err != nil {
        return err
    }

go version go1.13.4 linux/amd64 badger: 8097259 badgerhold: 723db32

Worth noting here is that I have a mutex that locks at the beginning of func call, so It's not concurrent.

Am I doing something wrong here?

Thanks in advance

[timshannon]

Interesting. A simple mutex lock / unlock shouldn't get progressively slower. I'm betting it's something in Badger.

You could try quickly replacing badgerhold with bolthold and trying again.

It should be fairly easy to update your code:

import (
    badgerhold "github.com/timshannon/bolthold"
)

Both APIs should be very, very similar.

[betim]

Yes,

With bolt it's not getting slower, however bolt is very slow out of the box.

I'm not using go modules, so I'm not sure how can I try v2 of badger.

Any ideas where should I look?

Regards

[timshannon]

So if you've already confirmed that it's an issue in Badger, then you might have to wait until I get around to issue #21

You could also try simplifying your code and use Badger 2 directly. If you are just adding and deleting records, to confirm that Badger 2 fixes your issue.

[betim]

Houston we have a problem: it's getting slower with v2 also.

With a new GOHOME/GOPATH I downloaded everything fresh. Example from badger where they say to go get github.com/dgraph-io/badger/v2 didn't work so I downloaded the zip and extracted it where it belongs.

Now this example:

package main

import (
    "time"
    "fmt"
    "github.com/timshannon/badgerhold"
)

type Item struct {
    Name    string
    Created time.Time
}

func main() {
    opt := badgerhold.DefaultOptions
    opt.Dir = "/tmp/testv2-5"
    opt.ValueDir = opt.Dir

    store, err := badgerhold.Open(opt)
    if err != nil {
        panic(err)
    }

    for i := 0; i < 1000; i++ {
        err = store.Insert(i, &Item{
            Name:    fmt.Sprintf("NM-%d", i),
            Created: time.Now(),
        })

        store.DeleteMatching(&Item{},
            badgerhold.Where("Name").Eq(fmt.Sprintf("NM-%d", i)))
    }
}

Is producing this output:

18:11 ~/test/src/test/test/  λ • go run main.go
badger 2020/04/09 18:11:37 INFO: All 0 tables opened in 0s
badger 2020/04/09 18:11:37 INFO: Replaying file id: 0 at offset: 0
badger 2020/04/09 18:11:37 INFO: Replay took: 2.54µs
badger 2020/04/09 18:11:37 DEBUG: Value log discard stats empty
18:11 ~/test/src/test/test/  λ • go run main.go
badger 2020/04/09 18:11:50 INFO: All 0 tables opened in 0s
badger 2020/04/09 18:11:50 INFO: Replaying file id: 0 at offset: 0
badger 2020/04/09 18:11:50 INFO: Replay took: 5.303642ms
badger 2020/04/09 18:11:50 DEBUG: Value log discard stats empty
18:12 ~/test/src/test/test/  λ • go run main.go
badger 2020/04/09 18:12:07 INFO: All 0 tables opened in 0s
badger 2020/04/09 18:12:07 INFO: Replaying file id: 0 at offset: 0
badger 2020/04/09 18:12:07 INFO: Replay took: 7.631842ms
badger 2020/04/09 18:12:07 DEBUG: Value log discard stats empty

Any ideas?

Regards,

[betim]

Apologies. After actually trying v2, I can confirm that it's not getting slower. Same here: panic: Transaction Conflict. Please retry same as the issue #21.

Closing