Get/Update don't work when opening existing DB

[altafan]

Hello,

I have noticed something strange that occurs when opening an existing DB:

If I use the Get() method to retrieve an element by key from the storage it isn't found, but if I use the Find() one passing a query filtering by key, the same element is now found.

The same happens also for the Update/UpdateMatching methods, the first returns an ErrNotFound while the second correctly updates the element which is indeed stored in the DB.

This doesn't happen at all the very first time when the DB is created instead of opened.

[timshannon]

When you say an existing DB, you mean a non-badgerhold DB created by some other process? I'm not sure how well that would work. Badgerhold prefixes all byte data with a datatype identifier that wouldn't be present in a DB created by any other process.

If you are talking about opening an older version of a badgerhold DB (like from V2 or V3), those will not be compatible, and you'll need to migrate your data using the older major version of the library to read the data, and the new major to insert the data into a new database file.

[altafan]

No it isn't a non-badgerhold DB and it's neither an older version of badgerhold DB.

I'm talking about a DB composed of multiple files, all opened and closed in the same way:

// Create/open the DB
func createDb(dbDir string, logger badger.Logger) (*badgerhold.Store, error) {
    opts := badger.DefaultOptions(dbDir)
    opts.Logger = logger
    opts.ValueLogLoadingMode = options.FileIO
    opts.Compression = options.ZSTD

    db, err := badgerhold.Open(badgerhold.Options{
        Encoder:          badgerhold.DefaultEncode,
        Decoder:          badgerhold.DefaultDecode,
        SequenceBandwith: 100,
        Options:          opts,
    })
    if err != nil {
        return nil, err
    }

    ticker := time.NewTicker(30 * time.Minute)
    go func() {
        for {
            <-ticker.C
            if err := db.Badger().RunValueLogGC(0.5); err != nil && err != badger.ErrNoRewrite {
                log.Error(err)
            }
        }
    }()

    return db, nil
}

// Close the DB
func closeDB(db *badgerhold.Store) error {
       return db.Close()
}

When I start my process and the files are already existing they are just opened normally instead of being created.

Furthermore, this strange behavior happens only for one of these DBs (files). Let's say for example that I have 3 files a.db, b.db and c.db this only happens for c.db but I use extensively Get() and Update() also in a.db and b.db.

I'm currently using badhgerhold/v2 and badger/v2. I tried also to update to badgerhold/v4 and badger/v3 by also migrating the db but the problem still persists.

[timshannon]

Sounds like a bug then. Can you put together a little bit to code to reproduce this? Specifically a sample type where you can replicate the Get not working but the Find on a Key does.

Thanks,

[altafan]

I created a tiny repo that includes the db directory with an exact copy of my "bugged" store, and a script that demonstrates how an element can be retrieved from it with a query but not by using Get.

It's just enough to run go run main.go to execute it.

[timshannon]

@altafan I need some sample code that shows how the data was inserted. This code doesn't do any inserts, it's just an existing DB.

[altafan]

@timshannon I pushed a commit to show how data is written to the store. The interesting thing is that the problem does not affect the new data inserted at all, but only the already existing entries. 🤔

[timshannon]

I can't replicate this issue, and if I grab your repo, and remove the existing database and upgrade to the latest Badger and Badgerhold the issue goes away there as well.

I believe you're possibly running into an older Badger bug, in a previous DB format. I recommend updating to use the latest Badger and Badgerhold versions.