boltdb / bolt

An embedded key/value database for Go.
MIT License
14.24k stars 1.51k forks source link

page already freed on certain builds #749

Open tehsphinx opened 6 years ago

tehsphinx commented 6 years ago

Sorry to bring up another issue with this headline... I already checked the other two issues and do not have the impression that they match my case.

Here my error

panic: page 4 already freed

goroutine 416 [running]: .../vendor/github.com/boltdb/bolt.(freelist).free(0x202f7ec0, 0x5, 0x0, 0x3014000) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/freelist.go:121 +0x263 .../vendor/github.com/boltdb/bolt.(node).spill(0x20356d40, 0x2064bc64, 0x6) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/node.go:363 +0x19f .../vendor/github.com/boltdb/bolt.(Bucket).spill(0x2032eb0c, 0xe, 0x38554104) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/bucket.go:570 +0x13d .../vendor/github.com/boltdb/bolt.(Tx).Commit(0x2032eb00, 0x0, 0x0) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/tx.go:163 +0xff .../vendor/github.com/boltdb/bolt.(*DB).Update(0x20244fc0, 0x2056de80, 0x0, 0x0) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/db.go:605 +0xc9

Here my Code snippet:

// Add adds a value to the db using NextSequence to autoincrement the key
func (s *DB) Add(tableName string, value []byte) (uint64, error) {
    var id uint64
    err := s.Conn.Update(func(tx *bolt.Tx) error {
        bt, err := tx.CreateBucketIfNotExists([]byte(tableName))
        if err != nil {
            return err
        }

        id, err = bt.NextSequence()
        if err != nil {
            return err
        }

        return bt.Put(itob(id), value)
    })
    return id, errors.WithStack(err)
}

// itob returns an 8-byte big endian representation of v.
func itob(i uint64) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, i)
    return b
}

The error occurs the second time this code is executed (every time).

The code runs fine on MacOS (amd64) and Windows (386) without cgo. Disabling cgo disables a few packages that use windows dlls. When compiling the final version with cgo (and dlls) using xgo --targets=windows/386 . the resulting windows executable will show the above issue.

I added some logs to the boltdb code and checked which pages are being freed. On every call the same pages are being freed. The node.pgid on node.go:363 is also the same on every call (calls are several secods apart). Why?

Since in one of the other issues on this topic there is talk about race conditions I made sure the entire connection is used by a single goroutine using the action pattern. This did not resolve the issue either:

// Add adds a value to the db using NextSequence to autoincrement the key
func (s *DB) Add(tableName string, value []byte) (uint64, error) {
    // added this to make sure there is no other goroutine using the byte array at the same time
    var valCopy = make([]byte, 0, len(value))
    valCopy = append(valCopy, value...)

    var id uint64
    chErr := make(chan error)
    s.Run(s.Action)
    s.Action <- func() {
        err := s.Conn.Update(func(tx *bolt.Tx) error {
            bt, err := tx.CreateBucketIfNotExists([]byte(tableName))
            if err != nil {
                return err
            }

            id, err = bt.NextSequence()
            if err != nil {
                return err
            }

            return bt.Put(itob(id), valCopy)
        })
        chErr <- err
    }
    err := <-chErr
    return id, errors.WithStack(err)
}

Anyone any idea on how to solve this issue?