search

dgraph-io / badger

Fast key-value DB in Go.
https://dgraph.io/badger
Apache License 2.0
13.51k stars 1.15k forks source link

[BUG]: Panic from a lot of DELs and SETs with very large keys #2020

Open nathanhack opened 8 months ago

nathanhack commented 8 months ago

What version of Badger are you using?

v4.2.0 But it also happened with v3.2103.2.

What version of Go are you using?

1.21.1

Have you tried reproducing the issue with the latest release?

None

What is the hardware spec (RAM, CPU, OS)?

128GiB, AMD Ryzen™ 9 5950X × 32, Fedora Linux 38 (Workstation Edition)

What steps will reproduce the bug?

running a bunch (several 100k) of DELs followed by SETs with large very large keys.

Expected behavior and actual result.

Expected: no panic.

Actual: a panic

panic: cannot grow buffer beyond 2 gigabytes

goroutine 1424649 [running]:
github.com/google/flatbuffers/go.(*Builder).growByteBuffer(...)
    /homeDIR/go/pkg/mod/github.com/google/flatbuffers@v23.5.26+incompatible/go/builder.go:216
github.com/google/flatbuffers/go.(*Builder).Prep(0xc152a58c80, 0x4, 0x5b48)
    /homeDIR/go/pkg/mod/github.com/google/flatbuffers@v23.5.26+incompatible/go/builder.go:270 +0x2b8
github.com/google/flatbuffers/go.(*Builder).CreateByteVector(0xc152a58c80, {0xc13207a000, 0x5b48, 0xc152a58b60?})
    /homeDIR/go/pkg/mod/github.com/google/flatbuffers@v23.5.26+incompatible/go/builder.go:390 +0x45
github.com/dgraph-io/badger/v4/table.(*Builder).writeBlockOffset(0xc07bb0e000?, 0xc152a58c80, 0xc0202ec410, 0x47765e50?)
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:591 +0x3d
github.com/dgraph-io/badger/v4/table.(*Builder).writeBlockOffsets(0xc001a15b90, 0x43f38e?)
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:579 +0xc5
github.com/dgraph-io/badger/v4/table.(*Builder).buildIndex(0xc001a15b90, {0xc001574000, 0xfbe5, 0x1156a})
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:540 +0x106
github.com/dgraph-io/badger/v4/table.(*Builder).Done(0xc001a15b90)
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/builder.go:450 +0x145
github.com/dgraph-io/badger/v4/table.CreateTable({0xc0791fa030, 0x16}, 0xc001a15b90)
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/table/table.go:258 +0x3d
github.com/dgraph-io/badger/v4.(*levelsController).subcompact.func4(0xc001a15b90, 0xc001a14090?)
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go:846 +0x109
created by github.com/dgraph-io/badger/v4.(*levelsController).subcompact in goroutine 1418647
    /homeDIR/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go:836 +0x68f

Additional information

No response