search

celestiaorg / celestia-node

Celestia Data Availability Nodes
Apache License 2.0
921 stars 921 forks source link

crash on node launch: Arena too small, toWrite:16678 newTotal:21810897 limit:21810342 #3097

Open tuxcanfly opened 8 months ago

tuxcanfly commented 8 months ago

Celestia Node version

v0.12.2

OS

macOS 14.1.0

Install tools

No response

Others

celestia-da e1ad7cf4

Steps to reproduce it

it crashed after launch and keeps crashing, probably due to using the same node store

Expected result

node launches without crash

Actual result

node crashes on launch with the following log:

Relevant log output

2024-01-11T09:27:46.968-0800    INFO    cmd     celestia-da/server.go:48        serving celestia-da over gRPC on: 127.0.0.1:57213
2024-01-11T09:27:46.971-0800    INFO    node    nodebuilder/module.go:25        Accessing keyring...
2024-01-11T09:27:46.974-0800    INFO    module/state    state/keyring.go:40     constructed keyring signer      {"backend": "test", "path": "/Users/tuxcanfly/.celestia-light/keys", "key name": "my_celes_key", "chain-id": "celestia"}
2024/01/11 09:27:47 Arena too small, toWrite:16678 newTotal:21810653 limit:21810342
github.com/dgraph-io/badger/v4/y.AssertTruef
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/y/error.go:62
github.com/dgraph-io/badger/v4/skl.(*Arena).putVal
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/skl/arena.go:81
github.com/dgraph-io/badger/v4/skl.newNode
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/skl/skl.go:114
github.com/dgraph-io/badger/v4/skl.(*Skiplist).Put
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/skl/skl.go:304
github.com/dgraph-io/badger/v4.(*memTable).UpdateSkipList.(*memTable).replayFunction.func1
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/memtable.go:257
github.com/dgraph-io/badger/v4.(*logFile).iterate
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/memtable.go:516
github.com/dgraph-io/badger/v4.(*memTable).UpdateSkipList
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/memtable.go:218
github.com/dgraph-io/badger/v4.(*DB).openMemTable
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/memtable.go:146
github.com/dgraph-io/badger/v4.(*DB).openMemTables
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/memtable.go:88
github.com/dgraph-io/badger/v4.Open
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.1-0.20231013074411-fb1b00959581/db.go:335
github.com/celestiaorg/go-ds-badger4.NewDatastore
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/celestiaorg/go-ds-badger4@v0.0.0-20230712104058-7ede1c814ac5/datastore.go:186
github.com/celestiaorg/celestia-node/nodebuilder.(*fsStore).Datastore
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/celestiaorg/celestia-node@v0.12.2/nodebuilder/store.go:125
reflect.Value.call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/go/src/reflect/value.go:596
reflect.Value.Call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/go/src/reflect/value.go:380
go.uber.org/dig.defaultInvoker
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/container.go:238
go.uber.org/dig.(*constructorNode).Call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/constructor.go:185
go.uber.org/dig.paramSingle.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:287
go.uber.org/dig.paramList.BuildList
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:150
go.uber.org/dig.(*constructorNode).Call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/constructor.go:155
go.uber.org/dig.paramSingle.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:287
go.uber.org/dig.paramList.BuildList
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:150
go.uber.org/dig.(*constructorNode).Call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/constructor.go:155
go.uber.org/dig.paramSingle.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:287
go.uber.org/dig.paramObjectField.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:484
go.uber.org/dig.paramObject.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:412
go.uber.org/dig.paramList.BuildList
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:150
go.uber.org/dig.(*constructorNode).Call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/constructor.go:155
go.uber.org/dig.paramSingle.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:287
go.uber.org/dig.paramList.BuildList
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:150
go.uber.org/dig.(*constructorNode).Call
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/constructor.go:155
go.uber.org/dig.paramSingle.Build
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:287
go.uber.org/dig.paramList.BuildList
        /Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/dig@v1.17.1/param.go:150

Notes

No response

walldiss commented 8 months ago

Does it constantly crash or it was a single event?

Nvm, found the answer

walldiss commented 8 months ago

The problem arises from the badger-db internal allocator, which failed to acquire memory. One possible reason for this issue could be that the store was initialized with a larger Memtable size than it actually ran on. This discrepancy might be due to the MemTable size being downscaled in the v0.12.1 release.

Is there a possibility that you upgraded the node application from a version pre-dating v0.12.1 without reinitializing the store?

tuxcanfly commented 8 months ago

Hi @walldiss I didn't remember which version I upgraded from, so I assume it was from v0.12.0. I tried running v0.12.0 with the same store, then upgraded to v0.12.2 and tried again. It didn't crash and I'm no longer able to reproduce this.

Also, is it generally recommended to reinit the store on minor version upgrades or major version upgrades or both? I'm asking because this could be useful information for service providers assuming not reiniting causes issues like this.

upnodedev commented 8 months ago

It also happens without upgrading the version.

  1. Sync light node v0.12.2
  2. Stop light node v0.12.2
  3. Start light node v0.12.2 again -> Arena too small

But the last time I upgraded to v0.12.3 it worked. Possibly this behavior is random.

Partykingdk commented 8 months ago

the same with me

MilosTodorovic commented 8 months ago

Facing same issue for 0.12.4