search

Fantom-foundation / go-opera

Opera blockchain protocol secured by the Lachesis consensus algorithm
https://fantom.foundation/
GNU Lesser General Public License v3.0
262 stars 268 forks source link

panic: runtime error: slice bounds out of range for archive node #508

Closed Johnaverse closed 10 months ago

Johnaverse commented 10 months ago

version: go-opera/v1.1.2-txtracing-rc.6-f8983789-1691829635

The ftm height is 67571797 and the opera crashed with following error

Aug 29 13:06:18 fantom-arch46 opera[1510228]: panic: runtime error: slice bounds out of range [:-8628854311100398772]
Aug 29 13:06:18 fantom-arch46 opera[1510228]: goroutine 6952 [running]:
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/txtrace.(*TraceStructLogger).CaptureState(0xc01669d340, 0x0?, 0xc4b1ca4d72b62a64?, 0xf1, 0x698d4, 0x0?, 0>
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/txtrace/trace_logger.go:178 +0x2710
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run.func4()
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/build/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/core/vm/interpreter.go:174 +0x123
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run(0xc02aaaa900, 0xc02da45740, {0xc002244c30, 0x44, 0x44}, 0x0)
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/build/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/core/vm/interpreter.go:234 +0xe2f
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc02aaa6a80, {0x19910e0, 0xc000493080}, {0x6e, 0x2f, 0x9b, 0x54, 0x48, 0x98, 0xf6, ...}, .>
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/build/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/core/vm/evm.go:237 +0xd05
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/evmcore.(*StateTransition).TransitionDb(0xc01a94d880)
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/evmcore/state_transition.go:292 +0x599
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/evmcore.ApplyMessage(0x168b2a0?, {0x19a2b08?, 0xc02aa80a20?}, 0xfc54f24db7c4adc2?)
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/evmcore/state_transition.go:173 +0x32
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/evmcore.applyTransaction({0xc000492f00, {0x0, 0x1a, 0x5, 0x55, 0x7b, 0xd0, 0xd9, 0x21, 0x2, ...}, ...}, .>
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/evmcore/state_processor.go:137 +0x4a8
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/evmcore.(*StateProcessor).Process(_, _, _, {0x1, {0x199c4e0, 0xc01669d340}, 0x0, 0x0, 0x0, {0x0, ...}, ..>
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/evmcore/state_processor.go:85 +0x60d
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/gossip/blockproc/evmmodule.(*OperaEVMProcessor).Execute(0xc02c932000, {0xc027efc0a0?, 0x6, 0xc0175923c0?})
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/gossip/blockproc/evmmodule/evm.go:91 +0x145
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/gossip.consensusCallbackBeginBlockFn.func1.3.4()
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/gossip/c_block_callbacks.go:322 +0x7ae
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/go-opera/gossip.consensusCallbackBeginBlockFn.func1.3.5()
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/go-opera/gossip/c_block_callbacks.go:469 +0x83
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/lachesis-base/utils/workers.worker(...)
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/build/pkg/mod/github.com/!fantom-foundation/lachesis-base@v0.0.0-20221208123620-82a6d15f995c/utils/worker>
Aug 29 13:06:18 fantom-arch46 opera[1510228]: github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start.func1()
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/build/pkg/mod/github.com/!fantom-foundation/lachesis-base@v0.0.0-20221208123620-82a6d15f995c/utils/worker>
Aug 29 13:06:18 fantom-arch46 opera[1510228]: created by github.com/Fantom-foundation/lachesis-base/utils/workers.(*Workers).Start
Aug 29 13:06:18 fantom-arch46 opera[1510228]:         /tmp/build.1692887881.1599549/build/pkg/mod/github.com/!fantom-foundation/lachesis-base@v0.0.0-20221208123620-82a6d15f995c/utils/worker>
Aug 29 13:06:18 fantom-arch46 systemd[1]: opera.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

My parms:

  --genesis=<path>
  --config=<path>
  --port=5050
  --maxpeers=200
  --datadir=<path>
  --http
  --http.addr=0.0.0.0
  --http.port=18545
  --http.corsdomain=*
  --http.vhosts=*
  --ws
  --ws.addr=0.0.0.0
  --ws.port=18546
  --ws.origins=*
  --db.preset=pbl-1
  --tracenode
  --http.api=eth,web3,net,ftm,trace
  --metrics
  --metrics.addr=0.0.0.0
  --metrics.port=6060
muratso commented 10 months ago

Same here using the same version.

muratso commented 10 months ago

It seems this has been fixed on the latest commit on that branch. Just not sure if it's safe to use

HonzaDajc commented 10 months ago

Yes, this issue is fixed in release/txtracing/1.1.2-rc.6, please use this updated branch and is safe to use. Node should recover itself.

datanexus-vincent commented 10 months ago

FYI, we're still running smoothly on v1.1.2-rc.5 after we missed the upgrade to rc.6.

uprendis commented 10 months ago

The latest hotfixes are now available in the following branches:

For version 1.1.2, refer to the branch release/txtracing/1.1.2-rc.6.1. For version 1.1.3, check out release/txtracing/1.1.3-rc.3.1. If you previously used the release/txtracing/1.1.2-rc.5 branch, we recommend upgrading directly to release/txtracing/1.1.2-rc.6.1.

While the earlier hotfix addressed this specific crash, there remained potential for other related crashes. These have now been addressed in the recent update.

If you experienced this crash, It's likely that your node DB is in now a dirty state and won't restart. You can execute the command opera --datadir /path/to/datadir db heal --experimental for recovery. When you restart your node, it will start the EVM snapshot reconstruction. This recovery process may be time-consuming, with the duration primarily influenced by your disk speed.

uprendis commented 10 months ago

Duplication of #505