search

Fantom-foundation / go-opera

Opera blockchain protocol secured by the Lachesis consensus algorithm
https://fantom.foundation/
GNU Lesser General Public License v3.0
283 stars 283 forks source link

SIGSEGV Segmentation violation #429

Closed linearray closed 1 year ago

linearray commented 1 year ago

Describe the bug During catching up to head after db heal node crashes with SIGSEGV. Node can be restarted, creates another state snapshot and runs again.

To Reproduce

./bin/opera                                      \
       --genesis     /svc/fantom/var/mainnet.g   \
       --datadir     /svc/fantom/var/            \
       --cache       20000                       \
       --v5disc                                  \
       --maxpeers    450                         \
       --http                                    \
       --http.api    txpool,eth                  \
       --http.addr   0.0.0.0                     \
       --http.vhosts '*'                         \
       --ws                                      \
       --ws.api      txpool,eth                  \
       --ws.origins  '*'                         \
       --syncmode    snap                        \
       --unlock      {{redacted}}                \
       --password    /svc/fantom/var/geth.pass   \
       --allow-insecure-unlock                   \
       --verbosity 3                             \
       --gcmode full                             \
       --db.migration.mode rebuild               \
       --db.preset pbl-1

Additional context

INFO [02-25|11:11:05.773] New block                                index=56310958 id=192810:6522:2f8909  gas_used=14,296,616 txs=19/0  age=2d17h41m       t=1.135s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x1b82023]

goroutine 8925108 [running]:
github.com/Fantom-foundation/go-opera/gossip.(*GPOBackend).TotalGasPowerLeft(0xc015ccbd58)
        /svc/fantom/opt/go-opera/gossip/gpo_backend.go:61 +0x403
github.com/Fantom-foundation/go-opera/gossip/gasprice.(*Oracle).constructiveGasPrice(0xc011847600, 0x0, 0x0, 0x60000000a?)
        /svc/fantom/opt/go-opera/gossip/gasprice/constructive.go:26 +0x4b
github.com/Fantom-foundation/go-opera/gossip/gasprice.(*Oracle).effectiveMinGasPrice(0xc011847600)
        /svc/fantom/opt/go-opera/gossip/gasprice/constructive.go:20 +0x75
github.com/Fantom-foundation/go-opera/gossip/gasprice.(*Oracle).EffectiveMinGasPrice(0xc011847600)
        /svc/fantom/opt/go-opera/gossip/gasprice/gasprice.go:205 +0x265
github.com/Fantom-foundation/go-opera/gossip.(*EvmStateReader).EffectiveMinTip(0xc014b3ec78)
        /svc/fantom/opt/go-opera/gossip/evm_state_reader.go:47 +0x9b
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).validateTx(0xc0000003c0, 0xc046925800, 0x0)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:643 +0x307
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).add(0xc0000003c0, 0xc046925800, 0x0)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:690 +0xe5
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).addTxsLocked(0xc0000003c0, {0xc054a820b8, 0x1, 0x819eb8?}, 0x80?)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:955 +0x119
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).addTxs(0xc0000003c0, {0xc1f4c6cee0, 0x1, 0xa003c51b88ad6e06?}, 0xce?, 0x0)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:930 +0x2c9
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).AddRemotes(0xe0e66d8051eed36b?, {0xc1f4c6cee0?, 0xa003c51b88ad6e06?, 0x49288187c2076cce?})
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:876 +0x25
github.com/Fantom-foundation/go-opera/gossip.(*handler).handleTxs(0xc010076600, 0xc1975c0e10?, {0xc1f4c6cee0?, 0x1, 0x4})
        /svc/fantom/opt/go-opera/gossip/handler.go:917 +0xcf
github.com/Fantom-foundation/go-opera/gossip.(*handler).handleMsg(0xc010076600, 0xc08bb36e00)
        /svc/fantom/opt/go-opera/gossip/handler.go:1036 +0x40de
github.com/Fantom-foundation/go-opera/gossip.(*handler).handle(0xc010076600, 0xc08bb36e00)
        /svc/fantom/opt/go-opera/gossip/handler.go:861 +0xe6d
github.com/Fantom-foundation/go-opera/gossip.MakeProtocols.func1(0x1?, {0x812518, 0xc0166e4320})
        /svc/fantom/opt/go-opera/gossip/service.go:371 +0x1ae
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1()
        /root/go/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/p2p/peer.go:407 +0x8c
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        /root/go/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/p2p/peer.go:405 +0xae
⏎

Desktop (please complete the following information): v1.1.2-rc.5 on FreeBSD 13.1-RELEASE-p2

linearray commented 1 year ago

It just crashed again on the same block:

INFO [02-25|20:14:06.138] Start/Switch to fullsync mode...         module=PM
INFO [02-25|20:14:07.182] New LLR summary                          last_epoch=193489 last_block=56508040 new_evs=0 new_ers=0 new_bvs=252 new_brs=0    age=1m39.383s
INFO [02-25|20:14:07.182] New DAG summary                          new=1 last_id=192810:6354:4a6c57 age=3d2h44m     t=4.976ms
INFO [02-25|20:14:07.254] New block                                index=56310944 id=192810:6334:dab95b gas_used=388,304 txs=4/0 age=3d2h44m     t=44.469ms
INFO [02-25|20:14:07.288] New block                                index=56310945 id=192810:6348:bf1860 gas_used=349,456 txs=5/0 age=3d2h44m     t=26.227ms
INFO [02-25|20:14:07.312] New block                                index=56310946 id=192810:6360:9a0253 gas_used=415,382 txs=4/0 age=3d2h44m     t=24.069ms
INFO [02-25|20:14:07.352] New block                                index=56310947 id=192810:6372:3641f9 gas_used=834,227 txs=2/0 age=3d2h44m     t=40.180ms
INFO [02-25|20:14:07.486] New block                                index=56310948 id=192810:6387:85dde2 gas_used=1,797,503 txs=9/0 age=3d2h44m     t=134.186ms
INFO [02-25|20:14:07.513] New block                                index=56310949 id=192810:6402:a9cdf0 gas_used=333,214   txs=3/0 age=3d2h44m     t=26.505ms
INFO [02-25|20:14:07.554] New block                                index=56310950 id=192810:6414:7b7496 gas_used=413,752   txs=2/0 age=3d2h44m     t=41.267ms
INFO [02-25|20:14:07.767] New block                                index=56310951 id=192810:6428:37261b gas_used=2,014,394 txs=9/0 age=3d2h44m     t=212.878ms
INFO [02-25|20:14:07.768] New block                                index=56310952 id=192810:6440:0171fb gas_used=21400     txs=1/0 age=3d2h44m     t=1.065ms
INFO [02-25|20:14:07.952] New block                                index=56310953 id=192810:6454:dfa9eb gas_used=3,221,864 txs=13/0 age=3d2h44m     t=175.214ms
INFO [02-25|20:14:07.985] New block                                index=56310954 id=192810:6468:fecf2a gas_used=851,997   txs=4/1  age=3d2h44m     t=32.597ms
INFO [02-25|20:14:08.025] New block                                index=56310955 id=192810:6483:93f301 gas_used=2,891,574 txs=12/0 age=3d2h44m     t=40.683ms
INFO [02-25|20:14:09.750] New block                                index=56310956 id=192810:6496:370456 gas_used=2,474,499 txs=19/0 age=3d2h44m     t=45.771ms
INFO [02-25|20:14:09.782] New block                                index=56310957 id=192810:6508:ef651f gas_used=943,583   txs=3/0  age=3d2h44m     t=31.858ms
INFO [02-25|20:14:11.253] New block                                index=56310958 id=192810:6522:2f8909 gas_used=14,296,616 txs=19/0 age=3d2h44m     t=1.470s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x1b82023]

goroutine 3475 [running]:
github.com/Fantom-foundation/go-opera/gossip.(*GPOBackend).TotalGasPowerLeft(0xc0152afb30)
        /svc/fantom/opt/go-opera/gossip/gpo_backend.go:61 +0x403
github.com/Fantom-foundation/go-opera/gossip/gasprice.(*Oracle).constructiveGasPrice(0xc000202d00, 0x0, 0x0, 0x60000000a?)
        /svc/fantom/opt/go-opera/gossip/gasprice/constructive.go:26 +0x4b
github.com/Fantom-foundation/go-opera/gossip/gasprice.(*Oracle).effectiveMinGasPrice(0xc000202d00)
        /svc/fantom/opt/go-opera/gossip/gasprice/constructive.go:20 +0x75
github.com/Fantom-foundation/go-opera/gossip/gasprice.(*Oracle).EffectiveMinGasPrice(0xc000202d00)
        /svc/fantom/opt/go-opera/gossip/gasprice/gasprice.go:205 +0x265
github.com/Fantom-foundation/go-opera/gossip.(*EvmStateReader).EffectiveMinTip(0xc015077710)
        /svc/fantom/opt/go-opera/gossip/evm_state_reader.go:47 +0x9b
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).validateTx(0xc0000005a0, 0xc0477be720, 0x0)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:643 +0x307
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).add(0xc0000005a0, 0xc0477be720, 0x0)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:690 +0xe5
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).addTxsLocked(0xc0000005a0, {0xc0723df9d0, 0x1, 0x819eb8?}, 0x0?)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:955 +0x119
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).addTxs(0xc0000005a0, {0xc074eb5600, 0x1, 0xcbea3579fde6bc5e?}, 0x47?, 0x0)
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:930 +0x2c9
github.com/Fantom-foundation/go-opera/evmcore.(*TxPool).AddRemotes(0x5e09ab28f7019ea5?, {0xc074eb5600?, 0xcbea3579fde6bc5e?, 0x5bf12bd3d5beaf47?})
        /svc/fantom/opt/go-opera/evmcore/tx_pool.go:876 +0x25
github.com/Fantom-foundation/go-opera/gossip.(*handler).handleTxs(0xc000d3e000, 0xc0427a3d00?, {0xc074eb5600?, 0x1, 0x4})
        /svc/fantom/opt/go-opera/gossip/handler.go:917 +0xcf
github.com/Fantom-foundation/go-opera/gossip.(*handler).handleMsg(0xc000d3e000, 0xc008a4cb00)
        /svc/fantom/opt/go-opera/gossip/handler.go:1036 +0x40de
github.com/Fantom-foundation/go-opera/gossip.(*handler).handle(0xc000d3e000, 0xc008a4cb00)
        /svc/fantom/opt/go-opera/gossip/handler.go:861 +0xe6d
github.com/Fantom-foundation/go-opera/gossip.MakeProtocols.func1(0x11bd03e?, {0x812518, 0xc008045900})
        /svc/fantom/opt/go-opera/gossip/service.go:371 +0x1ae
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1()
        /root/go/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/p2p/peer.go:407 +0x8c
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        /root/go/pkg/mod/github.com/!fantom-foundation/go-ethereum@v1.10.8-ftm-rc11/p2p/peer.go:405 +0xae
uprendis commented 1 year ago

Hello @linearray, Good day to you!

Please try this fix: https://github.com/uprendis/go-opera/tree/feature/fix-db-heal-db-epochs-erasing After rebuilding opera, try db heal command one more time and then start syncing

uprendis commented 1 year ago

Before trying db heal one more time, erase:

rm -r /svc/fantom/var/chaindata/leveldb-fsh/epoch-*
linearray commented 1 year ago

Thank you for the reply and the quick fix. However, I restored the node from a snapshot to get it back up running ASAP.

If the problem returns, I will try the fix and report.