geth crashing when tip of chain is catched

[janzhanal]

System information

Geth version: v1.101408.0 from us-west1-docker.pkg.dev/devopsre/celo-blockchain-public/op-geth:celo8 CL client & version: us-west1-docker.pkg.dev/devopsre/celo-blockchain-public/op-node:celo9 OS & Version: Linux Commit hash : (if develop)

Expected behaviour

The app runs without exits...

Actual behaviour

The app crashes everytime the chain is on tip. Same issue appears on two separate instances.

INFO [10-03|16:28:09.593] Imported new potential chain segment     number=27,017,569 hash=1249d0..8d5e55 blocks=1 txs=5 mgas=0.586 elapsed=98.043ms    mgasps=5.975  triedirty=0.00B
INFO [10-03|16:28:09.594] Chain head was updated                   number=27,017,569 hash=1249d0..8d5e55 root=707761..6252be elapsed="175.371µs"
panic: revision id 128 cannot be reverted

goroutine 4793 [running]:
github.com/ethereum/go-ethereum/core/state.(*StateDB).RevertToSnapshot(0xc006564000, 0x80)
    github.com/ethereum/go-ethereum/core/state/statedb.go:778 +0x11b
github.com/ethereum/go-ethereum/contracts.(*CeloBackend).CallContract(0xc005e5e240, {0x0?, 0xc007e06f30?}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
    github.com/ethereum/go-ethereum/contracts/celo_backend.go:57 +0x314
github.com/ethereum/go-ethereum/accounts/abi/bind.(*BoundContract).Call(0xc005e59408, 0xc0032b9ae0?, 0xc0032b9a80?, {0x1b3b945, 0xf}, {0xc0032b9ab0?, 0xc005e44600?, 0xc0032b9aa8?})
    github.com/ethereum/go-ethereum/accounts/abi/bind/base.go:212 +0x2cd
github.com/ethereum/go-ethereum/contracts/celo/abigen.(*FeeCurrencyDirectoryCaller).GetExchangeRate(0xc00a006760, 0xc0032b9bb8, {0x3, 0xd3, 0xda, 0xb8, 0x43, 0xe6, 0xc0, 0x3b, ...})
    github.com/ethereum/go-ethereum/contracts/celo/abigen/FeeCurrencyDirectory.go:259 +0x9b
github.com/ethereum/go-ethereum/contracts.getExchangeRatesForTokens(0xc00a006760, {0xc00297c690, 0xc, 0x282b790?})
    github.com/ethereum/go-ethereum/contracts/fee_currencies.go:281 +0xe5
github.com/ethereum/go-ethereum/contracts.GetFeeCurrencyContext({0x282b790?, 0xc005e5e240?})
    github.com/ethereum/go-ethereum/contracts/fee_currencies.go:216 +0xb2
github.com/ethereum/go-ethereum/core.GetExchangeRates(0xc005e0b688?, 0xc0006b4c80, {0x2850c18, 0xc006564000})
    github.com/ethereum/go-ethereum/core/celo_evm.go:33 +0xa5
github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlockParallel.func1()
    github.com/ethereum/go-ethereum/eth/tracers/api.go:686 +0x1b3
created by github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlockParallel in goroutine 4792
    github.com/ethereum/go-ethereum/eth/tracers/api.go:682 +0x33e

INFO [10-03|16:34:27.547] Imported new potential chain segment     number=27,017,947 hash=fbcc35..854056 blocks=1 txs=1 mgas=0.044 elapsed=4.492ms     mgasps=9.755  triedirty=0.00B
INFO [10-03|16:34:27.548] Chain head was updated                   number=27,017,947 hash=fbcc35..854056 root=b4fdf2..618b6c elapsed="178.732µs"
panic: revision id 134 cannot be reverted

goroutine 5008 [running]:
github.com/ethereum/go-ethereum/core/state.(*StateDB).RevertToSnapshot(0xc0032dc540, 0x86)
    github.com/ethereum/go-ethereum/core/state/statedb.go:778 +0x11b
github.com/ethereum/go-ethereum/contracts.(*CeloBackend).CallContract(0xc003ddb5c0, {0x0?, 0xc007d25bc0?}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...)
    github.com/ethereum/go-ethereum/contracts/celo_backend.go:57 +0x314
github.com/ethereum/go-ethereum/accounts/abi/bind.(*BoundContract).Call(0xc003dc8508, 0xc001f29b80?, 0x412fbb?, {0x1b39041, 0xd}, {0x0?, 0x0?, 0x0?})
    github.com/ethereum/go-ethereum/accounts/abi/bind/base.go:212 +0x2cd
github.com/ethereum/go-ethereum/contracts/celo/abigen.(*FeeCurrencyDirectoryCaller).GetCurrencies(0x0?, 0x0?)
    github.com/ethereum/go-ethereum/contracts/celo/abigen/FeeCurrencyDirectory.go:194 +0x45
github.com/ethereum/go-ethereum/contracts.GetRegisteredCurrencies(0x7a3665ae72fb1292?)
    github.com/ethereum/go-ethereum/contracts/fee_currencies.go:183 +0x45
github.com/ethereum/go-ethereum/contracts.GetFeeCurrencyContext({0x282b790?, 0xc003ddb5c0?})
    github.com/ethereum/go-ethereum/contracts/fee_currencies.go:212 +0x8b
github.com/ethereum/go-ethereum/core.GetExchangeRates(0xc00c593908?, 0xc000528640, {0x2850c18, 0xc0032dc540})
    github.com/ethereum/go-ethereum/core/celo_evm.go:33 +0xa5
github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlockParallel.func1()
    github.com/ethereum/go-ethereum/eth/tracers/api.go:686 +0x1b3
created by github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlockParallel in goroutine 5006
    github.com/ethereum/go-ethereum/eth/tracers/api.go:682 +0x33e

Steps to reproduce the behaviour

Everytime when tip is catched

Configuration of get used (running in docker, with network host mode)

geth
        --datadir=/data/geth
        --networkid=44787
        --gcmode=archive  
        --syncmode full
        --snapshot=true
        --maxpeers=120
        --port=30022
        --authrpc.addr=127.0.0.1
        --authrpc.port={{ chain_vars.ports.authrpc }}
        --authrpc.jwtsecret=/data/jwt.hex
        --authrpc.vhosts='*' 
        --http 
        --http.addr=0.0.0.0
        --http.port=8551
        --http.api eth,net,web3,debug,admin,personal,txpool,engine
        --http.vhosts='*'
        --http.corsdomain='*'
        --verbosity=3
        --bootnodes=...
        --rollup.historicalrpc=https://alfajores-forno.cel1.alfajores.celo-testnet.org

[piersy]

Hi @janzhanal, thanks for raising this, I'm taking a look.

[piersy]

Hi @janzhanal I think the issue is to do with calling debug traceBlock methods. I guess we have a problem in our trace block function. So I think if you avoid calling any trace block api endpoints then you shouldn't see this re-ocurring. You may be able to use debug_TraceTransaction in the meantime while we look at fixing block tracing.

[piersy]

Hi @janzhanal I've pushed a potential fix to this branch (https://github.com/celo-org/op-geth/tree/piersy/fix-block-tracing) you should be able to pull an image for that from here (us-west1-docker.pkg.dev/blockchaintestsglobaltestnet/dev-images/op-geth:26785e097572ea11957716554a5f40c29860869b). Could you let me know if that is solving your problem?

[janzhanal]

Hello, thx for quick reply, but crashed again (and keeps crashing):

ERROR[10-03|20:20:20.227] RPC method debug_traceBlockByNumber crashed: revision id 326 cannot be reverted
goroutine 5429 [running]:
github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
    github.com/ethereum/go-ethereum/rpc/service.go:199 +0x85
panic({0x1875be0?, 0xc0190cd230?})
    runtime/panic.go:770 +0x132
github.com/ethereum/go-ethereum/core/state.(*StateDB).RevertToSnapshot(0xc00e756380, 0x146)
    github.com/ethereum/go-ethereum/core/state/statedb.go:778 +0x11b
github.com/ethereum/go-ethereum/core/vm.(*EVM).Call(0xc017616f00, {0x2823ea0?, 0xc0179ee0d8?}, {0x11, 0x53, 0xb6, 0xb8, 0xf9, 0x99, 0xc1, ...}, ...)
    github.com/ethereum/go-ethereum/core/vm/evm.go:288 +0xc15
github.com/ethereum/go-ethereum/core.(*StateTransition).innerTransitionDb(0xc0179ecbb8)
    github.com/ethereum/go-ethereum/core/state_transition.go:651 +0x9d9
github.com/ethereum/go-ethereum/core.(*StateTransition).TransitionDb(0xc0179ecbb8)
    github.com/ethereum/go-ethereum/core/state_transition.go:537 +0xf7
github.com/ethereum/go-ethereum/core.ApplyMessage(0xc017616f00, 0xc0179d42a0, 0xc017562ad0)
    github.com/ethereum/go-ethereum/core/state_transition.go:250 +0x386
github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlockParallel(0xc00897ceb0, {0x2834400, 0xc0169ecc80}, 0xc00d12eb40, 0xc00e756380, 0xc00e39b7c0)
    github.com/ethereum/go-ethereum/eth/tracers/api.go:727 +0xb1c
github.com/ethereum/go-ethereum/eth/tracers.(*API).traceBlock(0xc00897ceb0, {0x2834400, 0xc0169ecc80}, 0xc00d12eb40, 0xc00e39b7c0)
    github.com/ethereum/go-ethereum/eth/tracers/api.go:630 +0x212
github.com/ethereum/go-ethereum/eth/tracers.(*API).TraceBlockByNumber(0xc00897ceb0, {0x2834400, 0xc0169ecc80}, 0x197cbf5, 0xc00e39b7c0)
    github.com/ethereum/go-ethereum/eth/tracers/api.go:458 +0x257
reflect.Value.call({0xc009d10cb0?, 0xc009d58320?, 0x60?}, {0x1b2f35a, 0x4}, {0xc00e3a7620, 0x4, 0xc00e3a7620?})
    reflect/value.go:596 +0xca6
reflect.Value.Call({0xc009d10cb0?, 0xc009d58320?, 0x2?}, {0xc00e3a7620?, 0x16?, 0x16?})
    reflect/value.go:380 +0xb9
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc009d66b40, {0x2834400, 0xc0169ecc80}, {0xc016b43110, 0x18}, {0xc016b55920, 0x2, 0x4debaf?})
    github.com/ethereum/go-ethereum/rpc/service.go:205 +0x36d
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc00e3b2b40?, {0x2834400?, 0xc0169ecc80?}, 0xc0171121c0, 0x2?, {0xc016b55920?, 0xc00abb5dc0?, 0x41bed8?})
    github.com/ethereum/go-ethereum/rpc/handler.go:568 +0x3c
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc00e3ac5a0, 0xc016b55860, 0xc0171121c0)
    github.com/ethereum/go-ethereum/rpc/handler.go:515 +0x22f
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc00e3ac5a0, 0xc016b55860, 0xc0171121c0)
    github.com/ethereum/go-ethereum/rpc/handler.go:473 +0x22d
github.com/ethereum/go-ethereum/rpc.(*handler).handleNonBatchCall(0xc00e3ac5a0, 0xc016b55860, 0xc0171121c0)
    github.com/ethereum/go-ethereum/rpc/handler.go:299 +0x18a
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1.1(0x2834400?)
    github.com/ethereum/go-ethereum/rpc/handler.go:272 +0x25
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
    github.com/ethereum/go-ethereum/rpc/handler.go:390 +0xbe
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc in goroutine 5154
    github.com/ethereum/go-ethereum/rpc/handler.go:386 +0x79

WARN [10-03|20:20:20.227] Served debug_traceBlockByNumber          conn=127.0.0.1:50748 reqid=0 duration=61.831267ms err="method handler crashed"
fatal error: concurrent map read and map write

I can paste whole trace if needed...

[janzhanal]

Thanks, based on your commit we found the right image which seems to be working: us-west1-docker.pkg.dev/blockchaintestsglobaltestnet/dev-images/op-geth:11727696daa36973cae5a1a8dd2268316f57465b

[piersy]

Hi @janzhanal my apologies for the incorrect docker link. Glad it seems to be working for you, please let me know if you face any other issues with it.

[janzhanal]

Till now it seems stable and no errors were reported...

[piersy]

Closed by https://github.com/celo-org/op-geth/pull/252