0xPolygon / polygon-edge

A Framework for Building Ethereum-compatible Blockchain Networks
https://polygon.technology/solutions/polygon-edge/
Apache License 2.0
1.02k stars 535 forks source link

Runtime error after dispatcher fails to dispatch #92

Closed dankostiuk closed 3 years ago

dankostiuk commented 3 years ago

Runtime error after dispatcher fails to dispatch

Description

Running locally, we've seen what appears to be a runtime error likely right after the error: [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getCode err="given root and slot not found in storage".

Your environment

Steps to reproduce

No runtime error.

Actual behaviour

Runtime error (see logs).

Logs

2021-07-16T10:39:20.673-0400 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getTransactionCount err="given root and slot not found in storage"
2021-07-16T10:39:40.661-0400 [DEBUG] polygon.dispatcher: request: method=eth_blockNumber id=3.325198870728132e+15
2021-07-16T10:39:49.879-0400 [DEBUG] polygon.dispatcher: request: method=eth_getBalance id=5.813328093453609e+15
2021-07-16T10:39:53.868-0400 [DEBUG] polygon.dispatcher: request: method=eth_gasPrice id=bfadc345-7925-44c1-bbe1-0af437654a53
2021-07-16T10:39:54.655-0400 [DEBUG] polygon.dispatcher: request: method=eth_getCode id=1d6f7e80-dd9b-4c76-afcb-4c404722bb40
2021-07-16T10:39:54.655-0400 [ERROR] polygon.dispatcher: failed to dispatch: method=eth_getCode err="given root and slot not found in storage"
2021-07-16T10:39:54.661-0400 [DEBUG] polygon.dispatcher: request: method=eth_estimateGas id=8.258272843514e+12
2021/07/16 10:39:54 http: panic serving 127.0.0.1:63872: runtime error: index out of range [-1]
goroutine 185 [running]:
net/http.(*conn).serve.func1(0xc0006c6460)
    /usr/local/go/src/net/http/server.go:1824 +0x153
panic(0x1a9e820, 0xc002119830)
    /usr/local/go/src/runtime/panic.go:971 +0x499
github.com/0xPolygon/minimal/crypto.RecoverPubkey(0x0, 0x0, 0x0, 0xc000581820, 0x20, 0x20, 0x7a, 0xc00209c200, 0xc000609070)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/crypto/crypto.go:128 +0x1aa
github.com/0xPolygon/minimal/consensus/ibft.ecrecoverImpl(0x0, 0x0, 0x0, 0xc0005817e0, 0x20, 0x20, 0x0, 0x0, 0xc000000000, 0xc0020282b0, ...)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/consensus/ibft/sign.go:22 +0xb8
github.com/0xPolygon/minimal/consensus/ibft.ecrecoverFromHeader(0xc000147d40, 0x0, 0x0, 0x0, 0xc000616c40, 0x4)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/consensus/ibft/sign.go:42 +0x119
github.com/0xPolygon/minimal/consensus/ibft.(*Ibft).GetBlockCreator(0xc0006aaf70, 0xc000147d40, 0x0, 0x0, 0x0, 0x0, 0x269e2e0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/consensus/ibft/ibft.go:921 +0x45
github.com/0xPolygon/minimal/minimal.(*jsonRPCHub).ApplyTxn(0xc0016dec30, 0xc000147d40, 0xc0002d1080, 0xc000147d40, 0x5a5d896ffadb8ffe, 0xc000616bc0, 0xc000616be0, 0xc0006e2580, 0xc0002d0fd0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/minimal/server.go:281 +0x5a
github.com/0xPolygon/minimal/jsonrpc.(*Eth).EstimateGas.func1(0x7ca24, 0xc00212a110, 0x0, 0x0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/eth_endpoint.go:312 +0x7c
github.com/0xPolygon/minimal/jsonrpc.(*Eth).EstimateGas(0xc0005901b8, 0xc002058ac0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/eth_endpoint.go:324 +0x332
reflect.Value.call(0xc000426880, 0xc000590230, 0x13, 0x1b194ca, 0x4, 0xc00211ceb0, 0x3, 0x3, 0x19690a0, 0xc00069b038, ...)
    /usr/local/go/src/reflect/value.go:476 +0x8e7
reflect.Value.Call(0xc000426880, 0xc000590230, 0x13, 0xc00211ceb0, 0x3, 0x3, 0x0, 0xa0, 0xc000430a60)
    /usr/local/go/src/reflect/value.go:337 +0xb9
github.com/0xPolygon/minimal/jsonrpc.(*Dispatcher).handleReq(0xc0006004b0, 0x1994d80, 0xc002103f00, 0xc002103f10, 0xf, 0xc0020141e0, 0x94, 0xa0, 0xc00011a060, 0xc00011a060, ...)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/dispatcher.go:234 +0x4cd
github.com/0xPolygon/minimal/jsonrpc.(*Dispatcher).Handle(0xc0006004b0, 0xc0006d7800, 0xdd, 0x200, 0x200, 0x0, 0x0, 0xc001fa5b90, 0x1335676)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/dispatcher.go:209 +0x13f
github.com/0xPolygon/minimal/jsonrpc.(*JSONRPC).handle(0xc0016df140, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /Users/dan/go/src/github.com/0xPolygon/polygon-sdk/jsonrpc/jsonrpc.go:149 +0x4b9
net/http.HandlerFunc.ServeHTTP(0xc00063eb70, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /usr/local/go/src/net/http/server.go:2069 +0x44
net/http.(*ServeMux).ServeHTTP(0x266d640, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /usr/local/go/src/net/http/server.go:2448 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00058e540, 0x1e46f00, 0xc0006d1260, 0xc002126100)
    /usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0006c6460, 0x1e498c0, 0xc0005c6880)
    /usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3013 +0x39b
2021-07-16T10:39:54.665-0400 [DEBUG] polygon.dispatcher: request: method=eth_estimateGas id=8.258272843514e+12

Proposed solution

If you have an idea of how to fix this issue, please write it down here, so we can begin discussing it

lazartravica commented 3 years ago

Hello @dankostiuk!

This issue is caused by the eth_estimateGas you can see above.

What happens is that you try to estimateGas for a specific block, and to do that we need to simulate the transaction inside that block. To run the gas estimation simulation we need to know the coinbase accounts for the given block which we unmarshal from the IBFT extradata header field of the block.

It is this checking of the IBFT extradata headers that fails and panics, since the block eth_estimateGas is ran on is the genesis block which contains no extradata headers containing the signature of the proposer.

This is symptomatic of the blockchain network not working, thus not producing blocks.

I would need you to check two things:

  1. Check that the peers are connected:

Run the peers list command: go run main.go peers list --grpc-address=... and you should see 3 peers of the given node.

  1. Check that the block production is working:

Although block production is visible in the main log output, it is much cleaner to run go run main.go monitor --grpc-address=... and you should be able to see live events like this:

[BLOCK EVENT]

Event Type   = ADD BLOCK
Block Number = 353
Block Hash   = 0xed5cc60a9e57bb414cddd9ae1033d8e85db16c0940101d7d1004ce38e9a1b16

If there are no ADD BLOCK events visible in the output of the monitor command, then we can safely say that the block production is not working for some reason and we can troubleshoot further.

dankostiuk commented 3 years ago

Thanks @lazartravica , I'll make sure to do this next time I see this error so I can provide more details.