search

OffchainLabs / nitro

Nitro goes vroom and fixes everything
Other
747 stars 444 forks source link

"couldn't find block with transactions. Reached 1891" when requesting block 1897 #2087

Open darkhorse-spb opened 9 months ago

darkhorse-spb commented 9 months ago

Describe the bug We have deployed several Arbitrum Mainnet nodes using this manual, they successfully bootstrapped from https://snapshot.arbitrum.io/mainnet/nitro.tar snapshot. Nodes are now in sync and successfully processing all requests except three blocks (1897, 1898, 1899). There is also an incident report on Quicknodes RPC provider about the same issue: https://status.quicknode.com/incidents/2z5zms94f39l

To Reproduce Steps to reproduce the behavior: curl --location --header 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x769",false],"id":0}' <node address> {"jsonrpc": "2.0", "id": 0, "error": {"code": -32000, "message": "couldn't find block with transactions. Reached 1891"}}

We discovered that multiple public RPC endpoints are facing the same issue, like curl --location --header 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x769",false],"id":0}' https://rpc.ankr.com/arbitrum {"jsonrpc":"2.0","id":0,"error":{"code":-32000,"message":"couldn't find block with transactions. Reached 1891"} curl --location --header 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x769",false],"id":0}' https://arbitrum-one.publicnode.com {"jsonrpc":"2.0","id":0,"error":{"code":-32000,"message":"couldn't find block with transactions. Reached 1891"}}

but some are not: https://arbitrum.llamarpc.com https://arbitrum-one.public.blastapi.io

Expected behavior Node should answer the block data: {"jsonrpc":"2.0","result":{"difficulty":"0x0","extraData":"0x","gasLimit":"0x112a717a","gasUsed":"0x0","hash":"0xa707c2f805d7e0f883e1c50a593326c969616809a8cb9db8bd97304d83ab8f46","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000000","number":"0x769","parentHash":"0xf7beabc1d044e75572604c473a621cdcb22b76de6836834cb3ddc2bbe1c9c6bf","receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x1ff","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x60c8386d","totalDifficulty":"0x0","transactions":[],"transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","uncles":[]},"id":0}

Additional context Version in use is Arbitrum Nitro v2.2.1

Johnaverse commented 8 months ago

Confirm that issue are still here with Arbitrum Nitro v2.2.2 Block is accessible directly query to classic endpoint

Sarahligbe commented 8 months ago

I also have the same problem and I'm running Arbitrum Nitro v2.2.2

@darkhorse-spb were you able to resolve it?

darkhorse-spb commented 8 months ago

@darkhorse-spb were you able to resolve it?

@Sarahligbe no, I'm waiting for some help from developers

Block is accessible directly query to classic endpoint

@Johnaverse could you please kindly share an example of request to classic endpoint?

Johnaverse commented 8 months ago

RPC request

curl --location 'http://<arbone_classic_ip>:<port>' \
--header 'Content-Type: application/json' \
--data '{
    "jsonrpc":"2.0",
    "method":"eth_getBlockByNumber",
    "params":[
        "0x769", 
        true
    ],
    "id":1
}'

Expected Response

{"jsonrpc":"2.0","id":1,"result":{"number":"0x769","hash":"0xa707c2f805d7e0f883e1c50a593326c969616809a8cb9db8bd97304d83ab8f46","parentHash":"0xf7beabc1d044e75572604c473a621cdcb22b76de6836834cb3ddc2bbe1c9c6bf","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000000","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","miner":"0x0000000000000000000000000000000000000000","difficulty":"0x0","totalDifficulty":"0x0","extraData":"0x","size":"0x0","gasLimit":"0x112a717a","gasUsed":"0x0","timestamp":"0x60c8386d","transactions":[],"uncles":[],"l1BlockNumber":"0xc0d3cf"}}
darkhorse-spb commented 8 months ago

@Sarahligbe FYI just found that this error is not happening in Arbitrum Nitro v2.1.3. Warning "couldn't find block with transactions. Reached 1891" is shown in logs, but node answers block information as expected.

Sarahligbe commented 8 months ago

@darkhorse-spb thanks that solved it for me

darkhorse-spb commented 8 months ago

@Sarahligbe unfortunately, you have to upgrade to v2.2.2 to be in sync after ArbOS 11 upgrade

Johnaverse commented 8 months ago

Still not fixed with v2.2.3 or we can say more worse. Now I am unable query all classic blocks through nitro.

ERROR[02-05|18:27:07.917] RPC method eth_getBlockByNumber crashed: decode called on LegacyTx)
goroutine 89930 [running]:
github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
        /workspace/go-ethereum/rpc/service.go:207 +0x89
panic({0x3e16600, 0x4e3a2a0})
        /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/ethereum/go-ethereum/core/types.(*LegacyTx).decode(...)
        /workspace/go-ethereum/core/types/tx_legacy.go:124
github.com/ethereum/go-ethereum/core/types.(*Transaction).decodeTyped(0xc0cdb502a0?, {0xc0882a7000, 0x178, 0x1000}, 0x1?)
        /workspace/go-ethereum/core/types/transaction.go:243 +0x287
github.com/ethereum/go-ethereum/core/types.(*Transaction).DecodeRLP(0x43e96c0?, 0xc0aa298f60?)
        /workspace/go-ethereum/core/types/transaction.go:175 +0x12e
github.com/ethereum/go-ethereum/rlp.decodeDecoder(0x43e96c0?, {0x4286820?, 0xc0aa298f60?, 0x15d42a5?})
        /workspace/go-ethereum/rlp/decode.go:542 +0x62
github.com/ethereum/go-ethereum/rlp.makeSimplePtrDecoder.func1(0x3da6b40?, {0x43e96c0?, 0xc08157eda0?, 0x1?})
        /workspace/go-ethereum/rlp/decode.go:465 +0x11d
github.com/ethereum/go-ethereum/rlp.decodeSliceElems(0xc0cdb502a0?, {0x3da6b40?, 0xc07d479540?, 0x1ae4bc2?}, 0xc000e12000)
        /workspace/go-ethereum/rlp/decode.go:329 +0x3ba
github.com/ethereum/go-ethereum/rlp.decodeListSlice(0xc0cdb502a0, {0x3da6b40?, 0xc07d479540?, 0x2?}, 0x182?)
        /workspace/go-ethereum/rlp/decode.go:306 +0x52f
github.com/ethereum/go-ethereum/rlp.makeListDecoder.func3(0x4167160?, {0x3da6b40?, 0xc07d479540?, 0x0?})
        /workspace/go-ethereum/rlp/decode.go:291 +0x2a
github.com/ethereum/go-ethereum/rlp.makeStructDecoder.func1(0xc0cdb502a0, {0x4167160?, 0xc07d479540?, 0x150313d?})
        /workspace/go-ethereum/rlp/decode.go:419 +0x519
github.com/ethereum/go-ethereum/rlp.(*Stream).Decode(0xc0cdb502a0?, {0x3d66e80?, 0xc07d479540?})
        /workspace/go-ethereum/rlp/decode.go:950 +0x222
github.com/ethereum/go-ethereum/rlp.DecodeBytes({0xc0152151e0, 0x182, 0x182}, {0x3d66e80, 0xc07d479540})
        /workspace/go-ethereum/rlp/decode.go:99 +0x13f
github.com/ethereum/go-ethereum/core/rawdb.ReadBody({0x7f81b82fd068?, 0xc000896438?}, {0x86, 0x54, 0x74, 0x41, 0xad, 0x93, 0x28, 0x70, ...}, ...)
        /workspace/go-ethereum/core/rawdb/accessors_chain.go:501 +0x9b
github.com/ethereum/go-ethereum/core/rawdb.ReadBlock({0x7f81b82fd068, 0xc000896438}, {0x86, 0x54, 0x74, 0x41, 0xad, 0x93, 0x28, 0x70, ...}, ...)
        /workspace/go-ethereum/core/rawdb/accessors_chain.go:797 +0xa5
github.com/ethereum/go-ethereum/core.(*BlockChain).GetBlock(0xc05d380c00, {0x86, 0x54, 0x74, 0x41, 0xad, 0x93, 0x28, 0x70, 0x2f, ...}, ...)
        /workspace/go-ethereum/core/blockchain_reader.go:164 +0xa5
github.com/ethereum/go-ethereum/core.(*BlockChain).GetBlockByNumber(0xc05d380c00, 0x4e6e328?)
        /workspace/go-ethereum/core/blockchain_reader.go:189 +0xd3
github.com/ethereum/go-ethereum/arbitrum.(*APIBackend).BlockByNumber(0xc07e21e6f0, {0x4e6e328?, 0xc07d479400?}, 0xc045a572a0?)
        /workspace/go-ethereum/arbitrum/apibackend.go:417 +0x16c
github.com/ethereum/go-ethereum/internal/ethapi.(*BlockChainAPI).arbClassicL1BlockNumber(0xc07e1d4740, {0x4e6e328, 0xc07d479400}, 0xc10ce121e0?)
        /workspace/go-ethereum/internal/ethapi/api.go:1534 +0x28a
github.com/ethereum/go-ethereum/internal/ethapi.(*BlockChainAPI).rpcMarshalBlock(0xc07e1d4740, {0x4e6e328, 0xc07d479400}, 0xc10ce121e0, 0x1, 0x62?)
        /workspace/go-ethereum/internal/ethapi/api.go:1550 +0x38d
github.com/ethereum/go-ethereum/internal/ethapi.(*BlockChainAPI).GetBlockByNumber(0xc07e1d4740, {0x4e6e328, 0xc07d479400}, 0x152dd48, 0x4?)
        /workspace/go-ethereum/internal/ethapi/api.go:806 +0x9d
reflect.Value.call({0xc0003c0770?, 0xc03fb89fd8?, 0x7f801068faf8?}, {0x4445f2b, 0x4}, {0xc0aa298f00, 0x4, 0x0?})
        /usr/local/go/src/reflect/value.go:586 +0xb07
reflect.Value.Call({0xc0003c0770?, 0xc03fb89fd8?, 0x16?}, {0xc0aa298f00?, 0x2?, 0x2?})
        /usr/local/go/src/reflect/value.go:370 +0xbc
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc07e972ba0, {0x4e6e328?, 0xc07d479400}, {0xc07a9ae6c0, 0x14}, {0xc080cbdd70, 0x2, 0x1593b77?})
        /workspace/go-ethereum/rpc/service.go:213 +0x3c5
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc0881bd900?, {0x4e6e328?, 0xc07d479400?}, 0xc0004b4f50, 0x2?, {0xc080cbdd70?, 0x150dbb0?, 0x400be20?})
        /workspace/go-ethereum/rpc/handler.go:565 +0x45
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc0ac732000, 0xc080cbdcb0, 0xc0004b4f50)
        /workspace/go-ethereum/rpc/handler.go:512 +0x239
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc0ac732000, 0xc080cbdd10?, 0xc0004b4f50)
        /workspace/go-ethereum/rpc/handler.go:470 +0x237
github.com/ethereum/go-ethereum/rpc.(*handler).handleNonBatchCall(0xc0ac732000, 0xc080cbdcb0, 0xc0004b4f50)
        /workspace/go-ethereum/rpc/handler.go:296 +0x1ae
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1.1(0x4e6e328?)
        /workspace/go-ethereum/rpc/handler.go:269 +0x27
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
        /workspace/go-ethereum/rpc/handler.go:387 +0xc5
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc
        /workspace/go-ethereum/rpc/handler.go:383 +0x8d

RPC Response

{
    "jsonrpc": "2.0",
    "id": 1,
    "error": {
        "code": -32603,
        "message": "method handler crashed"
    }
}
darkhorse-spb commented 8 months ago

Arbitrum Nitro v2.2.4 solved this issue for me, thanks to everyone involved!)