search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.15k stars 1.13k forks source link

EngineError when requesting block at slot 43200 on Zhejiang testnet #6976

Closed peterbitfly closed 1 year ago

peterbitfly commented 1 year ago

System information

Erigon version: erigon version 2.39.0-dev-f01e3d17

OS & Version: Linux

Commit hash:

Erigon Command (with flags/config):

/usr/bin/erigon
  --datadir=/home/ubuntu/eldata
  --networkid=1337803
  --http
  --http.api eth,erigon,engine,debug,trace,txpool
  --http.port 8545
  --http.addr 0.0.0.0
  --http.vhosts *
  --http.corsdomain *
  --ws
  --private.api.addr=127.0.0.1:9999
  --externalcl
  --rpc.batch.limit 10000000
  --rpc.returndata.limit 1000000000
  --authrpc.port=8551
  --authrpc.addr=127.0.0.1
  --authrpc.jwtsecret=/home/ubuntu/jwtsecret
  --batchSize=32m

--bootnodes=enode://691c66d0ce351633b2ef8b4e4ef7db9966915ca0937415bd2b408df22923f274873b4d4438929e029a13a680140223dcf701cabe22df7d8870044321022dfefa@64.225.78.1:30303,enode://89347b9461727ee1849256d78e84d5c86cc3b4c6c5347650093982b726d71f3d08027e280b399b7b6604ceeda863283dcfe1a01e93728b4883114e9f8c7cc8ef@146.190.238.212:30303,enode://c2892072efe247f21ed7ebea6637ade38512a0ae7c5cffa1bf0786d5e3be1e7f40ff71252a21b36aa9de54e49edbcfc6962a98032adadfa29c8524262e484ad3@165.232.84.160:30303,enode://71e862580d3177a99e9837bd9e9c13c83bde63d3dba1d5cea18e89eb2a17786bbd47a8e7ae690e4d29763b55c205af13965efcaf6105d58e118a5a8ed2b0f6d0@68.183.13.170:30303,enode://2f6cf7f774e4507e7c1b70815f9c0ccd6515ee1170c991ce3137002c6ba9c671af38920f5b8ab8a215b62b3b50388030548f1d826cb6c2b30c0f59472804a045@161.35.147.98:30303

Concensus Layer:

Concensus Layer Command (with flags/config):

/usr/bin/lighthouse
  --datadir /home/ubuntu/beacon
  --testnet-dir=/home/ubuntu/custom_config_data
  bn
  --prune-payloads false
  --disable-enr-auto-update
  --port=9000
  --discovery-port=9000
  --eth1
  --boot-nodes=enr:-Iq4QMCTfIMXnow27baRUb35Q8iiFHSIDBJh6hQM5Axohhf4b6Kr_cOCu0htQ5WvVqKvFgY28893DHAg8gnBAXsAVqmGAX53x8JggmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk,enr:-Ly4QOS00hvPDddEcCpwA1cMykWNdJUK50AjbRgbLZ9FLPyBa78i0NwsQZLSV67elpJU71L1Pt9yqVmE1C6XeSI-LV8Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhEDhTgGJc2VjcDI1NmsxoQIgMUMFvJGlr8dI1TEQy-K78u2TJE2rWvah9nGqLQCEGohzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA,enr:-MK4QMlRAwM7E8YBo6fqP7M2IWrjFHP35uC4pWIttUioZWOiaTl5zgZF2OwSxswTQwpiVCnj4n56bhy4NJVHSe682VWGAYYDHkp4h2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhJK-7tSJc2VjcDI1NmsxoQLDq7LlsXIXAoJXPt7rqf6CES1Q40xPw2yW0RQ-Ly5S1YhzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA,enr:-MS4QCgiQisRxtzXKlBqq_LN1CRUSGIpDKO4e2hLQsffp0BrC3A7-8F6kxHYtATnzcrsVOr8gnwmBnHYTFvE9UmT-0EHh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhKXoVKCJc2VjcDI1NmsxoQK6J-uvOXMf44iIlilx1uPWGRrrTntjLEFR2u-lHcHofIhzeW5jbmV0c4gAAAAAAAAAAIN0Y3CCIyiDdWRwgiMo,enr:-LK4QOQd-elgl_-dcSoUyHDbxBFNgQ687lzcKJiSBtpCyPQ0DinWSd2PKdJ4FHMkVLWD-oOquXPKSMtyoKpI0-Wo_38Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhES3DaqJc2VjcDI1NmsxoQNIf37JZx-Lc8pnfDwURcHUqLbIEZ1RoxjZuBRtEODseYN0Y3CCIyiDdWRwgiMo,enr:-KG4QLNORYXUK76RPDI4rIVAqX__zSkc5AqMcwAketVzN9YNE8FHSu1im3qJTIeuwqI5JN5SPVsiX7L9nWXgWLRUf6sDhGV0aDKQ7ijXswAAAHJGBQAAAAAAAIJpZIJ2NIJpcIShI5NiiXNlY3AyNTZrMaECpA_KefrVAueFWiLLDZKQPPVOxMuxGogPrI474FaS-x2DdGNwgiMog3VkcIIjKA
  --http
  --http-address=0.0.0.0
  --http-port=4000
  --listen-address=0.0.0.0
  --target-peers=100
  --http-allow-sync-stalled
  --execution-endpoint=http://127.0.0.1:8551
  --jwt-secrets=/home/ubuntu/jwtsecret
  --count-unrealized
  --slots-per-restore-point=32

Chain/Network: Zhejiang

When requesting the block body at slot 80646 via the beacon api curl -v http://localhost:4000/eth/v1/beacon/blocks/43200 we get the following error:

{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0x1c5f2c9cbaa2e7c7abdb6fa73e7562beea21ce6af8a0e52a5634f6283db07418, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

This is also the case for all successive blocks up to block 91765 after which the api returns again valid results.

We are using lh version v3.5.0 together with erigon (latest devel). According to the lh devs (see https://github.com/sigp/lighthouse/issues/4035) this is an issue with erigon.

michaelsproul commented 1 year ago

Lighthouse uses eth_getBlockByHash currently to reconstruct payloads, and is expecting all Capella blocks to have a non-null withdrawals field.

We will soon switch to using the new PayloadBodies methods, which will alleviate the issue on our side. The underlying Erigon issue with eth_getBlockByHash should probably still be resolved though.

hexoscott commented 1 year ago

Hi @peterbitfly, there was a commit merged prior to version 2.39.0 (#6893) that could well fix this issue. It would require a wipe and resync to be applied correctly though. Just checking if you did the re-sync before I investigate further, thanks

peterbitfly commented 1 year ago

Hello, we did a full resync with erigon version 2.40.0-stable-38fb38a9 but the issue still persists.

hexoscott commented 1 year ago

Thanks @peterbitfly, I'll take a further look

peterbitfly commented 1 year ago

Hello, we are now encountering the same issue on goerli with erigon v2.42.0 and lh v4.0.1:

curl localhost:4000/eth/v1/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

Currently this bug makes retrieval of historical blocks after the shanghai fork impossible.

michaelsproul commented 1 year ago

@peterbitfly We could also work around this on the LH side if we merge this PR: https://github.com/sigp/lighthouse/pull/4102. I'll update it for unstable this weekend

You could also work around this by syncing LH through the merge (not from current finalized) with --prune-payloads false

Giulio2002 commented 1 year ago

Lighthouse uses eth_getBlockByHash currently to reconstruct payloads, and is expecting all Capella blocks to have a non-null withdrawals field.

We will soon switch to using the new PayloadBodies methods, which will alleviate the issue on our side. The underlying Erigon issue with eth_getBlockByHash should probably still be resolved though.

what you are saying is that you guys are expecting [] instead of null, is that correct?

michaelsproul commented 1 year ago

Yeah, if it's a Capella execution payload the withdrawals should be [] when they're empty, not null or missing

Giulio2002 commented 1 year ago

Ok, I will fix that; however, you should consider properly handling it on your side as well (the null case that is). Just a nice thing to have :).

yperbasis commented 1 year ago

Hello, we are now encountering the same issue on goerli with erigon v2.42.0 and lh v4.0.1:

curl localhost:4000/eth/v1/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

Currently this bug makes retrieval of historical blocks after the shanghai fork impossible.

@peterbitfly But https://goerli.etherscan.io/block/5193760 predates Shanghai and so shouldn't have withdrawals. Do you observe the issue for Shanghai blocks?

guybrush commented 1 year ago

@peterbitfly But https://goerli.etherscan.io/block/5193760 predates Shanghai and so shouldn't have withdrawals. Do you observe the issue for Shanghai blocks?

https://goerli.beaconcha.in/slot/5193760 does not predate Shanghai: CAPELLA_FORK_EPOCH: 162304

we get the error from the beaconnode-api

curl localhost:4000/eth/v1/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

curl localhost:4000/eth/v2/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByHash","params":["0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d",false],"id":67}' localhost:8545
{"jsonrpc":"2.0","id":67,"result":{"baseFeePerGas":"0xe3a9b59f","difficulty":"0x0","extraData":"0xd883010b03846765746888676f312e32302e32856c696e7578","gasLimit":"0x1c9c380","gasUsed":"0xb4f86f","hash":"0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d","logsBloom":"0x10062000040944ac011800571c1805d054114000c809144100a06050d415b048002e100004381001028058e021840887500a050a024259b0185816860036620002817198944029484610022b62a0b602000700440580b000a8708108c2a0012142412401078c8c43c26c80ac2ab0a9000110003200025383182b055aac080049071a23748841843b00020440d810094504053489142604140001112452c3800002083003961605390d826082040a06a1b0449a14244d04c08100156c01060083002041325018a40f0402940145124410554c532420994800000638581806a5e25857182800400445e0140110a1e080a18000002808b90948018298000a8278c0","miner":"0x000095e79eac4d76aab57cb2c1f091d553b36ca0","mixHash":"0x0573d7cc2b3385a115860a5cea3f9f8cdd1dfc57dc148415d1aca59650ceb18a","nonce":"0x0000000000000000","number":"0x841508","parentHash":"0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599","receiptsRoot":"0x35862b7877512cbd009455089d2d1b396cef7279b639d982961d84e58c2c1c47","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x36390","stateRoot":"0xe5c12ba85ee8852a903361776d3212f02416fb64ac7ad467b0acec19b053b8a5","timestamp":"0x6410f5e0","totalDifficulty":"0xa4a470","transactions":["0x339dbca7d7bea4e12e2051bd844814415b427cb6a94dfa701ac32510f1db076f","0x1ab8b9775ca267fc64accaa78290e55dc5b801475e5e722fe2822815f5809499","0x9234b59f482f1f4df3da90766c93a0d8b97be59fb8a972954c02989e0879224e","0x5d5768ab15eaab6bcfbda8a7858327f3b45e7bc0e504cb6720135b971ab3cfea","0x0eae7d068c2d0ecbcd99653e80d5b38f9fc0ee9df2af824f1c45ab3393d99913","0xd88429518f37d55da414e8db551832c780723da5635c008f9b9802e08094388e","0x56e8285fa24d90aaf35f2d3d83c2411cdbc28d3f190153c2b5681df4ad5b9e1a","0xd30638813dac00827329354a2d4547e30c4076e0f60b448fa5b51a09fc119f5a","0xe78790a6c80057023a8b1d0b8a9b6aa208abbcbd099b5ebc33226aac7fc88fa1","0xff547c8c9851208a531aa46202f50ee56011845f84339576ca122e683d350386","0x21fbfed46ccbbf5268b82704004fb6a0e1f0b8470c0b874aaa476ee43366b2d5","0x22bc8078c3df62da7a14ad5b03736f8e178d4e93c396478767d5ed550cd65a12","0x541616056663f65bdda60acec719db727b7167b1c8c374534783c77111c8e312","0x7002e072d67eb2f6ceecac146330ca0e7cf7a1a7f3cbd88530d72c42690dd03c","0xf84f69287c643cb94cd47127869d43e534d618ad409647c9eb2f4e242afb6fef","0x221fe88f84b3e164b0613a1d245247c86e453844f6f55516b9d6453713592bf1","0xb48a427f2616a33e7fc82b3cd76c0feee2fcc4769fa7ff20092627fc1fe0bdee","0xd993948626c51194e0b421739d1902b0683f3808fe93e9a708b8e82f6599ca07","0xd94d2e33934ada23d68ff201fd5ee07189924117662351b91cf48ac6fb3a0df5","0xa008024264cd3a2a7c06af44c1f62e0f11d025f72d03def6073ae15e6f39437c","0xa704e3f4808656f78aefd4df52620264a1c509f71558e9f4a669aba95fe8d6cc","0x1c1cd78589e65b48f5dfbc999599eed758b8e16b12acb7d65dffa6a9eb1ad325","0xffe5a97aee8822c8ce6a49b27c9d1f8860eb2f76749535310b82f7c5d9a9854d","0xc35495669e9c94a009c40106b58d34a7efa0eb44b8d94c64b7fafcb772249f13","0xbbb5ff0956c38718ddd49fe0269594e1420fb6649f640ffe5afeb02a14f973b9","0x24ef38c89daa772a0e88903004258e7f6a192c190bec61d4c7fb713d8190086d","0xcacc5bcc63ba93b8491cad2c387b8e72622cf4a769faf2aec50dc64b3d7a2afb","0xb363e69abb51913de21365ebf523891488c164cf431a16fdefa1c472041d9846","0xb4764d71b89b4a7a4b3c103d74dd3f1cdb6541285422926da6b437571e9be136","0xc789bdbdf26e9e275f866d77b14a7030c2639088efabf1636c4ca52744b3dda7","0xc0e1ced729d7b4ffdfc1ed1d9b903cc5f7ff996c554f625a685ff660d15621ad","0xb9339fd5b6d275fdc31bd4ee6f9eb7cdc1033d18df78250e85cfa81e39ce2fc0","0x032e447a579546f0a88975ddd3d181c31752947572ab461dff53b9a95bd631b4","0x7b0d55252b51c93be26c944d456a74e48625121c569ab3c57d439f9d0bcf46f2","0x048f7b4d607f9c97c100560fc0e85cab82e0061631d2b5926c21e7e518f8a919","0x0c4800a409f0b3b85d741b9d1a8d58fe34e091c4e3312c31b43c9205228aef5c","0x012d9532ea2f555460a25d49c07b2902b375df9b7d8867626b05dd568027eb13","0xdd11b271d1f70aefba30d667b08fe63998028c88e2ce90c09cfb65f9636ac50a","0x305813f2494b4ede4538f2b3951451e73e1126a7d01324ba3e0c6ce042eff349","0x9efd36cc8a7c5828ce450d92eb58804706d31a5d8d8a75a946a9f1fa98a847f3","0xbfd129fa52d7e46ad31dcda35edcaaaab33b3210289bd532c07f8d60e37f6e0b","0xa6f996fb9413c6f21b65f14bef1e508fdab68861790872c078da46f93b09eae7","0x6df0388cb234d644704e7f9badf11a5c463e736a33fce8d4d4ff69f8cbb1a461","0x9f4a6c11bb4f221b3dd49b948ea4af62ff12196700224352a5f33b849f617022","0x89968267c8bf7c479705ed83c1481cb4fa7c5284305ca762cc8215a6572a9f2d","0xfc795f51ca1bf410f74aff45d6cd5d4fcb30f684d5c1fc4329af04d96eff0e99","0x8e625ab0d0e0a749aed740a7ec7de1e092cd33ddb58f26139bfa01f363e1db95","0xdc786e600f55206c4c4175b5827045891da6bfe81ead6ae25ed6b71137d5dc43","0x98b0cda5e24b6c390902a6e8f2456e18c314dfbba916e327ad30160e9d043a92","0xb2d4f3142d99ae01a8afb874ce34f4cbfa88ad88284f457303ed4a0c402db861","0x359dfd55766d430470c27928f4e061d26453e3132446d22507931b53c74bfa8e","0x2a57f15189d09a504d44c0436ca246e6b9fa87dd97b107898a536a314fdc87bc","0x69edfa771121c67ff95706135c6076a978032ad9102d79e03de4eb6c8c6c1017","0xed63a06251a911bdc212d8cf0c7819f62ddfdea4db4b90c1fab09e587be6aa1a","0x5c7e0aa84bc6a9df143f64182b03d5e58473beec91cd9af54d6d02566dfa65cc","0x72845022bf6c5a75ea3077b60099e74e9dd52a6078d6f4108c9ab2be467d0f9a","0xdd0f5695780f92427ffcf74bb2589cb4c8457e1c0bcf3060c41ecfbed5a95d8a","0x5055598ba3c339b95652813488cddb6e77bb8af4064f22abb4e8f822eaea666d","0x325de12ece40c1ba5966c287baa4026bccf0d3c75d0524a5c4a264103c5f5063","0xa8b20f643544f35de2b36d7b2fe0e102019b845a187d1ccbbecaf3fef03ccce0","0x64aaba902b903f3236cddd86c3665b5e5b2fefe80b17a0b01cbdde4cf37ec273"],"transactionsRoot":"0x2ee5e31f1db763d474a470058c8a3d8d6ab3fc0eebab3519d3bfb68ee4c034c6","uncles":[],"withdrawalsRoot":"0xab5edaeaec5c53fc12822210ece094b8e114bcc4780b5cb249deb44693ae4f57"}}