search

ChainSafe / lodestar

🌟 TypeScript Implementation of Ethereum Consensus
https://lodestar.chainsafe.io
Apache License 2.0
1.16k stars 286 forks source link

Error updating eth1 chain cache message=Invalid length index (kintsugi-lodestar-geth/nethermind) #3597

Closed SeaMonkey82 closed 2 years ago

SeaMonkey82 commented 2 years ago

Describe the bug

Flood of this error Error updating eth1 chain cache message=Invalid length index

Expected behavior

non-error output

Steps to Reproduce

Sync kintsugi-lodestar-geth or kintsugi-lodestar-nethermind (maybe all EL clients?)

Screenshots Output

Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)

Desktop (please complete the following information):

philknows commented 2 years ago

Interesting... haven't seen this one before @g11tech.

g11tech commented 2 years ago

@SeaMonkey82 is this coming now/reproducible? couldn't find this in the log of the recently updated kintsugi nodes

@philknows seems like deposit data is not correct, not a kintsugi issue though, but could be because of bad blocks that are being pushed into network as part of robustness/stress testing.

SeaMonkey82 commented 2 years ago

edit Different error now. 

➜  lodestar git:(master) ./lodestar --version
🌟 Lodestar: TypeScript Implementation of the Ethereum 2.0 Beacon Chain.
  * Version: v0.32.0/master/+85/b4facd38 (git)
Jan-14 07:16:08.737 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=208555 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":208555,"latestPermissibleSlot":208554}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
g11tech commented 2 years ago

this error looks legit, as the attestation arrive for a future slot it seems as we currently reject future slot attestation/arriving early than expected.

SeaMonkey82 commented 2 years ago 
Jan-14 14:24:03.405 [ETH1]            error: Error updating eth1 chain cache message=Invalid length index
Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)
g11tech commented 2 years ago 
Jan-14 14:24:03.405 [ETH1]            error: Error updating eth1 chain cache message=Invalid length index
Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)

Hi @SeaMonkey82 Since am not able to see this on our nodes. Could you help me with these? :slightly_smiling_face: which commit hash for lodestar, geth/nethermind (or better yet docker images) ? Is it a random occurrence or do you see this consistently ? could you include a bit more of log above and below this error?

SeaMonkey82 commented 2 years ago 
➜  lodestar git:(master) ./lodestar --version
🌟 Lodestar: TypeScript Implementation of the Ethereum 2.0 Beacon Chain.
  * Version: v0.32.0/master/+85/b4facd38 (git)
➜  lodestar git:(master) geth-kintsugi version
Geth
Version: 1.10.16-unstable
Git Commit: 63a89820ca9721ef2256859391db5a5920faf16c
Git Commit Date: 20220110
Architecture: amd64
Go Version: go1.17.5
Operating System: linux
➜  Nethermind.Runner git:(b565ffba0) ✗ dotnet run -c Release -- --version                                                                                               
2022-01-15 08-16-34.7059|Nethermind starting initialization.
2022-01-15 08-16-34.7367|Loading embedded plugins
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
Resolved executing directory as /home/seamonkey/Downloads/merge-testnets/kintsugi/nethermind/src/Nethermind/Nethermind.Runner/bin/Release/net6.0.
2022-01-15 08-16-34.7437|Loading 7 assemblies from /home/seamonkey/Downloads/merge-testnets/kintsugi/nethermind/src/Nethermind/Nethermind.Runner/bin/Release/net6.0/plugins
2022-01-15 08-16-34.7438|Loading assembly Nethermind.HealthChecks
2022-01-15 08-16-34.7450|  Found plugin type Nethermind.HealthChecks
2022-01-15 08-16-34.7450|Loading assembly Nethermind.Merge.Plugin
2022-01-15 08-16-34.7465|  Found plugin type Nethermind.Merge.Plugin
2022-01-15 08-16-34.7465|Loading assembly Nethermind.Consensus.AuRa
2022-01-15 08-16-34.7488|Loading assembly Nethermind.Api
2022-01-15 08-16-34.7491|Loading assembly Nethermind.Mev
2022-01-15 08-16-34.7504|  Found plugin type Nethermind.Mev
2022-01-15 08-16-34.7504|Loading assembly Nethermind.Consensus.Ethash
2022-01-15 08-16-34.7505|Loading assembly Nethermind.Consensus.Clique

Nethermind/v1.11.7-409-b565ffba0-20220115/X64-Linux/6.0.1

Occurs consistently. Here is the log from lodestar-geth. lodestar-nethermind looks similar.

Jan-15 08:13:44.366 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.367 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.367 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.519 [ETH1]            error: Error updating eth1 chain cache message=Invalid length index
Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)
Jan-15 08:13:44.659 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.659 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.660 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.797 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
g11tech commented 2 years ago

@SeaMonkey82 thanks! :slightly_smiling_face: I will try to repro at my end. I am not that worried about invalid attestation as an attestation could potentially arrive earlier than block causing this error (but I will still try validating if this is the case). In future lodestar could queue this attestation up to process later.

However I am more curious about the reasons behind Invalid length index which seems to be occurring while saving deposit. :thinking:

SeaMonkey82 commented 2 years ago

Update: This error was caused by including --network mainnet in my validator launch command. I tested by removing it and all three lodestar instances (with geth, nethermind, and besu) stopped seeing the error. I don't have a good reason for why that flag was there, but curiously enough, the only thing it broke was block proposal. Attestations worked fine.

Leaving this issue open in case of any further discussion or action. It seems to me that --network mainnet should conflict with --paramsFile="../config.yaml" (kintsugi in this case) and automatically exit with a message describing the conflict.

g11tech commented 2 years ago

Update: This error was caused by including --network mainnet in my validator launch command. I tested by removing it and all three lodestar instances (with geth, nethermind, and besu) stopped seeing the error. I don't have a good reason for why that flag was there, but curiously enough, the only thing it broke was block proposal. Attestations worked fine.

Leaving this issue open in case of any further discussion or action. It seems to me that --network mainnet should conflict with --paramsFile="../config.yaml" (kintsugi in this case) and automatically exit with a message describing the conflict.

Thanks @SeaMonkey82 . Yes, the validator does validate the config it fetches from beacon against its own config.yaml. will check it out why it hasn't happened here.

SeaMonkey82 commented 2 years ago

I guess the fact that these messages stopped after restarting my validators without --network mainnet was a coincidence. All three beacon nodes are doing it again.

blampe commented 2 years ago

I also saw this on my first run of chainsafe/lodestar:v0.33.0 using Prater. Command:

./node_modules/.bin/lodestar beacon --network=prater --metrics.enabled=true --metrics.serverPort=8008 --rootDir=/data --network.maxPeers=30 --eth1.providerUrls http://nethermind:8545 http://erigon:8545

This was reproducible after wiping the data directory and re-running.

I suspect there's a bad interaction between Lodestar/Nethermind, because after switching the order of eth1 providers (presumably to give Erigon priority over Nethermind) the problem went away. (Although I then encounter #3550 and something to do with BLOCK_ERROR_PARENT_UNKNOWN.)

FWIW, I have Nethermind using their recommended configuration for validators: --Pruning.Enabled=true --Sync.DownloadBodiesInFastSync=true --Sync.DownloadReceiptsInFastSync=true --Sync.AncientBodiesBarrier=11052984 --Sync.AncientReceiptsBarrier=11052984.

g11tech commented 2 years ago

thank you for the update! so it seems that it has nothing to do with the kintsugi/bellatrix but deposit tracker interaction with the EL. let me try to replicate using the nethermind master

g11tech commented 2 years ago

I also saw this on my first run of chainsafe/lodestar:v0.33.0 using Prater. Command:

./node_modules/.bin/lodestar beacon --network=prater --metrics.enabled=true --metrics.serverPort=8008 --rootDir=/data --network.maxPeers=30 --eth1.providerUrls http://nethermind:8545 http://erigon:8545

This was reproducible after wiping the data directory and re-running.

I suspect there's a bad interaction between Lodestar/Nethermind, because after switching the order of eth1 providers (presumably to give Erigon priority over Nethermind) the problem went away. (Although I then encounter #3550 and something to do with BLOCK_ERROR_PARENT_UNKNOWN.)

FWIW, I have Nethermind using their recommended configuration for validators: --Pruning.Enabled=true --Sync.DownloadBodiesInFastSync=true --Sync.DownloadReceiptsInFastSync=true --Sync.AncientBodiesBarrier=11052984 --Sync.AncientReceiptsBarrier=11052984.

Could you provide nethermind branch + commit /docker image details along with the (exact if possible) command that you used to spin up nethermind?

UPDATE: I was able to run lodestar on prater with eth1 as nethermind without any issue:

an-24 18:10:42.002 []                 info: Synced - slot: 2211653 - head: 2211653 0xa8a9…6162 - finalized: 0xcf0f…1522:69112 - peers: 9
Jan-24 18:10:54.001 []                 info: Synced - slot: 2211654 - head: 2211654 0xd4ff…003b - finalized: 0xcf0f…1522:69112 - peers: 10
Jan-24 18:11:06.001 []                 info: Synced - slot: 2211655 - head: 2211655 0x7f1f…6a3c - finalized: 0xcf0f…1522:69112 - peers: 10
Jan-24 18:11:17.999 []                 info: Synced - slot: 2211656 - head: 2211656 0x645b…7011 - finalized: 0xcf0f…1522:69112 - peers: 13
Jan-24 18:11:18.001 []                 info: Synced - slot: 2211656 - head: 2211656 0x645b…7011 - finalized: 0xcf0f…1522:69112 - peers: 13
Jan-24 18:11:30.000 []                 info: Synced - slot: 2211657 - head: 2211657 0x322a…3dc4 - finalized: 0xcf0f…1522:69112 - peers: 18
Jan-24 18:11:42.001 []                 info: Synced - slot: 2211658 - head: 2211658 0x8d9b…eb85 - finalized: 0xcf0f…1522:69112 - peers: 18
Jan-24 18:11:54.000 []                 info: Synced - slot: 2211659 - head: 2211659 0xe843…5a02 - finalized: 0xcf0f…1522:69112 - peers: 19
Jan-24 18:12:0

nethermind args: --config goerli --datadir /data/nethermind --JsonRpc.Enabled=true --JsonRpc.EnabledModules=net,eth,consensus,engine,subscribe,web3 --JsonRpc.Port=8545 --JsonRpc.WebSocketsPort=8546 --JsonRpc.Host=0.0.0.0

g11tech commented 2 years ago

@SeaMonkey82 @blampe

Et voila! This annoying error is now RESOLVED!!! :partying_face:
@SeaMonkey82 you might not be setting --eth1.depositContractDeployBlock argument which for kintsugi should be set to --eth1.depositContractDeployBlock 0. This causes the lodestar deposit tracker to fetch skipped data, from which deposit root can't be derived and hence the error!

And you @blampe seems to have mixed up the networks on CL and EL side, so this error could again comeup because of wrong/unexpected feed of the deposit roots resulting again to the scenario described above!

Anyway, now one doesn't need to worry about configuration, kintsugi configuration can now automatically be loaded with --network kintsugi (you still have to specify eth1... and execution... params to interface the EL)

cc: @philknows "repro-ing" this error was harder than solving it :rofl:

special thanks to Nethermind's @marekM25 and his teammate @AntiD2ta for helping repro this error!