search

prysmaticlabs / prysm

Go implementation of Ethereum proof of stake
https://www.offchainlabs.com
GNU General Public License v3.0
3.46k stars 988 forks source link

Could not process attestation for fork choice #13765

Open YuXiaoCoder opened 6 months ago

YuXiaoCoder commented 6 months ago

Describe the bug

GETH Version: v1.13.14 Beacon Version: v5.0.1

My beacon service reports the following error log, which prevents it from processing the validator's request to submit a attestation and causes my validator to lose the attestation

time="2024-03-01 18:18:02" level=info msg="Peer summary" activePeers=70 inbound=0 outbound=70 prefix=p2p
time="2024-03-01 18:18:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xc9657f3ba3da committeeIndex=62 error="could not process attestation: epoch 266833 root 0xc9657f3ba3da7bb8d46e28930f7dccb95b67a1044270acb11f012dc210065c1f: not a checkpoint in forkchoice" prefix=blockchain slot=8538684 targetRoot=0xc9657f3ba3da
time="2024-03-01 18:18:13" level=info msg="Synced new block" block=0x404f0eac... epoch=266834 finalizedEpoch=266832 finalizedRoot=0xec60848d... prefix=blockchain slot=8538689
time="2024-03-01 18:18:13" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x7664e92e5ceb prefix=blockchain slot=8538689 syncBitsCount=351 txCount=182

Validator Version: v5.0.0

time="2024-03-18 18:18:19" level=warning msg="No attestations to aggregate" error="rpc error: code = NotFound desc = Could not find attestation for slot and committee in pool" prefix=client slot=8661089
time="2024-03-18 18:18:19" level=warning msg="No attestations to aggregate" error="rpc error: code = NotFound desc = Could not find attestation for slot and committee in pool" prefix=client slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb7b8021aac92 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x878bdfebc81c slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x82d596ec9998 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb8f68eb33846 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xabaa57161211 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x95dfaaa6ec69 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xa5869cf9b051 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x8becc89e8bc0 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xad76e26774f0 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb6e6779f652c slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x8d3bf38bded3 slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xa33e6644531c slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb9a101c981ab slot=8661089
time="2024-03-18 18:18:23" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb700f81cb509 slot=8661089
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb80855f5d0b4 slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x80b1d5198fb2 slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xa1cc977316d6 slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x878140b79a9c slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x9994e2f5d9c4 slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xa2b0974cad64 slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0xb800729d6e67 slot=8661090
time="2024-03-18 18:18:35" level=error msg="Could not request attestation to sign at slot" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=client pubkey=0x9431e5e1eecd slot=8661090

Has this worked before in a previous version?

There have been lost certificates before, but not as many as now.

🔬 Minimal Reproduction

No response

Error

No response

Platform(s)

No response

What version of Prysm are you running? (Which release)

No response

Anything else relevant (validator index / public key)?

No response

YuXiaoCoder commented 6 months ago 
time="2024-03-16 19:21:23" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
time="2024-03-16 19:21:23" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
time="2024-03-17 05:49:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=12 beaconBlockRoot=0x1ced048b0824 committeeIndex=28 error="could not process attestation: target epoch 270315 does not match current epoch 270317 or prev epoch 270316" prefix=blockchain slot=8650098 targetRoot=0x81a453a2689a
time="2024-03-17 09:45:57" level=warning msg="Could not process attestation for fork choice" aggregationCount=21 beaconBlockRoot=0x95e0c9f6c5f1 committeeIndex=61 error="could not process attestation: target epoch 270352 does not match current epoch 270354 or prev epoch 270353" prefix=blockchain slot=8651290 targetRoot=0x961c30a3eafc
time="2024-03-17 09:52:21" level=warning msg="Could not process attestation for fork choice" aggregationCount=26 beaconBlockRoot=0x909b757fb815 committeeIndex=8 error="could not process attestation: target epoch 270353 does not match current epoch 270355 or prev epoch 270354" prefix=blockchain slot=8651322 targetRoot=0x1336dbe36b51
time="2024-03-17 11:41:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0x998ff0d073c2 committeeIndex=49 error="could not process attestation: target epoch 270370 does not match current epoch 270372 or prev epoch 270371" prefix=blockchain slot=8651868 targetRoot=0xa4f53f9738fe
cyclopes2011 commented 6 months ago

UPDATE 2024-03-21 17:32:00: After watching for a while, I don't see that my attestations are being affected.

I have similar log output on my beacon-chain running the same software Prysm (5.0.1) and Geth (1.13.14). I have two validators running (25044) and (392415) on Ubuntu 22.04.4 LTS in a virtual machine.

[2024-03-21 16:13:09] WARN blockchain: Could not process attestation for fork choice aggregationCount=478 beaconBlockRoot=0x5aa5f3ffc30f committeeIndex=46 error=could not process attestation: target epoch 271425 does not match current epoch 271427 or prev epoch 271426 slot=8685627 targetRoot=0xd02c6023e8de [2024-03-21 16:13:09] WARN blockchain: Could not process attestation for fork choice aggregationCount=477 beaconBlockRoot=0x5aa5f3ffc30f committeeIndex=27 error=could not process attestation: target epoch 271425 does not match current epoch 271427 or prev epoch 271426 slot=8685627 targetRoot=0xd02c6023e8de [2024-03-21 16:13:09] WARN blockchain: Could not process attestation for fork choice aggregationCount=476 beaconBlockRoot=0x5aa5f3ffc30f committeeIndex=30 error=could not process attestation: target epoch 271425 does not match current epoch 271427 or prev epoch 271426 slot=8685627 targetRoot=0xd02c6023e8de [2024-03-21 16:13:09] WARN blockchain: Could not process attestation for fork choice aggregationCount=472 beaconBlockRoot=0xad7632c4ef88 committeeIndex=42 error=could not process attestation: target epoch 271425 does not match current epoch 271427 or prev epoch 271426 slot=8685622 targetRoot=0xd02c6023e8de (removed) [2024-03-21 16:25:57] WARN blockchain: Could not process attestation for fork choice aggregationCount=473 beaconBlockRoot=0x03e8d3b4f8a3 committeeIndex=50 error=could not process attestation: target epoch 271427 does not match current epoch 271429 or prev epoch 271428 slot=8685690 targetRoot=0x9525a83c89e6 (removed) [2024-03-21 16:32:21] WARN blockchain: Could not process attestation for fork choice aggregationCount=474 beaconBlockRoot=0x1adbc63bca84 committeeIndex=56 error=could not process attestation: target epoch 271428 does not match current epoch 271430 or prev epoch 271429 slot=8685722 targetRoot=0xb9bb48a530ff (removed) [2024-03-21 16:45:09] WARN blockchain: Could not process attestation for fork choice aggregationCount=92 beaconBlockRoot=0x59b40efc46a7 committeeIndex=9 error=could not process attestation: target epoch 271430 does not match current epoch 271432 or prev epoch 271431 slot=8685788 targetRoot=0x063a32f9d74a

*Note timestamps and gaps.

Also, I had shutdown and restarted my validators prior to these logs in order to increase hard drive space on the virtual machine running these. You will see some missed attestations while things sync'ed again. I noticed the messages when I was monitoring the logs to ensure everything was okay. I do not know how long the messages have been occurring. I didn't notice them during the Dencun fork but was only watching for about an hour.

I am only an end-user so I can't offer any additional insight. Please let me know if you need additional information to help with this issue.

YuXiaoCoder commented 6 months ago

@terencechain @prestonvanloon I'm an end user and I've been disturbed for a long time, can you help me?

cyclopes2011 commented 6 months ago

@YuXiaoCoder Maybe try the Discord channel: Prysm Ethereum Client