search

prysmaticlabs / prysm

Go implementation of Ethereum proof of stake
https://www.offchainlabs.com/
GNU General Public License v3.0
3.41k stars 969 forks source link

holesky p2p blob error: could not validate blob data availability: context deadline waiting for blob sidecars: context deadline exceeded #13741

Closed SimonSMH1015 closed 4 months ago

SimonSMH1015 commented 4 months ago

Describe the bug

holesky prysm logs contains

time="2024-03-14 00:32:08" level=error msg="Could not handle p2p pubsub" error="could not validate blob data availability: context deadline waiting for blob sidecars: context deadline exceeded" prefix=sync topic="/eth2/69ae0e99/beacon_block/ssz_snappy"

And then my node stop getitng latest beacon blocks in time. After restarting the node, this error shows again.

Has this worked before in a previous version?

Never met this berofe. I updated 5.0.1 from 5.0.0 three days ago.

🔬 Minimal Reproduction

the beacon node command is

beacon-chain --datadir /workdir/consensus --min-sync-peers 1 --rpc-... --execution-endpoint http://127.0.0.1:8551 --grpc-gateway-host ...--p2p-local-ip ... --p2p-host-ip ... --p2p-tcp-port 30310 --p2p-udp-port 30311 --grpc-gateway-port 3500 --jwt-secret /workdir/execution/jwtsecret --genesis-beacon-api-url http:etc --checkpoint-sync-url http://etc --peer /ip4/.../tcp/30310/p2p/16Uiu2HAm5DJETH8phZWd6aRmyyFc2b7jW4PqojaWnaD3WmeHtGhe,/ip4/...tcp/30310/p2p/16Uiu2HAmABx4mxt1B7ykpfSZjpCww1bHeY9hAGB1GgWqt2q5YyrJ --p2p-priv-key /workdir/consensus/p2p-priv-key --holesky --disable-eip-4881 --accept-terms-of-use --enable-debug-rpc-endpoints --subscribe-all-subnets --p2p-allowlist private

Error

="2024-03-14 00:32:08" level=error msg="Could not handle p2p pubsub" error="could not validate blob data availability: context deadline waiting for blob sidecars: context deadline exceeded" prefix=sync topic="/eth2/69ae0e99/beacon_block/ssz_snappy"

blobError

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

5.0.1

Anything else relevant (validator index / public key)?

I also have 5000 validators running remotely connecting to this node.

terencechain commented 4 months ago

Hi @SimonSMH1015, could you please share additional logs from around the time of that error (approximately 2 minutes before and after)? You can send them to us on Discord. Thank you!

prestonvanloon commented 4 months ago

@terencechain did you receive any additional information on this report?

YuXiaoCoder commented 4 months ago

I'm having the same problem.

time="2024-03-26 20:45:10" level=info msg="Peer summary" activePeers=77 inbound=0 outbound=77 prefix=p2p
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_62/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_44/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_31/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_45/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_28/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_48/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_23/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_11/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_6/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_13/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_27/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_33/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_51/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_40/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_19/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_63/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_32/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_20/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_34/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_50/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_2/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_4/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_14/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_54/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/6a95a1a9/beacon_attestation_21/ssz_snappy"
time="2024-03-26 20:45:11" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=300
time="2024-03-26 20:45:11" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=300
time="2024-03-26 20:45:11" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=300
time="2024-03-26 20:45:11" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=300
time="2024-03-26 20:45:13" level=info msg="Synced new block" block=0xd72113ad... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719424
time="2024-03-26 20:45:13" level=info msg="Finished applying state transition" attestations=128 deposits=16 payloadHash=0x7c9ab7ad927e prefix=blockchain slot=8719424 syncBitsCount=509 txCount=173
time="2024-03-26 20:45:24" level=info msg="Synced new block" block=0xb04ff7a5... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719425
time="2024-03-26 20:45:24" level=info msg="Finished applying state transition" attestations=128 deposits=16 payloadHash=0x7542fdb48e16 prefix=blockchain slot=8719425 syncBitsCount=468 txCount=174 voluntaryExits=1
time="2024-03-26 20:45:37" level=info msg="Synced new block" block=0xaa9b0980... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719426
time="2024-03-26 20:45:37" level=info msg="Finished applying state transition" attestations=98 deposits=16 payloadHash=0xd631cc80a163 prefix=blockchain slot=8719426 syncBitsCount=508 txCount=179
time="2024-03-26 20:45:49" level=info msg="Synced new block" block=0x99f0ee9e... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719427
time="2024-03-26 20:45:49" level=info msg="Finished applying state transition" attestations=128 deposits=16 payloadHash=0x15bdedf7d5cf prefix=blockchain slot=8719427 syncBitsCount=508 txCount=168
time="2024-03-26 20:46:00" level=info msg="Synced new block" block=0xe4a20577... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719428
time="2024-03-26 20:46:00" level=info msg="Finished applying state transition" attestations=128 deposits=16 payloadHash=0xd4645d9d1e15 prefix=blockchain slot=8719428 syncBitsCount=507 txCount=194
time="2024-03-26 20:46:10" level=info msg="Peer summary" activePeers=77 inbound=0 outbound=77 prefix=p2p
time="2024-03-26 20:46:13" level=info msg="Synced new block" block=0x98d7e47b... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719429
time="2024-03-26 20:46:13" level=info msg="Finished applying state transition" attestations=93 deposits=16 payloadHash=0x0dbef04e9a87 prefix=blockchain slot=8719429 syncBitsCount=506 txCount=202
time="2024-03-26 20:46:25" level=info msg="Synced new block" block=0x15e54d34... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719430
time="2024-03-26 20:46:25" level=info msg="Finished applying state transition" attestations=66 deposits=16 payloadHash=0xfeab77206308 prefix=blockchain slot=8719430 syncBitsCount=508 txCount=279 voluntaryExits=4
time="2024-03-26 20:46:38" level=info msg="Synced new block" block=0x60fd7ddc... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719431
time="2024-03-26 20:46:38" level=info msg="Finished applying state transition" attestations=99 deposits=16 payloadHash=0x635cba2a4cb6 prefix=blockchain slot=8719431 syncBitsCount=507 txCount=178
time="2024-03-26 20:47:00" level=info msg="Synced new block" block=0x2564e1a0... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719433
time="2024-03-26 20:47:00" level=info msg="Finished applying state transition" attestations=81 deposits=16 kzgCommitmentCount=1 payloadHash=0x07ad42bf5399 prefix=blockchain slot=8719433 syncBitsCount=508 txCount=269 voluntaryExits=1
time="2024-03-26 20:47:10" level=info msg="Peer summary" activePeers=77 inbound=0 outbound=77 prefix=p2p
time="2024-03-26 20:47:15" level=info msg="Synced new block" block=0xd7f78d58... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719434
time="2024-03-26 20:47:15" level=info msg="Finished applying state transition" attestations=128 deposits=16 kzgCommitmentCount=6 payloadHash=0x91efb082d5ba prefix=blockchain slot=8719434 syncBitsCount=490 txCount=221 voluntaryExits=2
time="2024-03-26 20:47:21" level=error msg="Could not handle p2p pubsub" error="could not validate blob data availability: context deadline waiting for blob sidecars: context deadline exceeded" prefix=sync topic="/eth2/6a95a1a9/beacon_block/ssz_snappy"
time="2024-03-26 20:47:25" level=info msg="Synced new block" block=0xa301409b... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719435
time="2024-03-26 20:47:25" level=info msg="Finished applying state transition" attestations=128 deposits=16 kzgCommitmentCount=6 payloadHash=0xf8cebe1cd791 prefix=blockchain slot=8719435 syncBitsCount=466 txCount=143
time="2024-03-26 20:47:39" level=info msg="Synced new block" block=0x57bd9b3d... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719436
time="2024-03-26 20:47:39" level=info msg="Finished applying state transition" attestations=128 deposits=16 kzgCommitmentCount=6 payloadHash=0x464890f6583f prefix=blockchain slot=8719436 syncBitsCount=505 txCount=163
time="2024-03-26 20:47:49" level=info msg="Synced new block" block=0x49df512c... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719437
time="2024-03-26 20:47:49" level=info msg="Finished applying state transition" attestations=128 deposits=16 payloadHash=0xd86b423ccbeb prefix=blockchain slot=8719437 syncBitsCount=479 txCount=160
time="2024-03-26 20:48:02" level=info msg="Synced new block" block=0xa07d2da3... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719438
time="2024-03-26 20:48:02" level=info msg="Finished applying state transition" attestations=82 deposits=16 kzgCommitmentCount=2 payloadHash=0xb85d55b3385b prefix=blockchain slot=8719438 syncBitsCount=509 txCount=194
time="2024-03-26 20:48:10" level=info msg="Peer summary" activePeers=77 inbound=0 outbound=77 prefix=p2p
time="2024-03-26 20:48:13" level=info msg="Synced new block" block=0xa4ec06fe... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719439
time="2024-03-26 20:48:13" level=info msg="Finished applying state transition" attestations=70 deposits=16 payloadHash=0x967b678ae487 prefix=blockchain slot=8719439 syncBitsCount=503 txCount=273
time="2024-03-26 20:48:25" level=info msg="Synced new block" block=0x05561af6... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719440
time="2024-03-26 20:48:25" level=info msg="Finished applying state transition" attestations=66 deposits=16 payloadHash=0x5d30083c64d5 prefix=blockchain slot=8719440 syncBitsCount=509 txCount=193 voluntaryExits=1
time="2024-03-26 20:48:36" level=info msg="Synced new block" block=0x00a46207... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719441
time="2024-03-26 20:48:36" level=info msg="Finished applying state transition" attestations=102 deposits=16 payloadHash=0xa47d21e07c9e prefix=blockchain slot=8719441 syncBitsCount=504 txCount=176
time="2024-03-26 20:48:49" level=info msg="Synced new block" block=0x7fe958a7... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719442
time="2024-03-26 20:48:49" level=info msg="Finished applying state transition" attestations=128 deposits=16 payloadHash=0xcf2db31df13a prefix=blockchain slot=8719442 syncBitsCount=508 txCount=171
time="2024-03-26 20:49:01" level=info msg="Synced new block" block=0x162fa189... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719443
time="2024-03-26 20:49:01" level=info msg="Finished applying state transition" attestations=128 deposits=16 kzgCommitmentCount=6 payloadHash=0xe6c8df20d93a prefix=blockchain slot=8719443 syncBitsCount=505 txCount=168 voluntaryExits=1
time="2024-03-26 20:49:10" level=info msg="Peer summary" activePeers=77 inbound=0 outbound=77 prefix=p2p
time="2024-03-26 20:49:13" level=info msg="Synced new block" block=0x2612cc9d... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719444
time="2024-03-26 20:49:13" level=info msg="Finished applying state transition" attestations=128 deposits=8 payloadHash=0xce7c5a90ab79 prefix=blockchain slot=8719444 syncBitsCount=491 txCount=213
time="2024-03-26 20:49:25" level=info msg="Synced new block" block=0x9ec7a34b... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719445
time="2024-03-26 20:49:25" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xbdcfbeee6cf1 prefix=blockchain slot=8719445 syncBitsCount=507 txCount=150
time="2024-03-26 20:49:36" level=info msg="Synced new block" block=0xd967d7cc... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719446
time="2024-03-26 20:49:36" level=info msg="Finished applying state transition" attestations=64 payloadHash=0xbd531dba40a7 prefix=blockchain slot=8719446 syncBitsCount=502 txCount=164
time="2024-03-26 20:49:48" level=info msg="Synced new block" block=0x91197f4e... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719447
time="2024-03-26 20:49:48" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x0e46d3f19c43 prefix=blockchain slot=8719447 syncBitsCount=509 txCount=142
time="2024-03-26 20:50:00" level=info msg="Synced new block" block=0xe92a6523... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719448
time="2024-03-26 20:50:00" level=info msg="Finished applying state transition" attestations=85 payloadHash=0xcd3fff03e092 prefix=blockchain slot=8719448 syncBitsCount=510 txCount=202
time="2024-03-26 20:50:10" level=info msg="Peer summary" activePeers=77 inbound=0 outbound=77 prefix=p2p
time="2024-03-26 20:50:14" level=info msg="Synced new block" block=0x20deed68... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719449
time="2024-03-26 20:50:14" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0x39dbaa5bcf58 prefix=blockchain slot=8719449 syncBitsCount=509 txCount=264
time="2024-03-26 20:50:24" level=info msg="Synced new block" block=0xf838a577... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719450
time="2024-03-26 20:50:24" level=info msg="Finished applying state transition" attestations=80 payloadHash=0x681cb9f52061 prefix=blockchain slot=8719450 syncBitsCount=498 txCount=131
time="2024-03-26 20:50:38" level=info msg="Synced new block" block=0xca6daead... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719451
time="2024-03-26 20:50:38" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0xfb8f3ea921c8 prefix=blockchain slot=8719451 syncBitsCount=509 txCount=195
time="2024-03-26 20:50:49" level=info msg="Synced new block" block=0x45b60c1e... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719452
time="2024-03-26 20:50:49" level=info msg="Finished applying state transition" attestations=77 payloadHash=0xcdf4bc6217f0 prefix=blockchain slot=8719452 syncBitsCount=504 txCount=135
time="2024-03-26 20:51:00" level=info msg="Synced new block" block=0x01b28d62... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719453
time="2024-03-26 20:51:00" level=info msg="Finished applying state transition" attestations=65 payloadHash=0x5c2076de961d prefix=blockchain slot=8719453 syncBitsCount=508 txCount=51
time="2024-03-26 20:51:10" level=info msg="Peer summary" activePeers=79 inbound=0 outbound=79 prefix=p2p
time="2024-03-26 20:51:12" level=info msg="Synced new block" block=0x93dfe6cb... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719454
time="2024-03-26 20:51:12" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xa84f259ba5ae prefix=blockchain slot=8719454 syncBitsCount=507 txCount=216
time="2024-03-26 20:51:25" level=info msg="Synced new block" block=0xb6db21ed... epoch=272482 finalizedEpoch=272480 finalizedRoot=0x16dbdcc1... prefix=blockchain slot=8719455
time="2024-03-26 20:51:25" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xbbe912f8f9f1 prefix=blockchain slot=8719455 syncBitsCount=509 txCount=346
terencechain commented 4 months ago

From the look, this seems okay as we've seen block gossip without blobs. As long as your node stays syncing / not stuck, then it should be fine. Feel free to reopen this if it's not the case, and we'll take a further look.

YuXiaoCoder commented 4 months ago

I also encountered another error where I wasn't sure if there was a connection, causing the number of proofs lost to go up

time="2024-03-27 10:50:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xad249e3a6fdb committeeIndex=16 error="could not process attestation: epoch 272613 root 0xad249e3a6fdb42e16f84970249f531e27a3a2a18e2fcdc59edb69fba38ad588c: not a checkpoint in forkchoice" prefix=blockchain slot=8723647 targetRoot=0xad249e3a6fdb
time="2024-03-27 10:50:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xad249e3a6fdb committeeIndex=26 error="could not process attestation: epoch 272613 root 0xad249e3a6fdb42e16f84970249f531e27a3a2a18e2fcdc59edb69fba38ad588c: not a checkpoint in forkchoice" prefix=blockchain slot=8723644 targetRoot=0xad249e3a6fdb
time="2024-03-27 10:50:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=3 beaconBlockRoot=0xad249e3a6fdb committeeIndex=3 error="could not process attestation: epoch 272613 root 0xad249e3a6fdb42e16f84970249f531e27a3a2a18e2fcdc59edb69fba38ad588c: not a checkpoint in forkchoice" prefix=blockchain slot=8723647 targetRoot=0xad249e3a6fdb
time="2024-03-27 10:50:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=3 beaconBlockRoot=0xad249e3a6fdb committeeIndex=56 error="could not process attestation: epoch 272613 root 0xad249e3a6fdb42e16f84970249f531e27a3a2a18e2fcdc59edb69fba38ad588c: not a checkpoint in forkchoice" prefix=blockchain slot=8723647 targetRoot=0xad249e3a6fdb
time="2024-03-27 10:50:09" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xad249e3a6fdb committeeIndex=11 error="could not process attestation: epoch 272613 root 0xad249e3a6fdb42e16f84970249f531e27a3a2a18e2fcdc59edb69fba38ad588c: not a checkpoint in forkchoice" prefix=blockchain slot=8723647 targetRoot=0xad249e3a6fdb
time="2024-03-27 11:02:57" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xd869de7ffb2d committeeIndex=43 error="could not process attestation: epoch 272615 root 0xd869de7ffb2d39491da321d3add818837a2a40b5ba8e012b5611dbdc2ba13f68: not a checkpoint in forkchoice" prefix=blockchain slot=8723711 targetRoot=0xd869de7ffb2d
time="2024-03-27 11:02:57" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xd869de7ffb2d committeeIndex=34 error="could not process attestation: epoch 272615 root 0xd869de7ffb2d39491da321d3add818837a2a40b5ba8e012b5611dbdc2ba13f68: not a checkpoint in forkchoice" prefix=blockchain slot=8723710 targetRoot=0xd869de7ffb2d
time="2024-03-27 11:02:57" level=warning msg="Could not process attestation for fork choice" aggregationCount=2 beaconBlockRoot=0xd869de7ffb2d committeeIndex=27 error="could not process attestation: epoch 272615 root 0xd869de7ffb2d39491da321d3add818837a2a40b5ba8e012b5611dbdc2ba13f68: not a checkpoint in forkchoice" prefix=blockchain slot=8723711 targetRoot=0xd869de7ffb2d
time="2024-03-27 11:03:07" level=error msg="Could not handle p2p pubsub" error="could not validate blob data availability: context deadline waiting for blob sidecars: context deadline exceeded" prefix=sync topic="/eth2/6a95a1a9/beacon_block/ssz_snappy"
hwhe commented 2 months ago

i meet the same problem. it occur "could not validate blob data availability" error. Can you help identify the probable cause?

time="2024-04-28 02:59:59" level=info msg="Peer summary" activePeers=2 inbound=2 outbound=0 prefix=p2p time="2024-04-28 02:58:59" level=info msg="Peer summary" activePeers=2 inbound=2 outbound=0 prefix=p2p time="2024-04-28 02:57:59" level=info msg="Peer summary" activePeers=2 inbound=2 outbound=0 prefix=p2p time="2024-04-28 02:57:38" level=error msg="Could not handle p2p pubsub" error="could not validate blob data availability: context deadline waiting for blob sidecars slot: 1528485, BlockRoot: 0x58e28a75f6e597bda0ea5c843975f13c2394ae714357c7b105d852c37774248a: context deadline exceeded" prefix=sync topic="/eth2/69ae0e99/beacon_block/ssz_snappy" time="2024-04-28 02:57:12" level=error msg="Still waiting for DA check at slot end." blobsExpected=6 blobsWaiting=3 prefix=blockchain root=0x58e28a75f6e597bda0ea5c843975f13c2394ae714357c7b105d852c37774248a slot=1528485 time="2024-04-28 02:56:59" level=info msg="Peer summary" activePeers=2 inbound=2 outbound=0 prefix=p2p time="2024-04-28 02:56:49" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0x777edc5613ab prefix=blockchain slot=1528484 syncBitsCount=357 txCount=83 time="2024-04-28 02:56:49" level=info msg="Synced new block" block=0xbf97c55c... epoch=47765 finalizedEpoch=47763 finalizedRoot=0xda215c47... prefix=blockchain slot=1528484 time="2024-04-28 02:56:39" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x5658935227e3 prefix=blockchain slot=1528483 syncBitsCount=475 txCount=0 time="2024-04-28 02:56:39" level=info msg="Synced new block" block=0xc3970907... epoch=47765 finalizedEpoch=47763 finalizedRoot=0xda215c47... prefix=blockchain slot=1528483 time="2024-04-28 02:56:26" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x6127fe994f64 prefix=blockchain slot=1528482 syncBitsCount=12 txCount=20 time="2024-04-28 02:56:26" level=info msg="Synced new block" block=0x68cb6676... epoch=47765 finalizedEpoch=47763 finalizedRoot=0xda215c47... prefix=blockchain slot=1528482 time="2024-04-28 02:56:17" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=4 payloadHash=0x90d1ba2b95b2 prefix=blockchain slot=1528481 syncBitsCount=484 txCount=93 time="2024-04-28 02:56:17" level=info msg="Synced new block" block=0x82e7caf1... epoch=47765 finalizedEpoch=47763 finalizedRoot=0xda215c47... prefix=blockchain slot=1528481 time="2024-04-28 02:56:02" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:02" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:02" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500 time="2024-04-28 02:56:01" level=info msg="Updated fee recipient addresses for validator indices" prefix="rpc/validator" validatorCount=500