search

prysmaticlabs / prysm

Go implementation of Ethereum proof of stake
https://www.offchainlabs.com
GNU General Public License v3.0
3.47k stars 999 forks source link

the param blob-retention-epochs set failed #14563

Open xinzhongyoumeng opened 1 day ago

xinzhongyoumeng commented 1 day ago

Describe the bug

we run the prysm node with such command from scratch a month ago, now we found the blob is only save the 4096 epoch. we just want save the full blob info. the command:

/opt/ethmain/core/beacon-chain --mainnet --chain-id=1 --network-id=1 --blob-retention-epochs=999999999 --execution-endpoint=http://127.0.0.1:8551 --jwt-secret=/mnt/ethmain/node/jwt.hex --datadir=/mnt/ethmain/node/prysm/beacon --enable-historical-state-representation --save-full-execution-payloads --blob-save-fsync --rpc-host=0.0.0.0 --rpc-port=4000 --rpc-max-page-size=200000 --http-modules=prysm,eth --grpc-gateway-host=0.0.0.0 --grpc-gateway-port=3500 --grpc-gateway-corsdomain=* --grpc-max-msg-size=268435456 --p2p-tcp-port=13000 --p2p-udp-port=12000 --disable-monitoring --accept-terms-of-use

Has this worked before in a previous version?

don't know

🔬 Minimal Reproduction

No response

Error

No response

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

beacon-chain version Prysm/v5.1.0/b8cd77945df2b8fa8fe50520df0495309a52c2f3. Built at: 2024-08-20 17:16:40+00:00

Anything else relevant (validator index / public key)?

No response

nisdas commented 1 day ago

Could you share logs from this particular node ? We have been running with the flag enabled fine on our internal infrastructure.

xinzhongyoumeng commented 1 day ago

Could you share logs from this particular node ? We have been running with the flag enabled fine on our internal infrastructure.

https://gist.github.com/xinzhongyoumeng/35f1c41d81e8aef2b2608b5a9842e406

xinzhongyoumeng commented 1 day ago 
# curl -w "\n" http://127.0.0.1:3500/eth/v1/node/syncing
{"data":{"head_slot":"10229776","sync_distance":"1","is_syncing":false,"is_optimistic":false,"el_offline":false}}
xinzhongyoumeng commented 1 day ago

and the execution client is erigon, and fully synchronized with the execution-layer blockchain

xinzhongyoumeng commented 1 day ago

Could you share logs from this particular node ? We have been running with the flag enabled fine on our internal infrastructure.

if you are sync from a checkpoint?

nisdas commented 1 day ago

if you are sync from a checkpoint?

Yes

Do you have any other logs ? From what I can see, your node is simply optimistically syncing

xinzhongyoumeng commented 1 day ago

if you are sync from a checkpoint?

Yes

Do you have any other logs ? From what I can see, your node is simply optimistically syncing

the other log is same as this:

time="2024-10-22 16:05:28" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x19e07bb1cb5a headSlot=10230025 prefix=blockchain
time="2024-10-22 16:05:28" level=info msg="Synced new block" block=0x30840c38... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230025
time="2024-10-22 16:05:28" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x19e07bb1cb5a prefix=blockchain slot=10230025 syncBitsCount=481 txCount=156 voluntaryExits=16
time="2024-10-22 16:05:38" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xd54116594ac3 headSlot=10230026 prefix=blockchain
time="2024-10-22 16:05:38" level=info msg="Synced new block" block=0x4c10875b... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230026
time="2024-10-22 16:05:38" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xd54116594ac3 prefix=blockchain slot=10230026 syncBitsCount=461 txCount=114 voluntaryExits=4
time="2024-10-22 16:05:53" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x1dde79296306 headSlot=10230027 prefix=blockchain
time="2024-10-22 16:05:54" level=info msg="Synced new block" block=0x9c4286a1... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230027
time="2024-10-22 16:05:54" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x1dde79296306 prefix=blockchain slot=10230027 syncBitsCount=511 txCount=260
time="2024-10-22 16:06:04" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x9752d48cfb8f headSlot=10230028 prefix=blockchain
time="2024-10-22 16:06:04" level=info msg="Synced new block" block=0x9f4cc9cf... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230028
time="2024-10-22 16:06:04" level=info msg="Finished applying state transition" attestations=114 kzgCommitmentCount=6 payloadHash=0x9752d48cfb8f prefix=blockchain slot=10230028 syncBitsCount=479 txCount=168
time="2024-10-22 16:06:15" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x1ed426164d48 headSlot=10230029 prefix=blockchain
time="2024-10-22 16:06:15" level=info msg="Synced new block" block=0x95a23a7f... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230029
time="2024-10-22 16:06:15" level=info msg="Finished applying state transition" attestations=87 payloadHash=0x1ed426164d48 prefix=blockchain slot=10230029 syncBitsCount=511 txCount=122
time="2024-10-22 16:06:18" level=info msg="Connected peers" inboundTCP=0 outboundTCP=67 prefix=p2p total=67
time="2024-10-22 16:06:29" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xc29193b1b86a headSlot=10230030 prefix=blockchain
time="2024-10-22 16:06:29" level=info msg="Synced new block" block=0x1c0349f2... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230030
time="2024-10-22 16:06:29" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0xc29193b1b86a prefix=blockchain slot=10230030 syncBitsCount=511 txCount=181
time="2024-10-22 16:06:39" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x1791495ac97e headSlot=10230031 prefix=blockchain
time="2024-10-22 16:06:39" level=info msg="Synced new block" block=0x22ef8bbb... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230031
time="2024-10-22 16:06:39" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0x1791495ac97e prefix=blockchain slot=10230031 syncBitsCount=512 txCount=190
time="2024-10-22 16:06:51" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xc09b9ccb343f headSlot=10230032 prefix=blockchain
time="2024-10-22 16:06:51" level=info msg="Synced new block" block=0xd511d6bf... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230032
time="2024-10-22 16:06:51" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xc09b9ccb343f prefix=blockchain slot=10230032 syncBitsCount=512 txCount=158
time="2024-10-22 16:07:03" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xd9e72477002c headSlot=10230033 prefix=blockchain
time="2024-10-22 16:07:04" level=info msg="Synced new block" block=0x9c9dce71... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230033
time="2024-10-22 16:07:04" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0xd9e72477002c prefix=blockchain slot=10230033 syncBitsCount=511 txCount=145
time="2024-10-22 16:07:17" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xd2d284e22dc4 headSlot=10230034 prefix=blockchain
time="2024-10-22 16:07:17" level=info msg="Synced new block" block=0x46334001... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230034
time="2024-10-22 16:07:17" level=info msg="Finished applying state transition" attestations=128 payloadHash=0xd2d284e22dc4 prefix=blockchain slot=10230034 syncBitsCount=511 txCount=158
time="2024-10-22 16:07:18" level=info msg="Connected peers" inboundTCP=0 outboundTCP=70 prefix=p2p total=70
time="2024-10-22 16:07:27" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xed169eb2c9d9 headSlot=10230035 prefix=blockchain
time="2024-10-22 16:07:27" level=info msg="Synced new block" block=0x19aea9fd... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230035
time="2024-10-22 16:07:27" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0xed169eb2c9d9 prefix=blockchain slot=10230035 syncBitsCount=501 txCount=134
time="2024-10-22 16:07:40" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xa96292120829 headSlot=10230036 prefix=blockchain
time="2024-10-22 16:07:40" level=info msg="Synced new block" block=0x2130ce80... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230036
time="2024-10-22 16:07:40" level=info msg="Finished applying state transition" attestations=85 kzgCommitmentCount=6 payloadHash=0xa96292120829 prefix=blockchain slot=10230036 syncBitsCount=509 txCount=197
time="2024-10-22 16:07:51" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x5332640046f0 headSlot=10230037 prefix=blockchain
time="2024-10-22 16:07:51" level=info msg="Synced new block" block=0xb393dc18... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230037
time="2024-10-22 16:07:51" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0x5332640046f0 prefix=blockchain slot=10230037 syncBitsCount=500 txCount=100
time="2024-10-22 16:08:02" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0xacaaeef29718 headSlot=10230038 prefix=blockchain
time="2024-10-22 16:08:03" level=info msg="Synced new block" block=0xb71368b8... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230038
time="2024-10-22 16:08:03" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=3 payloadHash=0xacaaeef29718 prefix=blockchain slot=10230038 syncBitsCount=511 txCount=100
time="2024-10-22 16:08:18" level=info msg="Called fork choice updated with optimistic block" finalizedPayloadBlockHash=0x9f3efdabf794 headPayloadBlockHash=0x9f9f92194fdf headSlot=10230039 prefix=blockchain
time="2024-10-22 16:08:18" level=info msg="Synced new block" block=0xc07b2c20... epoch=319688 finalizedEpoch=319686 finalizedRoot=0x368ad7f8... prefix=blockchain slot=10230039
time="2024-10-22 16:08:18" level=info msg="Finished applying state transition" attestations=120 payloadHash=0x9f9f92194fdf prefix=blockchain slot=10230039 syncBitsCount=511 txCount=231
xinzhongyoumeng commented 1 day ago

if you are sync from a checkpoint?

Yes

Do you have any other logs ? From what I can see, your node is simply optimistically syncing

I really don't understand why syncing a block from 0 can't retain the full set of blob data.

nisdas commented 1 day ago

How are you checking for the existence of blobs ? From your logs it appears your node is constantly optimistically syncing

xinzhongyoumeng commented 1 day ago

How are you checking for the existence of blobs ? From your logs it appears your node is constantly optimistically syncing

yes, I make below command:

# curl http://127.0.0.1:3500/eth/v1/beacon/blob_sidecars/10080032
{"data":[]}
xinzhongyoumeng commented 1 day ago

when I make the command , the result is normal, the blob info is corrent

curl http://127.0.0.1:3500/eth/v1/beacon/blob_sidecars/10092827
nisdas commented 1 day ago

Can you check In your blobs directory, that you have a folder with this name 0x0e08a1093f4bb3caaa2faff3f4d643cbbfe8082d130948eda01d52b02e2b08e7 ?

xinzhongyoumeng commented 1 day ago

0x0e08a1093f4bb3caaa2faff3f4d643cbbfe8082d130948eda01d52b02e2b08e7

no this folder

nisdas commented 1 day ago

Ok but you have this folder ?

0x99aa590fdd02a142868f23fb729ae2bd8bc4fd957209767807ed86f8f11030c9
xinzhongyoumeng commented 1 day ago

0x99aa590fdd02a142868f23fb729ae2bd8bc4fd957209767807ed86f8f11030c9

yes,

/mnt/ethmain/node/prysm/beacon/blobs/0x99aa590fdd02a142868f23fb729ae2bd8bc4fd957209767807ed86f8f11030c9# ls
0.ssz  1.ssz  2.ssz  3.ssz  4.ssz  5.ssz
nisdas commented 1 day ago

Can you try running with --verbosity=debug to see if there is any interesting logs ?

xinzhongyoumeng commented 1 day ago

--verbosity=debug

tried it

https://gist.github.com/xinzhongyoumeng/8a5ce66adad7c0d8f89aa86541b646c2

nisdas commented 1 day ago

Thanks, we will take a look at it

xinzhongyoumeng commented 1 day ago

from 17:35 to 17:42 https://gist.github.com/xinzhongyoumeng/78fb99acf734a44bea31d369fd156422#file-beacon-17-42-log

xinzhongyoumeng commented 7 hours ago

any interesting logs

If you can find any interesting logs?