Upgrade to v5.0.1, restart node, data not synchronized

wetezos commented 7 months ago

Describe the bug

`/home/eth/prysm.sh beacon-chain --execution-endpoint=http://localhost:8551 --holesky --jwt-secret=/home/eth/jwt.hex --genesis-state=/home/eth/genesis.ssz --genesis-beacon-api-url=https://holesky.beaconstate.info --slots-per-archive-point 231072 Latest Prysm version is v5.0.0. Beacon chain is up to date. Verifying binary integrity. beacon-chain-v5.0.0-linux-amd64: OK gpg: Signature made Thu 22 Feb 2024 03:28:08 AM UTC gpg: using RSA key 0AE0051D647BA3C1A917AF4072E33E4DF1A5036E gpg: Good signature from "Preston Van Loon preston@pvl.dev" [unknown] gpg: aka "Preston Van Loon preston@prysmaticlabs.com" [unknown] gpg: aka "Preston Van Loon preston90@gmail.com" [unknown] gpg: aka "Preston Van Loon (0xf71E9C766Cdf169eDFbE2749490943C1DC6b8A55) preston@machinepowered.com" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 0AE0 051D 647B A3C1 A917 AF40 72E3 3E4D F1A5 036E Verified /home/eth/dist/beacon-chain-v5.0.0-linux-amd64 has been signed by Prysmatic Labs. Starting Prysm beacon-chain --execution-endpoint=http://localhost:8551 --holesky --jwt-secret=/home/eth/jwt.hex --genesis-state=/home/eth/genesis.ssz --genesis-beacon-api-url=https://holesky.beaconstate.info --slots-per-archive-point 231072 [2024-03-19 07:36:35] INFO Finished reading JWT secret from /home/eth/jwt.hex [2024-03-19 07:36:35] WARN flags: Running on the Holesky Beacon Chain Testnet [2024-03-19 07:36:35] WARN node: In order to receive transaction fees from proposing blocks, you must provide flag --suggested-fee-recipient with a valid ethereum address when starting your beacon node. Please see our documentation for more information on this requirement (https://docs.prylabs.network/docs/execution-node/fee-recipient). [2024-03-19 07:36:35] INFO node: Checking DB database-path=/home/eth/.eth2/beaconchaindata [2024-03-19 07:36:35] INFO db: Opening Bolt DB at /home/eth/.eth2/beaconchaindata/beaconchain.db [2024-03-19 07:36:35] WARN genesis: database contains genesis with htr=0x0ea3f6f9515823b59c863454675fefcd1d8b4f2dbe454db166206a41fda060a0, ignoring remote genesis state parameter [2024-03-19 07:36:35] INFO node: Deposit contract: 0x4242424242424242424242424242424242424242 [2024-03-19 07:36:35] INFO p2p: Running node with peer id of 16Uiu2HAm6bWTn5TuTbPBvjrvBGp4UA83o9Azi8hh3fpvTNHEjMWt

`

Has this worked before in a previous version?

yes, My node run normally on v5.0.0

🔬 Minimal Reproduction

I run the node o v5.0.0, then I upgraded the node to v5.0.1 and restart the node , the data not synchronized

my command :

/home/eth/prysm.sh beacon-chain --execution-endpoint=http://localhost:8551 --holesky --jwt-secret=/home/eth/jwt.hex --genesis-state=/home/eth/genesis.ssz --genesis-beacon-api-url=https://holesky.beaconstate.info --slots-per-archive-point 231072

Error

no error, but the data not synchronized

Platform(s)

No response

What version of Prysm are you running? (Which release)

v5.0.1

Anything else relevant (validator index / public key)?

No response

wetezos commented 7 months ago

![Uploading image.png…]()

nisdas commented 7 months ago

Any reason, you used this flag ? --slots-per-archive-point 231072 . Such a large value can cause long start up times

wetezos commented 7 months ago

Any reason, you used this flag ? --slots-per-archive-point 231072 . Such a large value can cause long start up times

Hi @nisdas , Before the upgrade, I restart the node several times and it run ok. This time I deleted the parameter --slots-per-archive-point 231072 but the data is still not synchronized

nisdas commented 7 months ago

If you move back down to v5.0.0 does it help ?

wetezos commented 7 months ago

If you move back down to v5.0.0 does it help ?

No, I have moved back down to v5.0.0, and the data is not synchronized anymore

nisdas commented 7 months ago

Your issue is most likely this flag:

-slots-per-archive-point 231072

I do not think we have ever had someone use such a high value. This possibly could have broken something. If you use this flag, does it work ?

wetezos commented 7 months ago

Your issue is most likely this flag:
-slots-per-archive-point 231072
I do not think we have ever had someone use such a high value. This possibly could have broken something. If you use this flag, does it work ?

We haven't tested it specifically, but we know that data from around 7 days can be accessed normally

prestonvanloon commented 7 months ago

It seems plausible that the node started and has to replay up to 231072 slots since that is the archive interval you've chosen. Could you try this again without that flag?

Edit: You may have to resync from a checkpoint or genesis without the flag and then restart to see if it worked.

james-prysm commented 7 months ago

any updates on this?

james-prysm commented 4 days ago

stale closing

prysmaticlabs / prysm