Consider checkpoint sync to main beacon node startup

[arnetheduck]

• My ideal mode of operation would be to add both --trusted-node-url and --backfill as command line arguments to the normal nimbus_beacon_node command (i.e. "no action"), and include them if the user specified that they would like to checkpoint sync. I would then set --trusted-node-url to whatever they specified, and --backfill to false so Nimbus pulls the historical blocks from gossip once checkpoint syncing has completed.
  • This is the behavior that both Lighthouse and Teku currently provide.
  • With the separate trustedNodeSync action, I have to manually determine whether or not Nimbus has sync'd before starting it up, and whether or not to start with the special action first and then restart into the conventional mode after. I'd strongly suggest removing the explicit action and building support for this into the "no action" mode so the client itself can make this determination and behave accordingly.

Originally posted by @jclapis in https://github.com/status-im/nimbus-eth2/issues/3209#issuecomment-1003598851

[arnetheduck]

TODO:

• [ ] Evaluate backfill mode in automated checkpoint sync - this is going to trip people syncing from infura etc
• [ ] Consider what to do when checkpoint is newer than head but head is newer than WS point: trust checkpoint or own head? (probably the latter)
• [ ] Consider what to do when head is older than WS point: who do we trust more?

[arnetheduck]

For those looking for behavior similar to checkpoint sync in other clients:

# Run TNS when database is missing, skipping the backfill
[ -f "$DATADIR/db/nbc.sqlite3" ] || build/nimbus_beacon_node --data-dir:"$DATADIR" trustedNodeSync --backfill:false
# now run beacon node as normal..
build/nimbus_beacon_node ...

[dapplion]

This would be nice!

[maxtreyver]

Yes, huge +1 on this. I'm new to the ETH staking community, and this particular UX discomfort was a big negative of Nimbus compared to others (Lighthouse, Teku, I think Prysm has it or something like it too?)

[michaelsproul]

Just ran into this after my disk filled up. Especially with Docker images and volumes to fuss around with, a flag is nicer than a separate command.

[etan-status]

https://nimbus.guide/start-syncing.html#checkpoint-sync

You can use --external-beacon-api-url with --trusted-state-root to sync from Checkpointz on main command. DB dir has to be empty though.

[etan-status]

--trusted-block-root doesn't work currently with checkpointz, as it needs LC data to be served. Tracking issue: https://github.com/ethpandaops/checkpointz/issues/143

Only --trusted-state-root works for now.

[michaelsproul]

Thanks @etan-status, I will use that next time!

Curious to try --trusted-block-root with Lighthouse's LC data server too.

[michaelsproul]

Are there any plans to support checkpoint sync without specifying a block root/state root ahead of time? I think it could still be slightly more ergonomic, as it wouldn't require editing .service files, or deployment scripts or similar. Instead the user could check the logs after syncing to ensure they're on the right chain. I'm aware lots of users do not check, so if you want to require them to check the block root, I think the mandatory block root/state root flag is a good middle ground.

[etan-status]

Yes, with EIP-7658 it would be possible to bake in a deeply finalized block root into the network metadata and start syncing from there. With light client data being backfillable, its availability on the network would improve, and when full data from the deeply finalized block to a recent one is available, syncing can be performed automatically.

Operation based on a trusted server requires the additional trustedNodeSync step on Nimbus. The main startup command only supports secure sync with an explicit (not "finalized") --trusted-block-root (can be an older checkpoint, but requires server to serve LC data; or alternatively with an explicit --trusted-state-root (has to be a recent checkpoint, but also works with checkpointz). For testnets with post-altair genesis, it also works without any root, as long as the server supports LC data.

[etan-status]

EIP-7658 is in this PR: https://github.com/ethereum/consensus-specs/pull/3614

[etan-status]

@michaelsproul re Lighthouse LC data server, would be great to get feedback on https://github.com/ethpandaops/checkpointz/issues/143 whether it is acceptable to proxy the LC related routes for you as well. Nimbus/Lodestar are fine with it.