Investigate running lodestar on Nimbus's insecura - the "long ranged attacked" pyrmont fork

g11tech commented 2 years ago

Nimbus team created a long range attack scenario on Pyrmont network as described here: https://notes.status.im/nimbus-insecura-network?view tl-dr: They rewrote pyrnmont chain from epoch 60000 for 12k validators way back by using these validators (leaking the other 100k something validators on this alternate chain) so that this alternate chain finally finalized after exiting the majority "non attacking validators". Now these node/validators are serving alternate chain. We want to test out the wss sync related issues using lodestar.

Check test following scenarios:

[x] Scenario 1: sync on the attacking chain using a compromised checkpoint and see further finalization: see comment
[x] Scenario 2: sync from the non attacked checkpoint (<= epoch 60k) to the attacking chain using the compromised bootnodes UPDATE: couldn't sync see comment
[x] Scenario 3: syncing the prater on forward sync from blank db see comment
- [x] Sync started - OPINION : sync shouldn't have started
- [x] validate if it sync to the present on the attacking chain

UPDATE: following two scenarios can't be tested because insecura is (invalid) continuation of the abondoned pyrmont chain, so no checkpoint exists on pyrmont which is not part of insecura to test the following two points: - [ ] Scenario 4: invalidate the sync on the attacking chain with compromised bootnodes but using a valid forward checkpoint (using the forward checkpoint PR) https://github.com/ChainSafe/lodestar/pull/3589 ~~- [ ] Scenario 5: Invalidate the backfill sync on the attacking chain with a wss sync but a valid checkpoint in past from the normal chain~~

g11tech commented 2 years ago

UPDATE: Nimbus's wss endpoint seems to be loaded/stuck, not moving past fetching the state from their provided sync url :

bash-5.1# ./lodestar beacon --network pyrmont --eth1.enabled false --rootDir /data/insecura --weakSubjectivitySyncLatest --weakSubjectivityServerUrl http://insecura.nimbus.team
Jan-22 10:37:05.729 []                 info: Lodestar version=v0.26.0/master/+465/f9cb593d (git), network=pyrmont
Jan-22 10:37:05.741 [DB]               info: Connected to LevelDB database name=/data/insecura/chain-db
Jan-22 10:37:05.742 []                 info: Fetching weak subjectivity state from http://insecura.nimbus.team/eth/v2/debug/beacon/states/finalized

^CJan-22 10:37:41.851 []                 info: Stopping gracefully, use Ctrl+C again to force process exit
bash-5.1# ./lodestar beacon --network pyrmont --eth1.enabled false --rootDir /data/insecura --weakSubjectivitySyncLatest --weakSubjectivityServerUrl http://insecura.nimbus.team --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp
Jan-22 10:38:53.208 []                 info: Lodestar version=v0.26.0/master/+465/f9cb593d (git), network=pyrmont
Jan-22 10:38:53.219 [DB]               info: Connected to LevelDB database name=/data/insecura/chain-db
Jan-22 10:38:53.221 []                 info: Fetching weak subjectivity state from http://insecura.nimbus.team/eth/v2/debug/beacon/states/finalized

g11tech commented 2 years ago

UPDATE: synced on the attacking/alternate/invalid chain using compromised wss checkpoint and compromised bootnodes serving invalid chain

an-22 12:19:13.001 []                 info: Synced - slot: 3096095 - head: 3096095 0x04ee…2461 - finalized: 0xa3c0…d03e:96748 - peers: 2
Jan-22 12:19:25.001 []                 info: Synced - slot: 3096096 (skipped 1) - head: 3096095 0x04ee…2461 - finalized: 0x7994…ffb9:96749 - peers: 2
Jan-22 12:19:37.001 []                 info: Synced - slot: 3096097 - head: 3096097 0xc9a6…f93b - finalized: 0x7994…ffb9:96749 - peers: 2

Sucessfully backfilling from compromised checkpoint:

g11tech commented 2 years ago

UPDATE: Error while trying to sync from a finalized checkpoint < epoch 60,000 (a correct checkpoint) but from the attacking chain as the checkpoint way behind the current clock epoch

bash-5.1# ./lodestar beacon --network pyrmont --metrics.enabled  --eth1.enabled false --rootDir /data/insecura1 --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp --weakSubjectivityServerUrl http://insecura.nimbus.team --weakSubjectivitySyncLatest --weakSubjectivityCheckpoint 0x1c8fc8f25001641b25a3572837962684def277fa8f525a86c349ccd8fd2d818b:59998
Jan-24 10:00:09.470 []                 info: Lodestar version=v0.26.0/g11tech/wssurlversion/+469/e6f2585a (git), network=pyrmont
Jan-24 10:00:09.612 [DB]               info: Connected to LevelDB database name=/data/insecura1/chain-db
Jan-24 10:00:10.834 []                 info: Fetching weak subjectivity state weakSubjectivityServerUrl=http://insecura.nimbus.team
 ✖ Error: Fetched weak subjectivity checkpoint not within weak subjectivity period.
    at initAndVerifyWeakSubjectivityState (/usr/app/packages/cli/src/cmds/beacon/initBeaconState.ts:58:11)
    at initBeaconState (/usr/app/packages/cli/src/cmds/beacon/initBeaconState.ts:115:12)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.beaconHandler [as handler] (/usr/app/packages/cli/src/cmds/beacon/handler.ts:75:41)

g11tech commented 2 years ago

Wss sync using a checkpoint from correct pyrmont chain but syncing via attacking chain failed because it seems that the pyrmont chain isn't finalizing, as it has been abandoned

curl https://************@eth2-beacon-prater.infura.io/eth/v1/beacon/states/finalized/finality_checkpoints 
{"data":{"previous_justified":{"epoch":"69034","root":"0x86144efe27f6e7338bd7480568b2d1b256fd0ee0757fe58978e82add875613e4"},"current_justified":{"epoch":"69035","root":"0xde0c88c56b246fbf90429992f565e9972d52305e3fe4e5da172951aee5910a79"},"finalized":{"epoch":"69034","root":"0x86144efe27f6e7338bd7480568b2d1b256fd0ee0757fe58978e82add875613e4"}}}

lodestar:

./lodestar beacon --network pyrmont --metrics.enabled  --eth1.enabled false --rootDir /data/insecura1 --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp --weakSubjectivityServerUrl http://insecura.nimbus.team --weakSubjectivitySyncLatest --weakSubjectivityCheckpoint 0x86144efe27f6e7338bd7480568b2d1b256fd0ee0757fe58978e82add875613e4:69034
Jan-24 10:11:53.198 []                 info: Lodestar version=v0.26.0/g11tech/wssurlversion/+469/e6f2585a (git), network=pyrmont
Jan-24 10:11:53.208 [DB]               info: Connected to LevelDB database name=/data/insecura1/chain-db
Jan-24 10:11:54.355 []                 info: Fetching weak subjectivity state weakSubjectivityServerUrl=http://insecura.nimbus.team
 ✖ Error: Unable to fetch weak subjectivity state: request to http://insecura.nimbus.team/eth/v2/debug/beacon/states/2209088 failed, reason: connect ECONNREFUSED 65.21.196.45:80
    at fetchWeakSubjectivityState (/usr/app/packages/cli/src/networks/index.ts:168:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at initBeaconState (/usr/app/packages/cli/src/cmds/beacon/initBeaconState.ts:109:37)
    at Object.beaconHandler [as handler] (/usr/app/packages/cli/src/cmds/beacon/handler.ts:75:41)

g11tech commented 2 years ago

lodestar started syncing on from blank db without using checkpoint sync...

./lodestar beacon --network pyrmont --metrics.enabled  --eth1.enabled false --rootDir /data/insecura --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp
Jan-24 10:48:33.411 []                 info: Lodestar version=v0.26.0/g11tech/wssurlversion/+469/e6f2585a (git), network=pyrmont
Jan-24 10:48:33.419 [DB]               info: Connected to LevelDB database name=/data/insecura/chain-db
Jan-24 10:48:39.611 []                 info: Initializing beacon state from anchor state slot=0, epoch=0, stateRoot=0x2bb257ca66d05a047a65fe43a5f457b674de445d917cca029efb09b3ba4758c4
Jan-24 10:48:43.480 [SYNC]             info: BackfillSync - initializing from Checkpoint root=0x6a89af5df908893eedbed10ba4c13fc13d5653ce57db637e3bfded73a987bb87, epoch=0, slot=0
Jan-24 10:48:43.488 [METRICS]          info: Starting metrics HTTP server port=8008
(node:7245) MaxListenersExceededWarning: Possible EventEmitter memory leak detec

UPDATE: still syncing :

Jan-25 05:26:13.001 []                 info: Syncing - 2.6 days left - 10.3 slots/s - slot: 3115630 (skipped 2341060) - head: 774570 0x89d8…5f73 - finalized: 0x316b…f0fc:24203 - peers: 8

###############################

UPDATE: sync stalled near epoch at which pyrmont was abandoned (~60,000)

Jan-26 15:22:49.002 []                 info: Searching peers - peers: 0 - slot: 3125813 (skipped 1082143) - head: 2043670 0x5275…15b2 - finalized: 0x1c8f…818b:59998
Jan-26 15:23:01.000 []                 info: Searching peers - peers: 0 - slot: 3125814 (skipped 1082144) - head: 2043670 0x5275…15b2 - finalized: 0x1c8f…818b:59998
Jan-26 15:23:13.000 []                 info: Searching peers - peers: 0 - slot: 3125815 (skipped 1082145) - head: 2043670 0x5275…15b2 - finalized: 0x1c8f…818b:59998

However on cleaning peer and restarting, it started syncing syncing on the insecura chain :

an-26 15:26:01.000 []                 info: Searching peers - peers: 0 - slot: 3125829 (skipped 1206149) - head: 1919680 0xc538…cfcd - finalized: 0xf09c…7ece:59988
Jan-26 15:26:13.007 []                 info: Syncing - ? left - 0.00 slots/s - slot: 3125830 (skipped 1206150) - head: 1919680 0xc538…cfcd - finalized: 0xf09c…7ece:59988 - peers: 3
Jan-26 15:26:25.001 []                 info: Syncing - 4.5 days left - 3.12 slots/s - slot: 3125831 (skipped 1205923) - head: 1919908 0x05c2…e0b7 - finalized: 0x5030…c8f0:59995 - peers: 3
Jan-26 15:26:37.033 []                 info: Syncing - 2.5 days left - 5.59 slots/s - slot: 3125832 (skipped 1205738) - head: 1920094 0x4171…6d32 - finalized: 0x2ccd…57e9:60001 - peers: 4
Jan-26 15:26:49.000 []                 info: Syncing - 1.9 days left - 7.42 slots/s - slot: 3125833 (skipped 1205557) - head: 1920276 0x3210…cf05 - finalized: 0xbeb0…f279:60006 - peers: 4
Jan-26 15:27:01.001 []                 info: Syncing - 1.6 days left - 8.73 slots/s - slot: 3125834 (skipped 1205385) - head: 1920449 0xae27…8be4 - finalized: 0x71a8…9f50:60012 - peers: 4

UPDATE: sucessfull synced to the insecura chainhead as expected

an-27 18:17:37.001 []                 info: Synced - slot: 3133887 (skipped 1) - head: 3133886 0x09ff…b8a1 - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:17:49.001 []                 info: Synced - slot: 3133888 (skipped 2) - head: 3133886 0x09ff…b8a1 - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:18:01.001 []                 info: Synced - slot: 3133889 - head: 3133889 0x26f3…559d - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:18:13.001 []                 info: Synced - slot: 3133890 (skipped 1) - head: 3133889 0x26f3…559d - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:18:25.000 []                 info: Synced - slot: 3133891 (skipped 2) - head: 3133889 0x26f3…559d - finalized: 0x6869…baf1:96544 - peers: 23

g11tech commented 2 years ago

CLOSING issue as Scenario 4,5 not possible to check, see description for the reasons.

dapplion commented 2 years ago

Thank you so much for doing this tests! Love to see the security protections in action

g11tech commented 2 years ago

Thank you so much for doing this tests! Love to see the security protections in action

@dapplion Should we apply weak subjectivity timeperiod checks on start from blank DB as well (Scenario 3 in the description)? which we don't do right now. This means one would not be able to start from a blank DB without providing a checkpoint.

dapplion commented 2 years ago

Should we apply weak subjectivity timeperiod checks on start from blank DB as well (Scenario 3 in the description)? which we don't do right now. This means one would not be able to start from a blank DB without providing a checkpoint.

What do other clients do? In that case you would do the WS math on the genesis state?

ChainSafe / lodestar

Investigate running lodestar on Nimbus's insecura - the "long ranged attacked" pyrmont fork #3658