search

OffchainLabs / eth-pos-devnet

MIT License
103 stars 124 forks source link

No new block is generated after the restart #19

Closed LilkkJerry closed 10 months ago

LilkkJerry commented 1 year ago

Hi, I tried to generate a stable private Ethereum PoS network for long-term test, and i am now using this project. It works well when I launch it at the first time, but after restarting(by docker-compose up -d, i deleted 2 create-genesis parts) the network, the generation of new block stop. Is there anyway to continue running the network?

Geth Log: INFO [04-25|07:03:32.179] Starting Geth on Ethereum mainnet... INFO [04-25|07:03:32.179] Bumping default cache on mainnet provided=1024 updated=4096 INFO [04-25|07:03:32.181] Maximum peer count ETH=50 LES=0 total=50 INFO [04-25|07:03:32.182] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" INFO [04-25|07:03:32.187] Set global gas cap cap=50,000,000 INFO [04-25|07:03:32.189] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB INFO [04-25|07:03:32.189] Using pebble as the backing database INFO [04-25|07:03:32.189] Allocated cache and file handles database=/execution/geth/chaindata cache=2.00GiB handles=524,288 INFO [04-25|07:03:32.208] Opened ancient database database=/execution/geth/chaindata/ancient/chain readonly=false INFO [04-25|07:03:32.208] Initialising Ethereum protocol network=1 dbversion=8 INFO [04-25|07:03:32.214]
INFO [04-25|07:03:32.214] --------------------------------------------------------------------------------------------------------------------------------------------------------- INFO [04-25|07:03:32.214] Chain ID: 32382 (unknown) INFO [04-25|07:03:32.214] Consensus: Beacon (proof-of-stake), merging from Clique (proof-of-authority) INFO [04-25|07:03:32.215]
INFO [04-25|07:03:32.215] Pre-Merge hard forks (block based): INFO [04-25|07:03:32.215] - Homestead: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) INFO [04-25|07:03:32.215] - Tangerine Whistle (EIP 150): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) INFO [04-25|07:03:32.215] - Spurious Dragon/1 (EIP 155): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) INFO [04-25|07:03:32.215] - Spurious Dragon/2 (EIP 158): #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) INFO [04-25|07:03:32.215] - Byzantium: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) INFO [04-25|07:03:32.215] - Constantinople: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) INFO [04-25|07:03:32.215] - Petersburg: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) INFO [04-25|07:03:32.215] - Istanbul: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) INFO [04-25|07:03:32.215] - Muir Glacier: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) INFO [04-25|07:03:32.215] - Berlin: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md) INFO [04-25|07:03:32.215] - London: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md) INFO [04-25|07:03:32.215] - Arrow Glacier: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md) INFO [04-25|07:03:32.215] - Gray Glacier: #0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md) INFO [04-25|07:03:32.215]
INFO [04-25|07:03:32.215] Merge configured: INFO [04-25|07:03:32.215] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md INFO [04-25|07:03:32.215] - Network known to be merged: false INFO [04-25|07:03:32.215] - Total terminal difficulty: 0 INFO [04-25|07:03:32.215]
INFO [04-25|07:03:32.215] Post-Merge hard forks (timestamp based): INFO [04-25|07:03:32.215] - Shanghai: @1682389237 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md) INFO [04-25|07:03:32.215]
INFO [04-25|07:03:32.215] --------------------------------------------------------------------------------------------------------------------------------------------------------- INFO [04-25|07:03:32.215]
INFO [04-25|07:03:32.215] Loaded most recent local block number=1413 hash=80dad6..5c8f53 td=1 age=2m43s INFO [04-25|07:03:32.215] Loaded most recent local finalized block number=1398 hash=803d45..8a6925 td=1 age=5m43s INFO [04-25|07:03:32.216] Loaded local transaction journal transactions=0 dropped=0 INFO [04-25|07:03:32.216] Regenerated local transaction journal transactions=0 accounts=0 INFO [04-25|07:03:32.217] Chain post-TTD, sync via beacon client INFO [04-25|07:03:32.217] Gasprice oracle is ignoring threshold set threshold=2 WARN [04-25|07:03:32.217] Engine API enabled protocol=eth INFO [04-25|07:03:32.217] Starting peer-to-peer node instance=Geth/v1.11.7-unstable-bbcb5ea3/linux-amd64/go1.20.3 INFO [04-25|07:03:32.234] IPC endpoint opened url=/execution/geth.ipc INFO [04-25|07:03:32.234] Loaded JWT secret file path=/execution/jwtsecret crc32=0xb3e8e920 INFO [04-25|07:03:32.234] New local node record seq=1,682,389,094,355 id=eb6994d83a9bbab5 ip=127.0.0.1 udp=0 tcp=30303 INFO [04-25|07:03:32.234] HTTP server started endpoint=[::]:8545 auth=false prefix= cors= vhosts=localhost INFO [04-25|07:03:32.234] Started P2P networking self="enode://da4ed38f11711618688672bf06bbe767f2b22522cf0e52c54cbc21bb92b8ef34d97c1f9f1e001d02bc68f544af017fbb9b2412661beb7b9c2c894d97e7db78de@127.0.0.1:30303?discport=0" INFO [04-25|07:03:32.234] WebSocket enabled url=ws://[::]:8551 INFO [04-25|07:03:32.234] HTTP server started endpoint=[::]:8551 auth=true prefix= cors=localhost vhosts=* INFO [04-25|07:03:33.036] Unlocked account address=0x123463a4B065722E99115D6c222f267d9cABb524 WARN [04-25|07:04:07.218] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain! WARN [04-25|07:09:07.251] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!

Beacon Log: time="2023-04-25 07:03:32" level=info msg="Finished reading JWT secret from /execution/jwtsecret" time="2023-04-25 07:03:32" level=warning msg="Running on custom Ethereum network specified in a chain configuration yaml file" prefix=flags time="2023-04-25 07:03:32" level=warning msg="Fee recipient 0x123463a4b065722e99115d6c222f267d9cabb524 is not a checksum Ethereum address. The checksummed address is 0x123463a4B065722E99115D6c222f267d9cABb524 and will be used as the fee recipient. We recommend using a mixed-case address (checksum) to prevent spelling mistakes in your fee recipient Ethereum address" prefix=node time="2023-04-25 07:03:32" level=info msg="Default fee recipient is set to 0x123463a4B065722E99115D6c222f267d9cABb524, recipient may be overwritten from validator client and persist in db. Default fee recipient will be used as a fall back" prefix=node time="2023-04-25 07:03:32" level=info msg="Checking DB" database-path="/consensus/beacondata/beaconchaindata" prefix=node time="2023-04-25 07:03:32" level=info msg="Opening Bolt DB at /consensus/beacondata/beaconchaindata/beaconchain.db" prefix=db time="2023-04-25 07:03:32" level=info msg="Deposit contract: 0x4242424242424242424242424242424242424242" prefix=node time="2023-04-25 07:03:32" level=warning msg="No bootstrap addresses supplied" prefix=p2p time="2023-04-25 07:03:32" level=info msg="Running node with peer id of 16Uiu2HAm6fY71dXUtH1hLVatY1X9PPudorMGyoWuWuH6w9WTiVxe " prefix=p2p time="2023-04-25 07:03:32" level=info msg="gRPC server listening on port" address="0.0.0.0:4000" prefix=rpc time="2023-04-25 07:03:32" level=warning msg="You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=rpc time="2023-04-25 07:03:32" level=info msg="Starting beacon node" prefix=node version="Prysm/v4.0.3/08d6eccfb3bca915ff7d6369677b3f9ab9bacb55. Built at: 2023-04-22 00:50:48+00:00" time="2023-04-25 07:03:32" level=info msg="Waiting for state to be initialized" prefix=initial-sync time="2023-04-25 07:03:32" level=info msg="Blockchain data already exists in DB, initializing..." prefix=blockchain time="2023-04-25 07:03:32" level=info msg="Starting API middleware" prefix=gateway time="2023-04-25 07:03:32" level=info msg="Starting gRPC gateway" address="0.0.0.0:3500" prefix=gateway time="2023-04-25 07:03:32" level=info msg="Starting initial chain sync..." prefix=initial-sync time="2023-04-25 07:03:32" level=info msg="Started discovery v5" ENR="enr:-MK4QJwrEgTD0WwVFIDrpyWvxHXb6hxrcTDMV5JK6R4KZFwzLG7s1TqrkEqbyxkXgis9jBHMlp86wOF70DZ5RCD_J6uGAYe3OYRsh2F0dG5ldHOIAAAAAAAAAACEZXRoMpBa8xKTIAAAkv__gmlkgnY0gmlwhMCo4AOJc2VjcDI1NmsxoQKm_chY0ZiFBgpCb0XGqlg2kR6lGYdp4tcdFXh_7iHZs4hzeW5jbmV0cwCDdGNwgjLIg3VkcIIu4A" prefix=p2p time="2023-04-25 07:03:32" level=info msg="Node started p2p server" multiAddr="/ip4/192.168.224.3/tcp/13000/p2p/16Uiu2HAm6fY71dXUtH1hLVatY1X9PPudorMGyoWuWuH6w9WTiVxe" prefix=p2p time="2023-04-25 07:03:32" level=info msg="gRPC client connected to beacon node" addr="192.168.224.4:48122" prefix=rpc time="2023-04-25 07:03:47" level=info msg="Connected to new endpoint: http://geth:8551" prefix=powchain time="2023-04-25 07:04:32" level=info msg="Peer summary" activePeers=0 inbound=0 outbound=0 prefix=p2p time="2023-04-25 07:05:32" level=info msg="Peer summary" activePeers=0 inbound=0 outbound=0 prefix=p2p time="2023-04-25 07:05:52" level=warning msg="Execution client is not syncing" prefix=powchain

Validator Log: time="2023-04-25 07:03:32" level=warning msg="Running on custom Ethereum network specified in a chain configuration yaml file" prefix=flags time="2023-04-25 07:03:32" level=info msg="Checking DB" databasePath="/consensus/validatordata" prefix=node Adding optimizations for validator slashing protection 100% [=========] [0s:0s] time="2023-04-25 07:03:32" level=warning msg="You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=validator time="2023-04-25 07:03:32" level=info msg="Starting validator node" prefix=node version="Prysm/v4.0.3/08d6eccfb3bca915ff7d6369677b3f9ab9bacb55. Built at: 2023-04-22 02:02:24+00:00" time="2023-04-25 07:03:32" level=info msg="Syncing with beacon node to align on chain genesis info" prefix=validator time="2023-04-25 07:03:32" level=info msg="Beacon chain started" genesisTime=2023-04-25 02:18:13 +0000 UTC prefix=validator time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa99a76ed7796 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb89bebc69976 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa3a32b0f8b4d time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x88c141df77cd time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x81283b7a20e1 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xab0bdda0f85f time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x9977f1c8b731 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa8d4c7c27795 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa6d310dbbfab time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x9893413c0028 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x876dd4705157 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xaec922bd7a9b time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x9314c6de0386 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x903e2989e744 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x84398f539a64 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x872c61b4a7f8 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8f467e5723de time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8dde83069208 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xab8d3a9bcc16 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8d5d3672a233 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa1c76af1545d time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8dd74e1bb522 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x954eb88ed120 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xaf344fce60db time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xae241af60691 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x96746aaba64d time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb9d1d914df3d time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x963528adb532 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb245d63d3f9d time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa98ed496c2f4 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xae00fc3de831 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa4855c83d868 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa9cf360aa15f time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xaef9162ee6f2 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb7e6e187ed81 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x81054bd51ce5 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xaecc56f2b1c4 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x9243ef5ed3bd time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x925b1fb57c06 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x9648b83a4f09 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa34febc12af0 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb8cd1cef89aa time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x92a93728c252 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb7ee0ef26144 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xafc0fa2ed6a2 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa5869ba554d1 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x92f43d79d9f4 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x9698d9519a02 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa852816b8e46 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8a298ee1ac04 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xae4d49364e4a time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb397692ccbf4 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x87c9f7605d07 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb08f7feb8678 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xa48cc260df1d time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xac9f4df3f20a time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x94f0c8535601 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0xb5bb0162a4f2 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8826e820179f time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x92977e713966 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x91ae4686b0d2 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x8a0d24195510 time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x80a2be2c7dbc time="2023-04-25 07:03:32" level=info msg="Validating for public key" prefix=validator publicKey=0x86a73886aa01 time="2023-04-25 07:03:38" level=info msg="Waiting for beacon node to sync to latest chain head" prefix=validator time="2023-04-25 07:03:44" level=info msg="Waiting for beacon node to sync to latest chain head" prefix=validator time="2023-04-25 07:03:50" level=info msg="Waiting for beacon node to sync to latest chain head" prefix=validator time="2023-04-25 07:03:56" level=info msg="Waiting for beacon node to sync to latest chain head" prefix=validator

mo-husseini-vc commented 1 year ago

@LilkkJerry Getting the same issue. Did u manage to find a fix by any chance?

LilkkJerry commented 1 year ago

@LilkkJerry Getting the same issue. Did u manage to find a fix by any chance?

Not yet, hope that someone can give the guidance

aaroua5 commented 1 year ago

Also facing the same issue.

mantisco-kangjuhyup commented 1 year ago

Has anyone solved this problem? The same problem happens to me.

bxlkm commented 11 months ago

same issue

aerielabio commented 11 months ago

same issue

bilaljawed commented 11 months ago

Same issue

nisdas commented 11 months ago

@bilaljawed can you share your geth logs at this time ?

bilaljawed commented 11 months ago

@nisdas Thank you for the reply, the logs before it runs and after it has issues:

ethpos-ethereum-node-1 = geth

BEFORE

ethpos-validate-node-1       | time="2023-08-02 06:20:14" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=43 bitsCount=128 blockRoot=0xe1f175f02b56 prefix=validator slot=6 slotStartTime=2023-08-02 06:20:12 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=2.184953037s
ethpos-validate-node-1       | time="2023-08-02 06:20:14" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=43 bitsCount=128 blockRoot=0xe1f175f02b56 prefix=validator slot=6 slotStartTime=2023-08-02 06:20:12 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=2.195138169s
ethpos-validate-node-1       | time="2023-08-02 06:20:14" level=info msg="Submitted new attestations" AggregatorIndices=[38] AttesterIndices=[7 28 38 48 43] BeaconBlockRoot=0xe1f175f02b56 CommitteeIndex=0 Slot=6 SourceEpoch=0 SourceRoot=0x000000000000 TargetEpoch=0 TargetRoot=0xf18338e7a650 prefix=validator
ethpos-chain-beacon-1        | time="2023-08-02 06:20:15" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=189.467379ms slot=7
ethpos-chain-beacon-1        | time="2023-08-02 06:20:15" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690957194 latestValidTime=1690928522 prefix="rpc/validator"
ethpos-ethereum-node-1       | INFO [08-02|06:20:15.190] Stopping work on payload                 id=0xf7c02f59c88b8766 reason=delivery
ethpos-chain-beacon-1        | time="2023-08-02 06:20:15" level=info msg="Finished building block" prefix="rpc/validator" sinceSlotStartTime=194.907652ms slot=7 validator=53
ethpos-ethereum-node-1       | INFO [08-02|06:20:15.207] Imported new potential chain segment     number=6 hash=3c141a..e7e380 blocks=1 txs=0 mgas=0.000 elapsed="128.856µs" mgasps=0.000 dirty=0.00B
ethpos-ethereum-node-1       | INFO [08-02|06:20:15.228] Chain head was updated                   number=6 hash=3c141a..e7e380 root=815538..801014 elapsed="41.793µs"
ethpos-ethereum-node-1       | INFO [08-02|06:20:15.228] Starting work on payload                 id=0xce6caa4dfbf8b93d
ethpos-ethereum-node-1       | INFO [08-02|06:20:15.229] Updated payload                          id=0xce6caa4dfbf8b93d number=7 hash=4999b5..8ced1a txs=0 withdrawals=0 gas=0 fees=0 root=815538..801014 elapsed="42.984µs"
ethpos-chain-beacon-1        | time="2023-08-02 06:20:15" level=info msg="Synced new block" block=0x76a9547e... epoch=0 finalizedEpoch=0 finalizedRoot=0x00000000... prefix=blockchain slot=7
ethpos-chain-beacon-1        | time="2023-08-02 06:20:15" level=info msg="Finished applying state transition" payloadHash=0x3c141a0f4a59 prefix=blockchain slot=7 syncBitsCount=512 txCount=0
ethpos-validate-node-1       | time="2023-08-02 06:20:15" level=info msg="Submitted new block" blockRoot=0x76a9547eff9d fork=bellatrix gasUtilized=0 graffiti="" numAttestations=0 numDeposits=0 parentHash=0x574fed550cb7 payloadHash=0x3c141a0f4a59 prefix=validator pubKey=0xb08f7feb8678 slot=7 txCount=0
ethpos-validate-node-1       | time="2023-08-02 06:20:16" level=info msg="Submitted new sync message" blockRoot=0x76a9547eff9d prefix=validator slot=7 slotStartTime=2023-08-02 06:20:15 +0000 UTC timeSinceSlotStart=1.006564708s validatorIndex=5

Initial Docker compose

version: "1.0"
services:

  cleanup:
    image: alpine:latest
    volumes:
      - ./execution_data/:/execution_data/
    command: sh -c "rm -rf /execution_data/geth"

  genesis-generation:
    image: "gcr.io/prysmaticlabs/prysm/cmd/prysmctl:latest"
    command:
      - testnet
      - generate-genesis
      - --fork=bellatrix
      - --num-validators=64
      - --output-ssz=/consensus_data/genesis.ssz
      - --chain-config-file=/consensus_data/config.yml
      - --geth-genesis-json-in=/execution_data/genesis.json
      - --geth-genesis-json-out=/execution_data/genesis.json
    volumes:
      - ./consensus_data:/consensus_data
      - ./execution_data:/execution_data

  genesis-ethereum:
    image: "ethereum/client-go:latest"
    command: --datadir=/execution_data init /execution_data/genesis.json
    volumes:
      - ./execution_data:/execution_data
      - ./execution_data/genesis.json:/execution_data/genesis.json
    depends_on:
      genesis-generation:
        condition: service_completed_successfully

  ethereum-node:
    image: "ethereum/client-go:latest"
    command:
      - --http
      - --http.api=eth,engine,net,web3,txpool,debug
      - --http.addr=0.0.0.0
      - --http.corsdomain=*
      - --http.vhosts=*
      - --authrpc.vhosts=*
      - --authrpc.addr=0.0.0.0
      - --authrpc.port=8552
      - --authrpc.jwtsecret=/execution_data/jwtsecret
      - --datadir=/execution_data
      - --allow-insecure-unlock
      - --unlock=0x123463a4b065722e99115d6c222f267d9cabb524
      - --password=/execution_data/geth_password.txt
      - --nodiscover
      - --syncmode=full
      - --gcmode=archive
    ports:
      - 8552:8552
      - 8545:8545
    depends_on:
      genesis-ethereum:
        condition: service_completed_successfully
    volumes:
      - ./execution_data:/execution_data
      - ./execution_data/geth_password.txt:/execution_data/geth_password.txt
      - ./jwtsecret:/execution_data/jwtsecret

  chain-beacon:
    image: "gcr.io/prysmaticlabs/prysm/beacon-chain:latest"
    command:
      - --datadir=/consensus_data/beacondata
      - --min-sync-peers=0
      - --genesis-state=/consensus_data/genesis.ssz
      - --bootstrap-node=
      - --chain-config-file=/consensus_data/config.yml
      - --chain-id=32382
      - --rpc-host=0.0.0.0
      - --contract-deployment-block=0
      - --grpc-gateway-host=0.0.0.0
      - --execution-endpoint=http://ethereum-node:8552
      - --accept-terms-of-use
      - --jwt-secret=/execution_data/jwtsecret
      - --suggested-fee-recipient=0x123463a4b065722e99115d6c222f267d9cabb524
      - --rpc-port=4001
      - --enable-debug-rpc-endpoints
      - --force-clear-db
    ports:
      - 3500:3500
    depends_on:
      ethereum-node:
        condition: service_started
    volumes:
      - ./consensus_data:/consensus_data
      - ./jwtsecret:/execution_data/jwtsecret

  validate-node:
    image: "gcr.io/prysmaticlabs/prysm/validator:latest"
    command:
      - --beacon-rpc-provider=chain-beacon:4001
      - --datadir=/consensus_data/validatordata
      - --accept-terms-of-use
      - --interop-num-validators=64
      - --interop-start-index=0
      - --chain-config-file=/consensus_data/config.yml
      - --force-clear-db
    depends_on:
      chain-beacon:
        condition: service_started
    volumes:
      - ./consensus_data:/consensus_data

AFTER

ethpos-validate-node-1  | time="2023-08-02 06:22:56" level=info msg="Submitted new attestations" AggregatorIndices=[48] AttesterIndices=[3 31 48 2 12] BeaconBlockRoot=0xf18338e7a650 CommitteeIndex=0 Slot=60 SourceEpoch=0 SourceRoot=0x000000000000 TargetEpoch=5 TargetRoot=0xf18338e7a650 prefix=validator
ethpos-chain-beacon-1   | time="2023-08-02 06:22:57" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=232.651173ms slot=61
ethpos-chain-beacon-1   | time="2023-08-02 06:22:57" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690957194 latestValidTime=1690928522 prefix="rpc/validator"
ethpos-ethereum-node-1  | INFO [08-02|06:22:57.270] Ignoring beacon update to old head       number=0 hash=42c600..aa2afa age=3m3s  have=6
ethpos-chain-beacon-1   | time="2023-08-02 06:22:57" level=error msg="Could not pack deposits and attestations" error="rpc error: code = Internal desc = Could not get ETH1 deposits: could not fetch eth1data height: could not query block with given hash: Post "http://ethereum-node:8552": context canceled" prefix="rpc/validator"
ethpos-validate-node-1  | time="2023-08-02 06:22:57" level=error msg="Failed to request block from beacon node" blockSlot=61 error="rpc error: code = Unknown desc = could not build block in parallel: rpc error: code = Internal desc = Could not get local payload: nil payload with block hash: 0x42c6007eb4472220a3eb0848a6186a912af9b02ce19cda49cc25db9bbcaa2afa" prefix=validator pubKey=0xb9d1d914df3d
ethpos-validate-node-1  | time="2023-08-02 06:22:58" level=info msg="Submitted new sync message" blockRoot=0xf18338e7a650 prefix=validator slot=61 slotStartTime=2023-08-02 06:22:57 +0000 UTC timeSinceSlotStart=1.006650611s validatorIndex=63
ethpos-validate-node-1  | time="2023-08-02 06:22:58" level=info msg="Submitted new sync message" blockRoot=0xf18338e7a650 prefix=validator slot=61 slotStartTime=2023-08-02 06:22:57 +0000 UTC timeSinceSlotStart=1.006880763s validatorIndex=27
ethpos-validate-node-1  | time="2023-08-02 06:22:58" level=info msg="Submitted new sync message" blockRoot=0xf18338e7a650 prefix=validator slot=61 slotStartTime=2023-08-02 06:22:57 +0000 UTC timeSinceSlotStart=1.007017923s validatorIndex=49
ethpos-chain-beacon-1   | time="2023-08-02 06:23:00" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=216.622066ms slot=62
ethpos-chain-beacon-1   | time="2023-08-02 06:23:00" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690957194 latestValidTime=1690928522 prefix="rpc/validator"
ethpos-ethereum-node-1  | INFO [08-02|06:23:00.253] Ignoring beacon update to old head       number=0 hash=42c600..aa2afa age=3m6s  have=6
ethpos-chain-beacon-1   | time="2023-08-02 06:23:00" level=error msg="Could not pack deposits and attestations" error="rpc error: code = Internal desc = Could not get ETH1 deposits: could not fetch eth1data height: could not query block with given hash: Post "http://ethereum-node:8552": context canceled" prefix="rpc/validator"
ethpos-validate-node-1  | time="2023-08-02 06:23:00" level=error msg="Failed to request block from beacon node" blockSlot=62 error="rpc error: code = Unknown desc = could not build block in parallel: rpc error: code = Internal desc = Could not get local payload: nil payload with block hash: 0x42c6007eb4472220a3eb0848a6186a912af9b02ce19cda49cc25db9bbcaa2afa" prefix=validator pubKey=0x88c141df77cd

Re running docker compose:

version: "1.0"
services:
  ethereum-node:
    image: "ethereum/client-go:latest"
    command:
      - --http
      - --http.api=eth,engine,net,web3,txpool,debug
      - --http.addr=0.0.0.0
      - --http.corsdomain=*
      - --http.vhosts=*
      - --authrpc.vhosts=*
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/execution_data/jwtsecret
      - --authrpc.port=8552
      - --datadir=/execution_data
      - --allow-insecure-unlock
      - --unlock=0x123463a4b065722e99115d6c222f267d9cabb524
      - --password=/execution_data/geth_password.txt
      - --nodiscover
      - --syncmode=full
      - --gcmode=archive
    ports:
      - 8552:8552
      - 8545:8545
    volumes:
      - ./execution_data:/execution_data
      - ./execution_data/geth_password.txt:/execution_data/geth_password.txt
      - ./jwtsecret:/execution_data/jwtsecret

  chain-beacon:
    image: "gcr.io/prysmaticlabs/prysm/beacon-chain:latest"
    command:
      - --datadir=/consensus_data/beacondata
      - --min-sync-peers=0
      - --genesis-state=/consensus_data/genesis.ssz
      - --bootstrap-node=
      - --chain-config-file=/consensus_data/config.yml
      - --chain-id=32382
      - --rpc-host=0.0.0.0
      - --contract-deployment-block=0
      - --grpc-gateway-host=0.0.0.0
      - --execution-endpoint=http://ethereum-node:8552
      - --accept-terms-of-use
      - --jwt-secret=/execution_data/jwtsecret
      - --suggested-fee-recipient=0x123463a4b065722e99115d6c222f267d9cabb524
      - --rpc-port=4001
      - --enable-debug-rpc-endpoints
    ports:
      - 3500:3500
    depends_on:
      ethereum-node:
        condition: service_started
    volumes:
      - ./consensus_data:/consensus_data
      - ./jwtsecret:/execution_data/jwtsecret

  validate-node:
    image: "gcr.io/prysmaticlabs/prysm/validator:latest"
    command:
      - --beacon-rpc-provider=chain-beacon:4001
      - --datadir=/consensus_data/validatordata
      - --accept-terms-of-use
      - --interop-num-validators=64
      - --interop-start-index=0
      - --chain-config-file=/consensus_data/config.yml
    depends_on:
      chain-beacon:
        condition: service_started
    volumes:
      - ./consensus_data:/consensus_data

Config.yml

CONFIG_NAME: interop
PRESET_BASE: interop

# Genesis
GENESIS_FORK_VERSION: 0x20000089

# Altair
ALTAIR_FORK_EPOCH: 0
ALTAIR_FORK_VERSION: 0x20000090

# Merge
BELLATRIX_FORK_EPOCH: 0
BELLATRIX_FORK_VERSION: 0x20000091
TERMINAL_TOTAL_DIFFICULTY: 0

CAPELLA_FORK_EPOCH: 2
CAPELLA_FORK_VERSION: 0x20000092

# Time parameters
SECONDS_PER_SLOT: 3
SLOTS_PER_EPOCH: 12

# Deposit contract
DEPOSIT_CONTRACT_ADDRESS: 0x4242424242424242424242424242424242424242
nisdas commented 11 months ago

I think this only works if you wait for the chain to finalized before you restart @bilaljawed . It appears you restart too early in this setup. Look for the FinalizedEpoch logs in prysm to see if you are finalizing the chain

bilaljawed commented 11 months ago

@nisdas I have even tried it after running it for 2-3 hours, I don't see FinalizedEpoch

Any more info on this? Do we need multiple validators/beacons to finalize chain?

nisdas commented 11 months ago

You can add --verbosity=debug to the beacon node, it will log out debug logs which will have it

bilaljawed commented 11 months ago

@nisdas I see FinalizedEpoch=4

2023-08-02 15:37:53 time="2023-08-02 11:37:53" level=debug msg="Broadcasting aggregated attestation and proof" aggregatedCount=5 committeeIndex=0 prefix="rpc/validator" slot=70 validatorIndex=37
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=194.557317ms slot=71
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=debug msg="Computed state root" beaconStateRoot=0x8078182524ae3d8d6b900dd71c180b6c4ad701f52d894bde667dea4a52fa5e13 prefix="rpc/validator"
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=info msg="Finished building block" prefix="rpc/validator" sinceSlotStartTime=207.379566ms slot=71 validator=26
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=debug msg="Broadcasting block" blockRoot=c35a2fe68d9f17fb2cb7a45119be409a1c0cb551c4245b6c02df4b1e61cafa76 prefix="rpc/validator"
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=debug msg="Synced new block" block=0xc35a2fe6... chainServiceProcessedTime=43.554077ms deposits=0 epoch=5 finalizedEpoch=3 finalizedRoot=0x03479d70... justifiedEpoch=4 justifiedRoot=0x1801c55c... parentRoot=0x46fba0a6... prefix=blockchain sinceSlotStartTime=264.303301ms slot=71 slotInEpoch=11 version=capella
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=debug msg="Synced new payload" blockHash=0x53ff252dc95c blockNumber=70 blsToExecutionChanges=0 gasUtilized=0.00 parentHash=0xe56407b744c4 prefix=blockchain withdrawals=0
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=info msg="Finished applying state transition" attestations=1 payloadHash=0x53ff252dc95c prefix=blockchain slot=71 syncBitsCount=512 txCount=0
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=debug msg="Block proposal received via RPC" prefix="rpc/validator" slot=71
2023-08-02 15:37:54 time="2023-08-02 11:37:54" level=warning msg="Could not update committee cache" error="context canceled" prefix=blockchain
2023-08-02 15:37:55 time="2023-08-02 11:37:55" level=debug msg="Latest eth1 chain event" blockHash=0x53ff252dc95c6edef3d911df12c1a834b17de770ff3e0ff35a479cebcb2fb2c3 blockNumber=70 prefix=powchain
2023-08-02 15:37:56 time="2023-08-02 11:37:56" level=debug msg="Broadcasting aggregated attestation and proof" aggregatedCount=6 committeeIndex=0 prefix="rpc/validator" slot=71 validatorIndex=63
2023-08-02 15:37:57 time="2023-08-02 11:37:57" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=221.549274ms slot=72
2023-08-02 15:37:57 time="2023-08-02 11:37:57" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:37:57 time="2023-08-02 11:37:57" level=debug msg="Computed state root" beaconStateRoot=0x72ccc8d838a9e33e3df4970bf9cf2448da70b68624803f934453e099e31355fa prefix="rpc/validator"
2023-08-02 15:37:57 time="2023-08-02 11:37:57" level=info msg="Finished building block" prefix="rpc/validator" sinceSlotStartTime=229.282076ms slot=72 validator=40
2023-08-02 15:37:57 time="2023-08-02 11:37:57" level=debug msg="Broadcasting block" blockRoot=5d6937597e546799f71d7fcbaf7dca45c29168b5b995da591909962ef2ed716e prefix="rpc/validator"
2023-08-02 15:37:57 time="2023-08-02 11:37:57" level=debug msg="Synced new block" block=0x5d693759... chainServiceProcessedTime=65.437574ms deposits=0 epoch=6 **finalizedEpoch=4** finalizedRoot=0x1801c55c... justifiedEpoch=5 justifiedRoot=0xf19f6496... parentRoot=0xc35a2fe6... prefix=blockchain sinceSlotStartTime=308.81043ms slot=72 slotInEpoch=0 version=capella
bilaljawed commented 11 months ago

After restarting beacon:

2023-08-02 15:39:27 time="2023-08-02 11:39:27" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/bls_to_execution_change/ssz_snappy"
2023-08-02 15:39:27 time="2023-08-02 11:39:27" level=info msg="Started discovery v5" ENR="enr:-MK4QEqWQlkA2nH5GVTHKeMjFn2ghKnSZ1c-Q6hPlVi5ZqpPcqoR6mRLSlkk_h_Fuoj1VJ2DwutdnAkQminWKi4LldGGAYm2C7Ouh2F0dG5ldHOIAAAAAAAAAACEZXRoMpBa8xKTIAAAkv__________gmlkgnY0gmlwhKwUAAOJc2VjcDI1NmsxoQMXmGBX3zzy1TakRd2UL9-YxnGGuLwrqBWExndYelDyqIhzeW5jbmV0cwCDdGNwgjLIg3VkcIIu4A" prefix=p2p
2023-08-02 15:39:27 time="2023-08-02 11:39:27" level=info msg="Node started p2p server" multiAddr="/ip4/172.20.0.3/tcp/13000/p2p/16Uiu2HAmEF6VNu69LxZu32Hp2EmcBTthg1yUL8SrdvMKzbAQ2iS3" prefix=p2p
2023-08-02 15:39:27 time="2023-08-02 11:39:27" level=info msg="gRPC client connected to beacon node" addr="172.20.0.4:33070" prefix=rpc
2023-08-02 15:39:28 time="2023-08-02 11:39:28" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:30 time="2023-08-02 11:39:30" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:30 time="2023-08-02 11:39:30" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=248.076819ms slot=103
2023-08-02 15:39:30 time="2023-08-02 11:39:30" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:30 time="2023-08-02 11:39:30" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:39:31 time="2023-08-02 11:39:31" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:31 time="2023-08-02 11:39:31" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x17f7e00ec33c51af93d0ebb76ef9a229363ab56cba58e0a3d4b24eeb56fd3f03 prefix=blockchain slot=72
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/sync_committee_0/ssz_snappy"
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=warning msg="numPerSlot is 0, skipping initializing topic scoring" prefix=p2p
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_58/ssz_snappy"
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=204.468814ms slot=104
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:33 time="2023-08-02 11:39:33" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:39:34 time="2023-08-02 11:39:34" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:34 time="2023-08-02 11:39:34" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x17f7e00ec33c51af93d0ebb76ef9a229363ab56cba58e0a3d4b24eeb56fd3f03 prefix=blockchain slot=72
2023-08-02 15:39:36 time="2023-08-02 11:39:36" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:36 time="2023-08-02 11:39:36" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=202.111027ms slot=105
2023-08-02 15:39:36 time="2023-08-02 11:39:36" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:36 time="2023-08-02 11:39:36" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:39:37 time="2023-08-02 11:39:37" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:37 time="2023-08-02 11:39:37" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x17f7e00ec33c51af93d0ebb76ef9a229363ab56cba58e0a3d4b24eeb56fd3f03 prefix=blockchain slot=72
2023-08-02 15:39:39 time="2023-08-02 11:39:39" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:39 time="2023-08-02 11:39:39" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=214.889425ms slot=106
2023-08-02 15:39:39 time="2023-08-02 11:39:39" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:39 time="2023-08-02 11:39:39" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:39:40 time="2023-08-02 11:39:40" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:40 time="2023-08-02 11:39:40" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x17f7e00ec33c51af93d0ebb76ef9a229363ab56cba58e0a3d4b24eeb56fd3f03 prefix=blockchain slot=72
2023-08-02 15:39:42 time="2023-08-02 11:39:42" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:42 time="2023-08-02 11:39:42" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=227.076033ms slot=107
2023-08-02 15:39:42 time="2023-08-02 11:39:42" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:42 time="2023-08-02 11:39:42" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:39:42 time="2023-08-02 11:39:42" level=info msg="Connected to new endpoint: http://ethereum-node:8551" prefix=powchain
2023-08-02 15:39:42 time="2023-08-02 11:39:42" level=error msg="Beacon node is not respecting the follow distance" prefix=powchain
2023-08-02 15:39:43 time="2023-08-02 11:39:43" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:43 time="2023-08-02 11:39:43" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x17f7e00ec33c51af93d0ebb76ef9a229363ab56cba58e0a3d4b24eeb56fd3f03 prefix=blockchain slot=72
2023-08-02 15:39:45 time="2023-08-02 11:39:45" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:45 time="2023-08-02 11:39:45" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=261.66693ms slot=108
2023-08-02 15:39:45 time="2023-08-02 11:39:45" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:45 time="2023-08-02 11:39:45" level=warning msg="voting period before genesis + follow distance, using eth1data from head" genesisTime=1690976061 latestValidTime=1690947389 prefix="rpc/validator"
2023-08-02 15:39:46 time="2023-08-02 11:39:46" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:46 time="2023-08-02 11:39:46" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x17f7e00ec33c51af93d0ebb76ef9a229363ab56cba58e0a3d4b24eeb56fd3f03 prefix=blockchain slot=72
2023-08-02 15:39:48 time="2023-08-02 11:39:48" level=error msg="Could not compute head from new attestations" error="0x713365b20efeb1791d899a3e82ef6eaf53a840b385d5de631777d6056dfac850: unknown justified root" prefix=blockchain
2023-08-02 15:39:48 time="2023-08-02 11:39:48" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=222.194889ms slot=109

GETH:

2023-08-02 15:39:30 INFO [08-02|11:39:30.279] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m33s have=98
2023-08-02 15:39:31 INFO [08-02|11:39:31.035] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m34s have=98
2023-08-02 15:39:33 INFO [08-02|11:39:33.233] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m36s have=98
2023-08-02 15:39:34 INFO [08-02|11:39:34.024] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m37s have=98
2023-08-02 15:39:36 INFO [08-02|11:39:36.237] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m39s have=98
2023-08-02 15:39:37 INFO [08-02|11:39:37.027] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m40s have=98
2023-08-02 15:39:39 INFO [08-02|11:39:39.237] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m42s have=98
2023-08-02 15:39:40 INFO [08-02|11:39:40.042] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m43s have=98
2023-08-02 15:39:42 INFO [08-02|11:39:42.247] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m45s have=98
2023-08-02 15:39:43 INFO [08-02|11:39:43.058] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m46s have=98
2023-08-02 15:39:45 INFO [08-02|11:39:45.283] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m48s have=98
2023-08-02 15:39:46 INFO [08-02|11:39:46.027] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m49s have=98
2023-08-02 15:39:48 INFO [08-02|11:39:48.239] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m51s have=98
2023-08-02 15:39:49 INFO [08-02|11:39:49.032] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m52s have=98
2023-08-02 15:39:51 INFO [08-02|11:39:51.224] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m54s have=98
2023-08-02 15:39:52 INFO [08-02|11:39:52.029] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m55s have=98
2023-08-02 15:39:54 INFO [08-02|11:39:54.249] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m57s have=98
2023-08-02 15:39:55 INFO [08-02|11:39:55.032] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=1m58s have=98
2023-08-02 15:39:57 INFO [08-02|11:39:57.240] Ignoring beacon update to old head       number=71 hash=17f7e0..fd3f03 age=2m    have=98
indowebdeveloper commented 11 months ago

same issue here

nisdas commented 11 months ago

@bilaljawed Is geth not being restarted ?

bilaljawed commented 11 months ago

@bilaljawed Is geth not being restarted ?

It is, the logs are shared in previous comment

neodiz commented 11 months ago

Seeing the same problem. I also see a memory leak problem in prysm. I launched the application on the server with 32GB RAM and the next day the server stopped responding. Only a restart helped.

nisdas commented 11 months ago

@neodiz https://github.com/OffchainLabs/eth-pos-devnet/issues/23#issuecomment-1588548599 This should fix the memory usage issue

nisdas commented 11 months ago

On the issues with the restart, it appears geth does not allow updates to an old head on restart, we start from the previous finalized checkpoint and geth cannot build a payload with it.

neodiz commented 11 months ago

@bxlkm how fix it ?

neodiz commented 11 months ago

Create private network on 2 signers(beacon chain + geth + validator). Start first signer using the same configuration in issue. And signer2 which connects to signer1. Time is running. If you restart signer1, the network dies and cannot be started.

DenkevichOleg commented 10 months ago

@nisdas hi, so issue on geth side? wich execution client should be used to avoid this? should someone create issue about this in geth repo?

nisdas commented 10 months ago

I suspect there might be issues with other execution clients here too, but from my understanding geth does not allow building a payload from too far back here. This is most likely a safety mechanism, you could try opening up an issue with the geth repo to get their thoughts but I would not hold too much hope as there is most likely a good reason why it can't be done. One way to resolve this in prysm would be to start from the previously saved head rather than the finalized checkpoint, however I do not have the bandwidth to implement this right now.

rauljordan commented 10 months ago

Hi @LilkkJerry you will need to restart the chain from scratch. Try make clean and docker compose up -d

lyfsn commented 8 months ago

@LilkkJerry @bilaljawed

Hello, may I ask how you resolved this issue afterward?

hentai8 commented 7 months ago

Hi @LilkkJerry you will need to restart the chain from scratch. Try make clean and docker compose up -d

Hi ! If I don't want to clean my chain data, how can i restart?

sergey-msu commented 1 month ago

Any updates here? Getting the same issue @rauljordan Is it possible to restart the node without clean all the data?

lyfsn commented 1 month ago

This problem is caused by the Prysm client. There is no resolution if a network starts from block 0. I suggest using another consensus layer client such as Lighthouse as a starter for a new network. Also, I recommend a great tool to start a network quickly: https://github.com/kurtosis-tech/ethereum-package