Network is hitting the terminal total difficulty, but lighthouse is not activating.

entertainment-veks commented 1 month ago

Description

I'm trying to upgrade my private network from POW to POS. I'm using geth as EL and lighthouse as CL. I set up terminal total difficulty for all nodes and started waiting. Once I reach terminal total difficulty geth started sending such a logs:

WARN [08-23|12:38:19.605] Local chain is post-merge, waiting for beacon client sync switch-over...

Seems as expected, but lighthouse is not activating, so network is not upgrading to POS.

Configuration

geth-bootnode:
    hostname: geth-bootnode
    container_name: geth-bootnode
    env_file:
      - .env
    build:
      context: .
      dockerfile: Dockerfile.geth
      args:
        - ACCOUNT_PASSWORD="${ACCOUNT_PASSWORD}"
    command:
      - --bootnodes=enode://XXX
      - --networkid=${NETWORK_ID}
    volumes:
      - geth-bootnode:/root/.ethereum
    ports:
      - '30303:30303/tcp'
      - '30303:30303/udp'

geth-rpc:
    hostname: geth-rpc
    container_name: geth-rpc
    env_file:
      - .env
    build:
      context: .
      dockerfile: Dockerfile.geth
      args:
        - ACCOUNT_PASSWORD="${ACCOUNT_PASSWORD}"
    depends_on:
      - geth-bootnode
    command:
      - --bootnodes=enode://4244b36585419268be23058d4a879fd2842545ce6bb971027841f52e816418625bcc0e725820bfebfee0cee6607a3e7fb049a58448b90df76a47508d9455dcbe@geth-bootnode:30303
      - --allow-insecure-unlock
      - --http
      - --http.addr=0.0.0.0
      - --http.api=admin,debug,eth,miner,net,personal,txpool,web3
      - --http.corsdomain=*
      - --http.vhosts=*
      - --ws
      - --ws.origins=*
      - --ws.addr=0.0.0.0
      - --authrpc.jwtsecret=/secrets/secret.hex
      - --authrpc.addr=0.0.0.0
      - --authrpc.port=8551
      - --authrpc.vhosts=*
      - --ws.api=eth
      - --syncmode=full
      - --gcmode=archive
      - --networkid=${NETWORK_ID}
    ports:
     - '8545:8545'
     - '8546:8546'
    volumes:
      - geth-rpc:/root/.ethereum
    restart: always

lighthouse-beacon:
    hostname: lighthouse-beacon
    container_name: lighthouse-beacon
    build:
      context: .
      dockerfile: Dockerfile.lighthouse
    command:
      - beacon_node
      - --datadir=/lighthouse/node1
      - --testnet-dir=/consensus
      - --execution-endpoint=http://geth-rpc:8551
      - --execution-jwt=/secrets/secret.hex
      - --enable-private-discovery
      - --staking
      - --enr-address=127.0.0.1
      - --enr-udp-port=9000
      - --enr-tcp-port=9000
      - --port=9000
      - --http
      - --http-port=4000
      - --disable-packet-filter
      - --eth1
    ports:
      - '5052:5052'
    volumes:
      - ./consensus:/consensus
      - lighthouse-beacon:/lighthouse

I'm using custom images, but there is nothing really important:

Dockerfile.lighthouse:

FROM sigp/lighthouse:v3.1.2

ADD ./secrets /secrets

ENTRYPOINT ["lighthouse"]

Dockerfile.geth:

FROM ethereum/client-go:v1.11.6

ARG ACCOUNT_PASSWORD

WORKDIR /eth

ADD ./genesis.json /eth
ADD ./secrets /secrets

RUN geth init /eth/genesis.json \
    && rm -f ~/.ethereum/geth/nodekey \
    && echo ${ACCOUNT_PASSWORD} > /eth/password \
    && geth account new --password /eth/password \
    && rm -f /eth/password

ENTRYPOINT ["geth"]

Config, that contains in --testnet-dir:

PRESET_BASE: mainnet
TERMINAL_TOTAL_DIFFICULTY: '5425000000000'
TERMINAL_BLOCK_HASH: '0x0000000000000000000000000000000000000000000000000000000000000000'
TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH: '18446744073709551615'
SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY: '128'
MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: '192'
MIN_GENESIS_TIME: '1722859200'
GENESIS_FORK_VERSION: '0x70000069'
GENESIS_DELAY: '120'
ALTAIR_FORK_VERSION: '0x01000001'
ALTAIR_FORK_EPOCH: '1'
BELLATRIX_FORK_VERSION: '0x02000001'
BELLATRIX_FORK_EPOCH: '2'
CAPELLA_FORK_VERSION: '0xffffffff'
CAPELLA_FORK_EPOCH: '18446744073709551615'
DENEB_FORK_VERSION: '0xffffffff'
DENEB_FORK_EPOCH: '18446744073709551615'
SECONDS_PER_SLOT: '12'
SECONDS_PER_ETH1_BLOCK: '12'
MIN_VALIDATOR_WITHDRAWABILITY_DELAY: '256'
SHARD_COMMITTEE_PERIOD: '256'
ETH1_FOLLOW_DISTANCE: '1'
SUBNETS_PER_NODE: '2'
INACTIVITY_SCORE_BIAS: '4'
INACTIVITY_SCORE_RECOVERY_RATE: '16'
EJECTION_BALANCE: '16000000000'
MIN_PER_EPOCH_CHURN_LIMIT: '4'
MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT: '8'
CHURN_LIMIT_QUOTIENT: '32'
PROPOSER_SCORE_BOOST: '40'
DEPOSIT_CHAIN_ID: '12020498'
DEPOSIT_NETWORK_ID: '12020498'
DEPOSIT_CONTRACT_ADDRESS: 0xc4648117106a08c3040b8573eb3d8e242c9726be
GOSSIP_MAX_SIZE: '10485760'
MAX_REQUEST_BLOCKS: '1024'
EPOCHS_PER_SUBNET_SUBSCRIPTION: '256'
MIN_EPOCHS_FOR_BLOCK_REQUESTS: '33024'
MAX_CHUNK_SIZE: '10485760'
TTFB_TIMEOUT: '5'
RESP_TIMEOUT: '10'
ATTESTATION_PROPAGATION_SLOT_RANGE: '32'
MAXIMUM_GOSSIP_CLOCK_DISPARITY_MILLIS: '500'
MESSAGE_DOMAIN_INVALID_SNAPPY: '0x00000000'
MESSAGE_DOMAIN_VALID_SNAPPY: '0x01000000'
ATTESTATION_SUBNET_EXTRA_BITS: '0'
ATTESTATION_SUBNET_PREFIX_BITS: '6'
MAX_REQUEST_BLOCKS_DENEB: '128'
MAX_REQUEST_BLOB_SIDECARS: '768'
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: '4096'
BLOB_SIDECAR_SUBNET_COUNT: '6'

Geth genesis:

{
  "config": {
    "chainId": 12020498,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "constantinopleBlock": 0,
    "petersburgBlock": 0,
    "istanbulBlock": 0,
    "berlinBlock": 0,
    "londonBlock": 4033065,
    "arrowGlacierBlock": 4033065,
    "terminalTotalDifficulty": 5425000000000,
    "ethash": {}
  },
  "nonce": "0x0",
  "difficulty": "0x0",
  "timestamp": "0x63690E50",
  "gasLimit": "0x8000000",
  "alloc": {
    "0x65490838F9d353246Da7C16D358AEf5bddf5c0bc": {
      "balance": "1000000000000000000000000000000"
    }
  },
  "number": "0x0",
  "gasUsed": "0x0",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
}

Lighthouse logs:

lighthouse-beacon  | Aug 23 12:45:19.002 INFO Ready for the merge                     current_difficulty: 5425003009187, terminal_total_difficulty: 5425000000000, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:26.796 WARN Execution endpoint is not synced        action: trying fallback, last_seen_block_unix_timestamp: 1724357260, endpoint: http://geth-rpc:8551/, auth=true, service: deposit_contract_rpc
lighthouse-beacon  | Aug 23 12:45:26.796 ERRO No synced execution endpoint            advice: ensure you have an execution node configured via --execution-endpoint or if pre-merge, --eth1-endpoints, service: deposit_contract_rpc
lighthouse-beacon  | Aug 23 12:45:26.796 ERRO Error updating deposit contract cache   error: "All fallbacks errored: http://geth-rpc:8551/, auth=true => EndpointError(FarBehind)", retry_millis: 60000, service: deposit_contract_rpc
lighthouse-beacon  | Aug 23 12:45:31.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:31.000 INFO Searching for peers                     current_slot: 71786, head_slot: 0, finalized_epoch: 0, finalized_root: 0x980e…b4f6, peers: 0, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:31.000 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:31.002 INFO Ready for the merge                     current_difficulty: 5425003009187, terminal_total_difficulty: 5425000000000, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:43.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:43.001 INFO Searching for peers                     current_slot: 71787, head_slot: 0, finalized_epoch: 0, finalized_root: 0x980e…b4f6, peers: 0, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:43.001 WARN Syncing deposit contract block cache    est_blocks_remaining: initializing deposits, service: slot_notifier
lighthouse-beacon  | Aug 23 12:45:43.004 INFO Ready for the merge                     current_difficulty: 5425003009187, terminal_total_difficulty: 5425000000000, service: slot_notifier

Could you help me please to figure out that problem and help me to upgrade my network to the POS? Thanks!

michaelsproul commented 1 month ago

As far as I know Geth doesn't support merging any more. We are planning to delete merge support from Lighthouse soon too.

We can't help, sorry, it would be too time consuming and not really relevant to Lighthouse's development. Deleting the merge logic will be a great relief for us, and J cannot encourage you to rely on it.

entertainment-veks commented 1 month ago

@michaelsproul You are right, latest versions geth and lighthouse are not supporting Merge, thats why I use sigp/lighthouse:v3.1.2 and ethereum/client-go:v1.11.6. I'm not requesting any kind of fixes etc, just looking for and advice, how to solve my problem. Thanks!

michaelsproul commented 1 month ago

I can give a few tips but I can't commit to helping debug this because debugging private testnets remotely is very time consuming and of no benefit to me

Make sure CL and EL both have peers and consider themselves "synced"
Make sure Geth is running with --syncmode=full if it is not already synced
Read the logs and try to find the point in the code where the log comes from. You'll need to look at Geth logs and Lighthouse logs.

If you're not attached to your existing network (but I guess you are) you could also just launch a new network that is merged from the beginning.

I can't help you more than that, sorry. Good luck.

sigp / lighthouse