OpenEthereum: Peers do not join bootnode

[jkuhlenkamp]

We try to make an OpenEthereum node join another OpenEthereum node in the same network. We use docker-compose. This deploys two containers: bootnode and joinnode.

Problem

The bootnode and joinnode do not recognize each other as peers.

Current implementation

Before starting OpenEthereum on the joinnode, the init.sh script executes in the container. The script recognizes the environment variable BOOTNODE_IP and the optional BOOTNODE_PORT. The init.sh script calls the get_enode function. The get_enodefunction polls the bootnode for its enode. If successful, the get_enodefunction sets the global ENODE and BOOTNODE variables. In the node config template is created based on a template. The content of the BOOTNODE environment variable is inserted at the { BOOTNODE} placeholder. OpenEthereum starts on the joinnode. The joinnode tries to connect to the bootnode under the given bootnode.

What is going wrong here?

[jkuhlenkamp]

Thesis 1:

• What: Both containers are not visible on the network.
• Result: Invalid
• Proof: The joinnode successfully optains the enode form the bootnode via a RPC call.

[jkuhlenkamp]

Thesis 2:

• What: The containers use the wrong ports for intra peer communication.
• Result: Invalid
• Proof: The port for intra peer communication is set in the node config file under the flag port in group [network]. The default is 30303. The node config template. Specifies exactly this for both containers. As an alternative the docker-compose file can specify a another port mapping.

For the bootnode it is: This is correct.

For the joinnode it is:

The port mapping specifies host_port:container_port (see docs). Thus, no port collision on the host. Both containers use port 30303internally. This seems fine.

For further validation we obtain the network port from both containers via RPC from the container host : This seems fine.

[jkuhlenkamp]

Thesis 3

• What: The joinnode uses the wrong node for the bootnode.
• Result: Invalid. The joinode obtains the correct enode value from the bootnode.
• Proof:

The bootnode outputs its enode, i.e., public URL, at startup: enode://f5b49083d3e71419f1daf5fabca49c1c8e49911621c80b5945aaad3879d5482ab4cbf5905c6cdbb567fa01f901e94bfbc1250defbe88408a69ef15726f7035af@172.16.0.10:30303

We compare this value with the node config used at the joinnode: enode://f5b49083d3e71419f1daf5fabca49c1c8e49911621c80b5945aaad3879d5482ab4cbf5905c6cdbb567fa01f901e94bfbc1250defbe88408a69ef15726f7035af@172.16.0.10:30303

[jkuhlenkamp]

Thesis 4

• What: The field bootnode from group network in the node config of the joinnode is not processed correctly.
• Result: TODO
• Proof: We try to set the CLI option --bootnodesthat takes precedence over the node config.

[jkuhlenkamp]

We need more data. Thus, I increase logging level to trace in node config.

[misc]
logging  = "trace"                          
log_file = "/home/blogpv/blogpv.log"  

Scanning the log provides an interesting insight. The joinnode seems to use a different genesis block. Apparently, this hinders a connection.

joinnode_1  | 2020-07-24 08:17:53 UTC IO Worker #1 TRACE sync  New peer 0 (protocol: 4, network: 8995, difficulty: Some(131072), latest:0xb2f4…8c22, genesis:0xb2f4…8c22, snapshot:Some(0), private_tx_enabled:false, client_version: OpenEthereum/node0/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1)
joinnode_1  | 2020-07-24 08:17:53 UTC  TRACE mio::deprecated::event_loop  event=Event { kind: Writable, token: Token(0) }; idx=0
joinnode_1  | 2020-07-24 08:17:53 UTC IO Worker #1 TRACE sync  Peer 0 genesis hash mismatch (ours: 0xb4eb…80e8, theirs: 0xb2f4…8c22)

Proposed solution: Ensure same genesis block on bootnode and joinnode.

[jkuhlenkamp]

The genesis block is defined in the chain config file. Here is an excerpt from the genesis property and all properties that can vary of the chain config file:

{
  "name": "{ CHAIN_NAME }",
  "engine": {
    "authorityRound": {
      "params": {
        "stepDuration": 5,
        "blockReward": "0x38D7EA4C68000",
        "maximumUncleCountTransition": 0,
        "maximumUncleCount": 0,
        "validators" : {
          "list": ["{ ACCOUNT_ADDRESS }"]
        }
      }
    }
  },
  "params": {
    "gasLimitBoundDivisor": "0x400",
    "maximumExtraDataSize": "0x20",
    "minGasLimit": "0x1388",
    "networkID": "0x2323",
    "eip140Transition": "0x0",
    "eip211Transition": "0x0",
    "eip214Transition": "0x0",
    "eip658Transition": "0x0"
  },
  "genesis": {
    "seal": {
      "authorityRound": {
        "step": "0x0",
        "signature": "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
      }
    },
    "difficulty": "0x20000",
    "gasLimit": "0x165A0BC00"
  },
  "accounts": {
    "0000000000000000000000000000000000000005": {
      "builtin": {
        "name": "modexp",
        "activate_at": "0x0",
        "pricing": {
          "modexp": {
            "divisor": 20
          }
        }
      }
    },
    "0000000000000000000000000000000000000006": {
      "builtin": {
        "name": "alt_bn128_add",
        "activate_at": "0x0",
        "pricing": {
          "linear": {
            "base": 500,
            "word": 0
          }
        }
      }
    },
    "0000000000000000000000000000000000000007": {
      "builtin": {
        "name": "alt_bn128_mul",
        "activate_at": "0x0",
        "pricing": {
          "linear": {
            "base": 40000,
            "word": 0
          }
        }
      }
    },
    "0000000000000000000000000000000000000008": {
      "builtin": {
        "name": "alt_bn128_pairing",
        "activate_at": "0x0",
        "pricing": {
          "alt_bn128_pairing": {
            "base": 100000,
            "pair": 80000
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000001": {
      "balance": "1",
      "builtin": {
        "name": "ecrecover",
        "pricing": {
          "linear": {
            "base": 3000,
            "word": 0
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000002": {
      "balance": "1",
      "builtin": {
        "name": "sha256",
        "pricing": {
          "linear": {
            "base": 60,
            "word": 12
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000003": {
      "balance": "1",
      "builtin": {
        "name": "ripemd160",
        "pricing": {
          "linear": {
            "base": 600,
            "word": 120
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000004": {
      "balance": "1",
      "builtin": {
        "name": "identity",
        "pricing": {
          "linear": {
            "base": 15,
            "word": 3
          }
        }
      }
    },
    "{ ACCOUNT_ADDRESS }": {
      "balance": "10000000000000000000000000000"
    }
  }
}

[jkuhlenkamp]

This leaves us with two options. The error results from ...

• (A) different validator addresses.
• (B) different user accounts.

[jkuhlenkamp]

We test A by removing the node account of the joinnode from the list of validators.

[...]
bootnode_1  |   "name": "blogpv",
bootnode_1  |   "engine": {
bootnode_1  |     "authorityRound": {
bootnode_1  |       "params": {
bootnode_1  |         "stepDuration": 5,
bootnode_1  |         "blockReward": "0x38D7EA4C68000",
bootnode_1  |         "maximumUncleCountTransition": 0,
bootnode_1  |         "maximumUncleCount": 0,
bootnode_1  |         "validators" : {
bootnode_1  |           "list": ["0xdc231a4bb89fd5049e8a435129accf32ed68214a"]
bootnode_1  |         }
bootnode_1  |       }
bootnode_1  |     }
bootnode_1  |   },
[...]

[...]
joinnode_1  |   "name": "blogpv",
joinnode_1  |   "engine": {
joinnode_1  |     "authorityRound": {
joinnode_1  |       "params": {
joinnode_1  |         "stepDuration": 5,
joinnode_1  |         "blockReward": "0x38D7EA4C68000",
joinnode_1  |         "maximumUncleCountTransition": 0,
joinnode_1  |         "maximumUncleCount": 0,
joinnode_1  |         "validators" : {
joinnode_1  |           "list": [""]
joinnode_1  |         }
joinnode_1  |       }
joinnode_1  |     }
[...]

DEBUG LOGS:

joinnode_1  | 2020-07-24 09:40:44 UTC IO Worker #0 DEBUG network  Connecting peers: 0 sessions, 1 pending + 0 started
joinnode_1  | 2020-07-24 09:40:44 UTC IO Worker #0 DEBUG discovery  Got unexpected Pong from V4(172.16.0.10:30303) ; request not found
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG engine  Zooming to epoch after block 0x145e…5aa2
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG engine  Zooming to epoch after block 0x145e…5aa2
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG miner  Attempting to push 0 transactions.
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG miner  Pushed 0 transactions in 0 ms
joinnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG network  Connecting peers: 0 sessions, 0 pending + 1 started
joinnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG network  Hello: OpenEthereum/node0/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1 v5 0xa97e…ae2f [SessionCapabilityInfo { protocol: [101, 116, 104], version: 64, packet_count: 17, id_offset: 16 }, SessionCapabilityInfo { protocol: [112, 97, 114], version: 4, packet_count: 32, id_offset: 33 }, SessionCapabilityInfo { protocol: [112, 105, 112], version: 1, packet_count: 9, id_offset: 65 }]
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #3 DEBUG network  Hello: OpenEthereum/node1/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1 v5 0x237d…b162 [SessionCapabilityInfo { protocol: [101, 116, 104], version: 64, packet_count: 17, id_offset: 16 }, SessionCapabilityInfo { protocol: [112, 97, 114], version: 4, packet_count: 32, id_offset: 33 }, SessionCapabilityInfo { protocol: [112, 105, 112], version: 1, packet_count: 9, id_offset: 65 }]
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #3 DEBUG sync  Deactivating peer 0
bootnode_1  | 2020-07-24 09:40:45 UTC IO Worker #3 DEBUG pip  Disabling peer 0: Wrong network
joinnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG sync  Deactivating peer 0
joinnode_1  | 2020-07-24 09:40:45 UTC IO Worker #2 DEBUG pip  Disabling peer 0: Wrong network

Fazit still breaks. => Two remaining options A1) Validator list MUST be exactly the same or B) Its due to the user account.

[jkuhlenkamp]

Removing user account from chain config.

SUCCESS!!!

[jkuhlenkamp]

POST MORTEM:

• The genesis block MUST be the same for all nodes in the network
• Accounts from the chain config file seem to become part of the genesis block.
• PoA validators specified as list NEED not to be the same.

[jkuhlenkamp]

Commit fix in e52c3b86af114ad376fa956a44daf8b86f88a009