P2P Seeds Set up

[MaxMustermann2]

In cmd/exocored/init.go, there are hardcoded seeds added when initializing an empty node. We need to remove these values and replace with our own nodes.

seeds := []string{
    "40f4fac63da8b1ce8f850b0fa0f79b2699d2ce72@seed.evmos.jerrychong.com:26656",                 // jerrychong
    "e3e11fca4ecf4035a751f3fea90e3a821e274487@bd-evmos-mainnet-seed-node-01.bdnodes.net:26656", // blockdaemon
    "fc86e7e75c5d2e4699535e1b1bec98ae55b16826@bd-evmos-mainnet-seed-node-02.bdnodes.net:26656", // blockdaemon
}

Steps

• Set up a node (or 3) to act as seed node
• Point a domain's A and AAAA record to the seed node
• Back up the node's P2P key for future use since that id is part of the seed as well as the persistent peers (the part before the @)

See also https://github.com/cosmos/chain-registry/blob/master/evmos/chain.json

[MaxMustermann2]

We need to

• Set pex = true in the seed node as well as other nodes to allow for peer exchange. Currently, on our local-only nodes, this is explicitly set to false.
• Set seed_mode = true in the config.toml file of the node designated as the seed node. By default, this is false.
• Retain the config/node_key.json for future use. Such a key is regenerated, by default, at every call to exocored init with no in-built functionality to recover it from a mnemonic (p2p.LoadOrGenNodeKey(config.NodeKeyFile())).

The loss of a node key is not catastrophic; other peers can be informed about a new ID if it is lost, with no problems (beyond the time it would take for this information to disseminate). To guard against this, running at least 2 seed nodes (with 1 for backup) is a good idea.

The p2p ID can be seen by running exocored tendermint show-node-id. The connection port is the one set in p2p.laddr, with the resulting seed address as id@ip:port required to be set in the p2p.seeds line of config.toml. Alternatively, a different value may be supplied in p2p.external_address provided the requisite proxy is in place. In this case the address because id@external_address.

For our use case, I picked the last node for my testing. It is the node bound to 127.0.0.x, and not 0.0.0.0. With a view of not creating conflicts with other IP:port bindings, I did not change that. Instead, I used a proxy to handle requests received on public_ip:26656 and forward them to 127.0.0.x:port.

The p2p connection operates over TCP and not regular HTTP, and hence, Caddy (our currently deployed reverse proxy), cannot handle it fully. However, with the l4 plugin, it can be done. I have used this plugin for personal projects in the past.

I made the change to support this on the server with one of the nodes set to seed_mode and the public_ip:26656 value forwarded to the p2p.laddr of this node using Caddy (although that has made the config difficult to read since the l4 plugin does not support Caddyfile; only yaml or json). In addition, I set the p2p.external_address equal to public_ip:26656 in the config.toml.

I then tested the configuration by creating two nodes: one at home, and one on my personal VPS (with no Exocore stuff on it until now). Both were connected to the same seed node and started with the same genesis file. They were not connected directly with their configurations. At first, the nodes connected to the seed node, which reported incoming connections from each of the two nodes (separately) in its logs. Then, my laptop sent out a pex request to the seed node for more peers, to which it responded with my VPS as a peer. The connection was initiated after this response and all the 3 nodes were speaking to each other.

[MaxMustermann2]

I have done some more research into the topic of setting up a seed node without any persistent peers.

A seed node is designed, by definition, to crawl the network and find out peers. It does so by asking other peers for more peers, persistently. When other peers ask the seed node for new peers, it is able to respond from this potentially massive list obtained via crawling.

Whereas, non-seed nodes do no such thing. They simply aim to stay connected to their target count of peers and only look for more peers if the active connections are below this number. Other nodes may still ask non-seed node for new peers (using the same mechanism that the seed node uses); however, the list of peers available to such a node will be smaller. This is because it isn't actively crawling the network for more peers.

Based on only the above information, it made sense to me that a node could act as a standalone seed node and still sync with a single validator. I tested this out, and it works as expected, although it takes some time. This is because the seed node is designed to avoid DoS attacks, which it does by responding with a list of peers, saving the incoming peer in-memory, and then terminating the connection immediately. In our example case of a single validator and a single seed node, this results in a seed node with no active connections + 1 saved peer and a validator that is producing blocks by itself.

After some time (crawlPeerPeriod = 30 * time.Second), however, the goroutine designed to find connections is triggered on the seed node. It sends a request to the validator node, which establishes the connections and allows syncing.