This repository contains the infrastructure code used to setup ~all~ dev/testnets. A lot of the code uses reusable components either provided by our ansible collection or our helm charts for kubernetes.
Status | Network | Links | Ansible | Terraform | Kubernetes | |
---|---|---|---|---|---|---|
š¢ Active | verkle-gen-devnet-6 | verkle-gen-devnet-6 | Network config / Inventory / Validator ranges | š | š | š |
š“ Off | verkle-gen-devnet-5 | verkle-gen-devnet-5 | Network config / Inventory / Validator ranges | š | š | š |
š“ Off | verkle-gen-devnet-4 | verkle-gen-devnet-4 | Network config / Inventory / Validator ranges | š | š | š |
š“ Off | verkle-gen-devnet-3 | verkle-gen-devnet-3 | Network config / Inventory / Validator ranges | š | š | š |
š“ Off | verkle-gen-devnet-2 | verkle-gen-devnet-2 | Network config / Inventory / Validator ranges | š | š | š |
We're using asdf to make sure that we all use the same versions across tools. Our repositories should contain versions defined in .tools-versions.
You can then use ./setup.sh
to install all dependencies.
Make sure you select either hetzner or digitalocean (default is digitialocean), if you want to use hetzner rename digitalocean.tf to digitalocean.tf.disabled and rename hetzner.tf.disabled to hetzner.tf and vice versa.
terraform init
terraform apply
To install the nodes according to the inventory file that is generated by the terraform template run the following commands from ./ansible/
./install_dependencies.sh
ansible-playbook -i inventories/devnet-0/inventory.ini playbook.yaml
In order to clean up the deployment
ansible-playbook -i inventories/devnet-0/inventory.ini cleanup_ethereum.yaml
To create a new testnet using the infrastructure code, follow these steps:
Open the main.tf file located in the terraform/devnet-0/ directory.
Locate the sections that define the different nodes and their corresponding validator ranges, for this example this is variable "digitalocean_vm_groups"
Adjust the validator indexes in the terraform/main.tf file based on your desired allocation of validators to nodes.
For example, let's say you want to assign validator index 0-24 to the lodestar-besu-1 node and validator index 25-224 to the lighthouse-nethermind-1 node. Update the main.tf file as follows:
{
id = "lodestar-besu"
vms = {
"1" = { ansible_vars : "validator_start=0 validator_end=25" }
},
},
{
id = "lighthouse-nethermind"
vms = {
"1" = { ansible_vars : "validator_start=25 validator_end=225" }
},
},
Make sure to adjust the validator ranges according to your requirements and the number of validators in your network. This configuration ensures that validators within the specified ranges will be allocated to the corresponding nodes during the deployment. By customizing the validator indexes in the Terraform configuration, you can allocate validators to specific nodes in your network according to your desired configuration.
terraform apply
will create a the machines and the inventory file for you. The inventory file will be located in the ansible/inventories/devnet-0 directory.
The inventory.ini file will have the list of all the nodes that were created by Terraform. The inventory file will also have the validator ranges that were specified in the Terraform configuration. The validator ranges will be used by the Ansible playbook to allocate validators to the corresponding nodes.
[lodestar_besu]
lodestar-besu-1 ansible_host=167.99.34.241 cloud=digitalocean cloud_region=ams3 validator_start=0 validator_end=25
...
Adjust the total number of validators in the ansible/inventories/devnet-0/group_vars/all.yaml file (ethereum_genesis_generator_config_files.values.env.NUMBER_OF_VALIDATORS) to match with your total number of validators that you are running.. This will be used by the Ansible playbook to generate the validator keys and deposit data for the network.
ansible-playbook -i inventories/devnet-0/inventory.ini playbook.yaml
from the ansible/ directory to deploy the network. This will generate the genesis file, validators and deploy the network according to the configuration parameters specified in the ansible/inventories/devnet-0/group_vars/all.yaml file.
Don't forget the following gotchas:
ethereum_genesis_chain_id
value in ansible/inventories/devnet-0/group_vars/all.yaml to avoid clashing with an existing networkdocker
running on your local machine, this is essential for generating some post-testnet filesbootstrap_default_user_authorized_keys_github_...
, otherwise ansible will fail on the bootstrap step
ansible-playbook -i inventories/devnet-0/inventory.ini cleanup_ethereum.yaml
from the ansible/ directory to clean up the network. This will delete the genesis file, validators and clean up the network on all the nodes.
Run
ansible-playbook -i inventories/devnet-0/inventory.ini ansible-playbook playbook.yaml -t ethereum_genesis -e ethereum_genesis_cleanup=true
from the ansible/ directory to clean up the network-configs and validators directories on your local machine. This step is required if you would like to reuse the nodes but with a different genesis configuration. (For example, if you would like to change the validator indexes assigned to the nodes, due to a relaunch).
Run terraform destroy
from the terraform/devnet-0/ directory to delete the nodes. This will remove all the virtual machines and the inventory file. Be careful when running this command, as it will delete all the nodes and the inventory file. You will need to run terraform apply
again to create the nodes and the inventory file.
helm dependency update
from the tool directory. This will update the dependencies for the helm chart. Commit the changes to the repository and ArgoCD will automatically deploy the updated tooling to the Kubernetes cluster.terraform output
from the terraform/devnet-0/ directory.curl -s https://bootnode-1.devnet-0.ethpandaops.io/meta/api/v1/validator-ranges.json
ethdo --connection=https://user:password@bn.lighthouse-nethermind-1.devnet-0.ethpandaops.io block info --blockid 100 --json | jq -r .message.proposer_index | ./whose_validator.zsh
from the ansible/ directory.
curl -s https://config.devnet-0.ethpandaops.io/api/v1/nodes/inventory | jq -r '.ethereum_pairs[] | .execution.enode'
curl -s https://config.devnet-0.ethpandaops.io/api/v1/nodes/inventory | jq -r '.ethereum_pairs[] | .consensus.enr'
# Find all .sops.* and *.enc.* files and update their keys
find . -type d -name "vendor" -prune -o \( -type f \( -name "*.sops.*" -o -name "*.enc.*" \) \) -exec sops updatekeys {} -y \;
Here's a table of where the keys are used
Account Index | Component Used In | Private Key Used | Public Key Used | Comment |
---|---|---|---|---|
0 | tx_fuzz_txs | ā | Spams tx on the network | |
1 | faucet1&2 | ā | manually funded these two due to k8s secrets issue | |
2 | tx_fuzz_txs | ā | ||
3 | tx_fuzz_txs ā | |||
4 | ||||
5 |