Configure IPv6 addresses for Nimbus nodes

[jakubgs]

Based on request from @arnetheduck we need to add IPv6 addresses to Nimbus hosts.

I have already opened a ticket with InnovaHosting: https://client.innovahosting.net/viewticket.php?tid=769921&c=UeLgmsq7

[jakubgs]

We got an IPv6 range from InnovaHosting to use, I requested 512 addresses for now:

Network	2a0a:d580:40:60::/64
Gateway	2a0a:d580:40:60::1
Start	2a0a:d580:40:60::100
End	2a0a:d580:40:60::2ff

One issue with this tho is that Innova doesn't provide IPs via DHCP but with static configuration:

jakubgs@geth-01.ih-eu-mda1.nimbus.holesky:~ % sudo cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      addresses: [ 194.33.40.71/24 ]
      gateway4: 194.33.40.1
      nameservers:
        addresses: [ 194.33.40.10, 1.1.1.1 ]
  version: 2

[jakubgs]

Wrote a small utility to generate and validate IPv6 ranges:

• infra-utils#084cfbba - network/ipv6gen.py: tool for generating ipv6 ranges

Mostly as a way to familiarize myself with it.

 > ./network/ipv6gen.py --network F00D:CAFE::/64 --start F00D:CAFE::15 --end F00D:CAFE::1f --count 20 
Network: f00d:cafe::/64
Start: f00d:cafe::15
End: f00d:cafe::1f
Count: 20

f00d:cafe::15
f00d:cafe::16
f00d:cafe::17
f00d:cafe::18
f00d:cafe::19
f00d:cafe::1a
f00d:cafe::1b
f00d:cafe::1c
f00d:cafe::1d
f00d:cafe::1e
Address beyond range: f00d:cafe::1f

[jakubgs]

I've enable IPv6 for our AWS VPC and subnet:

• infra-tf-aws-vpc#1057a690 - enable IPv6 addresses on VPC and subnet
• infra-tf-aws-vpc#f29a70c7 - upgrade provider from 4.67.0 to 5.40.0
• infra-tf-aws-vpc#229bbe68 - allow public access to IPv6 addresses

But it appears that's not enough.

[jakubgs]

Some relevant links:

• https://docs.aws.amazon.com/whitepapers/latest/ipv6-on-aws/amazon-vpc-internet-access.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#assign-ipv6-address
• https://medium.com/@mattias.holmlund/setting-up-ipv6-on-amazon-with-terraform-e14b3bfef577
• https://colinbarker.me.uk/blog/2023-02-11-enabling-ipv6-on-aws-using-terraform/

[jakubgs]

This is kinda dumb, Terraform doesn't know how to assign the IP without re-creating the instance:

But it can be easily done via the Web UI:

[jakubgs]

Which indeed works:

jakubgs@bootstrap-01.aws-eu-central-1a.nimbus.mainnet:~ % ip addr show ens5
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:2e:16:a8:92:1c brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.67/24 metric 100 brd 172.20.1.255 scope global dynamic ens5
       valid_lft 3320sec preferred_lft 3320sec
    inet6 2a05:d014:113c:2400:6e6f:d115:b30c:9a12/128 scope global dynamic noprefixroute 
       valid_lft 445sec preferred_lft 135sec
    inet6 fe80::2e:16ff:fea8:921c/64 scope link 
       valid_lft forever preferred_lft forever

[jakubgs]

Relevant changes in AWS instance role:

• infra-tf-amazon-web-services#4ed017e4 - add IPv6 address to all EC2 instances
• infra-tf-amazon-web-services#03fde5eb - upgrade provider from 4.67.0 to 5.40.0
• infra-tf-amazon-web-services#d012d2e7 - add public_ips_v6 to outputs
• infra-tf-amazon-web-services#f53902ea - add CloudFlare AAAA record for IPv6 address

And now we can reach it via nmap(ICMP is not enabled in security groups):

jakubgs@geth-01.ih-eu-mda1.nimbus.holesky:~ % nmap -6 -Pn -p22,9100 bootstrap-01.aws-eu-central-1a.nimbus.mainnet.statusim.net
Starting Nmap 7.80 ( https://nmap.org ) at 2024-03-14 15:16 UTC
Nmap scan report for bootstrap-01.aws-eu-central-1a.nimbus.mainnet.statusim.net (2a05:d014:113c:2400:6e6f:d115:b30c:9a12)
Host is up (0.034s latency).
Other addresses for bootstrap-01.aws-eu-central-1a.nimbus.mainnet.statusim.net (not scanned): 3.120.104.18

PORT     STATE  SERVICE
22/tcp   open   ssh
9100/tcp closed jetdirect

jakubgs@geth-01.ih-eu-mda1.nimbus.holesky:~ % nmap -6 -Pn -p22,9100 bootstrap-02.aws-eu-central-1a.nimbus.mainnet.statusim.net
Starting Nmap 7.80 ( https://nmap.org ) at 2024-03-14 15:35 UTC
Nmap scan report for bootstrap-02.aws-eu-central-1a.nimbus.mainnet.statusim.net (2a05:d014:113c:2400:bce8:1cea:4e02:a8fd)
Host is up (0.034s latency).
Other addresses for bootstrap-02.aws-eu-central-1a.nimbus.mainnet.statusim.net (not scanned): 3.64.117.223

PORT     STATE  SERVICE
22/tcp   open   ssh
9100/tcp closed jetdirect

I actually don't know why the LibP2P port 9100 appears as closed but at least it's not filtered.

[jakubgs]

For InnovaHosting I'm not sure how I want to do it. Managing static IPs with bootstrap Ansible role just for ~50 hosts from Innova seems kinda overkill. But at the same time, we do need a place to store those IPv6 addresses and their mappings to hosts.

[jakubgs]

Added IPv6 to Consul advertised addresses at bootstrapping time:

• infra-role-bootstrap-linux#aaad7e7b - consul: add IPv4 and IPv6 to advertised addresses

We can also in the future use the addresses in Beacon node ansible role.

[jakubgs]

Adding it manually on InnovaHosts definitely works:

jakubgs@geth-01.ih-eu-mda1.nimbus.holesky:~ % sudo cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      addresses:
        - 194.33.40.71/24
        - 2a0a:d580:40:60::100/64
      gateway4: 194.33.40.1
      gateway6: 2a0a:d580:40:60::1
      nameservers:
        addresses: [ 194.33.40.10, 1.1.1.1 ]
  version: 2

It is reachable:

jakubgs@bootstrap-01.aws-eu-central-1a.nimbus.mainnet:~ % ping -c4 2a0a:d580:40:60::100
PING 2a0a:d580:40:60::100(2a0a:d580:40:60::100) 56 data bytes
64 bytes from 2a0a:d580:40:60::100: icmp_seq=1 ttl=55 time=34.6 ms
64 bytes from 2a0a:d580:40:60::100: icmp_seq=2 ttl=55 time=34.6 ms
64 bytes from 2a0a:d580:40:60::100: icmp_seq=3 ttl=55 time=34.6 ms
64 bytes from 2a0a:d580:40:60::100: icmp_seq=4 ttl=55 time=34.9 ms

--- 2a0a:d580:40:60::100 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 34.567/34.677/34.922/0.144 ms

[jakubgs]

Took a BUNCH of work to enable IPv6 addresses on eth2.prod fleet since the fallback hosts were using the default VPC and subnet which do not have support for IPv6:

• infra-eth2#7b00dd3c - fallback.tf: use dedicated VPC and subnet

Since instances cannot be moved between subnets I had to re-created fallback hosts.

[jakubgs]

I decided to do a custom solution in infra-nimbus since this is just for InnovaHosting hosts.

I've copied existing network configuration files from the hosts using:

grep 'nimbus.holesky$' ansible/inventory/test | sort -u \
  | xargs -I{} scp {}:/etc/netplan/00-installer-config.yaml ansible/vars/addresses/{}.yml

And then added the IPv6 addresses using:

#!/usr/bin/env bash
GATEWAY='2a0a:d580:40:60::1'
ADDR_PREFIX='2a0a:d580:40:60:'
COUNTER=256

function update_yaml() {
    [[ "${FILE}" == "update.sh" ]] && return
    ADDR="${ADDR_PREFIX}:$(printf '%x\n' "${COUNTER}")"
    sed -i "s/addresses: \[ \([0-9./]\+\) \]$/addresses:\n        - \1/" "${1}"
    sed -i "/gateway4:/i \        - ${ADDR}/64" "${1}"
    sed -i "/gateway4:/a \      gateway6: ${GATEWAY}" "${1}"
    COUNTER=$((COUNTER+1))
}

for FILE in geth-*;   do update_yaml "${FILE}"; done
for FILE in erigon-*; do update_yaml "${FILE}"; done
for FILE in neth-*;   do update_yaml "${FILE}"; done

Result:

 > cat erigon-10.ih-eu-mda1.nimbus.holesky.yml 
# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      addresses:
        - 194.33.40.246/24
        - 2a0a:d580:40:60::113/64
      gateway4: 194.33.40.1
      gateway6: 2a0a:d580:40:60::1
      nameservers:
        addresses: [ 194.33.40.10, 1.1.1.1 ]
  version: 2

[jakubgs]

Here's the changes for three main networks?

• infra-nimbus#581336ef - nimbus.holesky: add IPv6 addresses and netplan role
• infra-nimbus#90edd3c5 - nimbus.mainnet: add IPv6 addresses and netplan role
• infra-nimbus#9ef7a7a8 - nimbus.sepolia: add IPv6 addresses and netplan role

There's no DNS entries for now. If they are necessary they can be added.