Randomize IPv6 prefix to avoid reused addresses?

[natesales]

A beta tester brought up that when deleting and creating VMs quickly, it's possible for a new VM to have the same assigned prefix as an old (now deleted) VM. This means a SSH fingerprint warning because the VM no longer exists. Maybe we should randomize the prefix assignment process to minimize this?

[lanefu]

what are you using for MAC address generation? IPV6 IP should be influenced by that

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:a1:7d:b0 brd ff:ff:ff:ff:ff:ff altname enp1s0 inet6 2a0e:8f00:fe7d:ffff::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fea1:7db0/64 scope link valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:7e:8e:2a brd ff:ff:ff:ff:ff:ff altname enp1s0 inet6 2a0e:8f00:fe7e:fffe::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe7e:8e2a/64 scope link valid_lft forever preferred_lft forever

[haylinmoore]

That's how actually not how the public IPv6 distribution works at all. If you look at these lines https://github.com/natesales/aarch64/blob/main/api.py#L394-L399, when assigning a VM with an IP they simply loop through the list of prefixes assigned to a host and assign the first one that is not used. The ::1 of the prefix is then used for the gateway as shown on lin 398, and the ::2 is used on the VM as shown on line 399.

Now the link local (fe80::/10) here is be based on the mac address and it is standard to use the EUI-64 format for that, but link local addresses are not publically routable, so nobody will be SSHing into a machine on that.

To properly randomize the v6 block a VM is placed in one would just have to import random into the API and then change the for loop to use random.shuffle(list(ipaddress.ip_network(host_prefix).subnets(new_prefix=64)))

[lanefu]

Gotcha.. so do all VMs running on a given VM host share the same /64 space?

[haylinmoore]

No, they don't share a /64, they get their own /64 inside of /48 dedicated to each host

[lanefu]

No, they don't share a /64, they get their own /64 inside of /48 dedicated to each host

Great. thanks for clarifying. Sorry I think i was up to late to ask questions last night, and up too early to ask this morning :)

Something to consider in the future as development progresses might be to delegate address management to IPAM.

Netbox has a good API and great ansible collection for interacting with it.

Initial rollout of netbox can be a little tedious at first, but its not bad..

[haylinmoore]

Honestly I don't see the use for an IPAM system for this. None of the vm hosts will ever use up their 65k /64s. And since each vm host has its own unique /48 with the /64s inside one can quickly check what host an IP is from by checking the admin list of vm hosts and their /48

[natesales]

We do use Netbox at Fosshost for IPAM, but as @hamptonmoore pointed out I don't see a reason to an external IPAM tool for this. It would be useful to write some tooling to replicate VM configuration from aarch64 to Netbox purely for an ops perspective in being able to see all resources from a single UI. That said, it would go against the "single source of truth" paradigm that we're hoping to follow with Netbox.

[lanefu]

Oh the solution at hand is certainly sufficient. Main benefit of course with NetBox is the single pane of glass view, audit trail etc.

Although extremely unlikely, it might help with a race condition of IP state if the API gets slammed. In theory delegating the responsibility of transaction safety to IPAM might help with unforeseen overhead as your platform grows and evolves in the future.

In this particular scenario, applying a filter of last_updated: gt 1w during a get_address API call to netbox would assure that customer gets an IP that wasn't used recently.

[natesales]

That's a good point. For now I'm happy with the current solution, but let's leave this issue open and we can continue discussing this as a future addition.