displague
commented
3 years ago From an Equinix Metal product offering perspective, this is possible in one of two ways.
Every Equinix Metal device has a management address assigned to the first bond, by default. This per device /30 (by default) assignment is part of a 10.x.x.x/25 address range isolated to device peers within the same EM project and facility (a /56 is also created as an IPv6 management range). This https://github.com/equinix/terraform-metal-anthos-on-baremetal project used this solution, keeping the node communication on the 10.x.x.x network.
Alternatively, a device may be provisioned with VLANs. Until recently, all EM Devices used to have to toggle Layer2 support on a port or bond to enroll in this behavior. This is represented by the layer2-bonded, layer2-individual, hybrid network modes available in the Terraform provider with similar names in the EM console UI.
It is now possible to add VLANs to a device without pre-enrolling the device in Layer2 modes, but this new functionality is limited to devices in a subset of facilities (those with the IBX feature flag). We'll have to actively enable layer2 features or detect when this step can be skipped.
It is worth noting that the network modes reported in the UI and represented in the Terraform provider are not settings that can be toggled in the EM API. Network mode is a product of port bondedness, layer-2 or layer-3 state, the presence of management addresses, and the presence of VLANs.
After a few different iterations on how to represent these features in Terraform, we are coming around to the idea of giving the user control over the ports directly without offering network mode hand-waving.
Expressed statefully, and adhering closely to the EM API spec, this could look like this:
- plan: "n2.x-large"
ports: # network_ports in the API naming
bond0: # always comprised of ports eth0+eth1 on 2 port devices, or eth0+eth2 on 4 port devices
bonded: true # default
layer2: false # this is default on new devices. changes result in /ports/id/convert/layer-[2|3] API calls
vlans: [1234, 5678, 1234] # shortened this from the API "virtual_networks"
native_vlan: 5678 # optional, must be one of the above if set
ip_addresses:
- reservation_id: uuid # reserved addresses by uuid, may include global ips
- cidr_notation: "1.2.3.4/24" # reserved addresses by address, may include global ips
- type: private_ipv4 # dynamically assigned addresses, available via metadata.platformequinix.net
cidr: 30
bond1: # comprised of port eth1+eth3 on 4 port devices
bonded: false
eth1: # unbonded eth ports can use most of the same attributes bond ports can use
layer2: true
vlans: [7654]
eth3:
layer2: true
vlans: [9876]Users are free to customize the ports, bonds, addresses, and VLANs with all the flexibility afforded by the API. Invalid configurations will bubble up through API errors and input does not have to be validated by the CAPP provider. It is difficult to validate these values because the plan, facility, state of the port, state of the bonds, and other factors affect the success of toggling a field. Through Kubernetes reconciliation loops, eventually, all bonds will be connected, disconnected, or the device will reach the desired layer-2/layer-3 mode. The desired addresses will be provisioned and assigned.
A port and bond configuration not supported today, maybe supported later or may be supported in a different plan or facility setting.
This was discussed for consideration in the Terraform provider within a packngo PR: https://github.com/packethost/packngo/pull/239#discussion_r572120683
I don't know how well CAPP can capture this approach. CAPP would need to know which address ranges to use for what purposes, dynamically allocated ranges like the project management range could be referred to generically without knowing the precise range. In other cases, the precise range may be known in advance (IP reservations). In VLAN cases, the nodes would need special userdata to configure the network and CAPP would need to be told what ranges to manage.
Userdata scripts would need to be hearty, allowing time for the ports to be bonded, VLANs to be attached, and addresses to become available.
fejta-bot
detiber
davidspek
Lirt
As an operator I'd like to create clusters with nodes that are completely isolated from the public internet. Instead, they should only be accessible through authorized IPs or bastion nodes.
Detailed Description
In CAPP's current implementation, all control plane and worker nodes have public IP addresses, kubelets are configured to speak the public IP address for the control plane, and anyone on the internet can attempt connections to control plane nodes. While TLS+authentication add a layer of security, it would be nice to have the ability to create cluster nodes without public IP addresses, analogous to GKE private clusters. All egress traffic would be directed to NAT gateways and inbound traffic will only be allowed from authorized IPs..
/kind feature