Configure cluster control-plane and worker nodes with custom nameservers on vSphere

[christianang]

Describe the feature request

As a cluster operator, I would like to configure different DNS servers per cluster on vSphere. I would also like to be able to configure the control-plane node's DNS servers independently from the DNS servers on the worker nodes. I would like to configure this via a data value in my cluster's configuration file.

This is currently only scoped to vSphere because each cluster-api provider will have a different way to configure nameservers on their nodes. We can add support for other IaaSes after vSphere. Additionally, I haven't investigated if all providers allow you to configure nameservers through their cluster-api provider API so there may be additional work that needs to be done on providers to make this work for all IaaSes.

Design/Acceptance

Given I set WORKER_NODE_NAMESERVERS to the IP of my nameserver(s) in cluster configuration yaml And I create the cluster on vSphere using tanzu cluster create my-cluster -f ./my-configuration.yaml When I ssh onto the worker node and I resolvectl status Then I see my nameserver(s) in the list of dns servers.

Given I set CONTROL_PLANE_NODE_NAMESERVERS to the IP of my nameserver(s) in cluster configuration yaml And I create the cluster on vSphere using tanzu cluster create my-cluster -f ./my-configuration.yaml When I ssh onto the control-plane node and I resolvectl status Then I see my nameserver(s) in the list of dns servers.

Given I see either WORKER_NODE_NAMESERVERS or CONTROL_PLANE_NODE_NAMESERVERS set, but I am not using vSphere When I run tanzu cluster create my-cluster -f ./my-configuration.yaml Then I see a ytt template validation error

(I should also be able to do all the above using tanzu management-cluster create as well)

Describe alternatives you've considered

Affected product area (please put an X in all that apply)

• [ ] APIs
• [ ] Addons
• [X] CLI
• [X] Docs
• [ ] Installation
• [X] Plugin
• [ ] Security
• [ ] Test and Release
• [X] User Experience

Additional context

For reference, when we start work on this, we can accomplish this by using something similar to the following overlay:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#! This is for control plane nodes. `-control-plane` can be replaced by `-worker` for worker nodes.
#@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-control-plane"}})
---
spec:
  template:
    spec:
      network:
        devices:
        #@overlay/match by=overlay.all, expects="1+"
        -
          #@overlay/match missing_ok=True
          nameservers: ["1.2.3.4"]

[mcwumbly]

Wondering whether we should also validate that the items in nameservers are are all valid IP addresses, and that they match the TKG_IP_FAMILY.

[rosskukulinski]

Hi @christianang what's the use-case for different settings on worker nodes vs control plane nodes?

[rosskukulinski]

And what's the impetuous/urgency for this request?

[christianang]

Hi @christianang what's the use-case for different settings on worker nodes vs control plane nodes?

The primary use-case, that I know of, for different settings on the control plane node vs worker node is to allow the control plane to discover and communicate with the vCenter, but not allow processes on the worker node to discover the vCenter address.

And what's the impetuous/urgency for this request?

Not particularly urgent, but our team has spare cycles and we know this is a feature that users have asked for.

[rosskukulinski]

Ack. I'm +1 on this, though authoritative PM will be Disha Chopra. She's not a member of this repo yet, so reassigning to @christianang as implementer.

[christianang]

Wondering whether we should also validate that the items in nameservers are are all valid IP addresses, and that they match the TKG_IP_FAMILY.

@mcwumbly after thinking about this a bit, I'm tempted to not do the validation of the IPs against the TKG_IP_FAMILY. Currently we would have to put this kind of validation in the CLI itself. I'm not entirely confident there are any clear use-cases for using an IP that doesn't match the TKG_IP_FAMILY, but I'm also not sure if I want to entirely disallow it, if a user wanted to do this for whatever reason. Perhaps we can revisit if we are sure this isn't something we would want a user to do or perhaps if we can do this validation in ytt (which will be possible if this https://github.com/vmware-tanzu/carvel-ytt/pull/433 PR to ytt is merged).

[mcwumbly]

we can revisit if we are sure this isn't something we would want a user to do or perhaps if we can do this validation in ytt

sounds good. we can revisit later.