Set cluster domain inside a tenant cluster

We want to generate a tenant cluster with clusterAPI and Kubevirt that can be used there own cluster domain.

We set the serviceDomain at the cluster definition:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: "${CLUSTER_NAME}"
  namespace: "${NAMESPACE}"
  labels:
    cluster.x-k8s.io/cluster-name: "${CLUSTER_NAME}"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.243.0.0/16
    serviceDomain: ${CLUSTER_NAME}.local  
    services:
      cidrBlocks:
      - 10.96.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: KamajiControlPlane
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: KubevirtCluster
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}

Setup the ControlPlane with this config:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  dataStoreName: ${KAMAJI_DATASTORE}
  addons:
    coreDNS:
      dnsServiceIPs:
      - 10.96.0.10
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
    - InternalIP
    - ExternalIP
  network:
    serviceType: LoadBalancer
    serviceAnnotations:
      lbipam.cilium.io/ips: "192.168.108.8"
  deployment: {}
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: "${KUBERNETES_VERSION}"

Kamaji Version that we used:

helm search repo kamaji
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
clastix/kamaji          1.0.0           v1.0.0          Kamaji is the Hosted Control Plane Manager for ...
clastix/kamaji-console  0.0.5           v0.0.2          Kamaji deploys and operates Kubernetes at scale...
clastix/kamaji-etcd     0.7.0           3.5.6           Helm chart for deploying a multi-tenant `etcd` ...
clastix/console         0.0.1           0.0.1           Kamaji is a tool aimed to build and operate Kub...

Problem is that the resulting cluster use the default "cluster.local" domain

kubectl proxy &
k get nodes
curl -X GET http://127.0.0.1:8001/api/v1/nodes/tenant1-md-0-j8r46-qp5kc/proxy/configz | jq .

    "clusterDomain": "cluster.local",
    "clusterDNS": [
      "10.96.0.10"
    ],

Also included at the coreDNS core file:

kubectl get cm coredns -n kube-system -o jsonpath="{.data.Corefile}" \
  | grep ".local " \
  | awk -F ' ' '{print $2}'
cluster.local
k get cm -n kube-system coredns -o yaml
## output
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2024-09-13T08:31:44Z"
  name: coredns
  namespace: kube-system
  ownerReferences:
  - apiVersion: rbac.authorization.k8s.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: ClusterRoleBinding
    name: system:coredns
    uid: 3341239f-17aa-4a6c-8c53-3daf4fcc4e94
  resourceVersion: "3615354"
  uid: a519dba3-fd61-4b99-8ad8-aebc5f35eb8a

Is my configuration wrong?

clastix / kamaji

Set cluster domain inside a tenant cluster #579