search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.02k stars 1.51k forks source link

WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration #3590

Closed aborrero closed 2 months ago

aborrero commented 2 months ago

This is to report something weird, but to the best of my knowledge did not affect how kind works (so, maybe not a bug?)

What happened:

When running kind create in verbose mode, I discovered a concerning warning:

user@debian:~$ kind create -v 100 cluster --name "toolforge" --config "kind.yaml" --image "kindest/node:v1.24.17"@"sha256:bad10f9b98d54586cba05a7eaa1b61c6b90bfc4ee174fdc43a7b75ca75c95e51"
Creating cluster "toolforge" ...
DEBUG: docker/images.go:58] Image: kindest/node:v1.24.17@sha256:bad10f9b98d54586cba05a7eaa1b61c6b90bfc4ee174fdc43a7b75ca75c95e51 present locally
 ✓ Ensuring node image (kindest/node:v1.24.17) 🖼
 ✓ Preparing nodes 📦  
DEBUG: config/config.go:96] Using the following kubeadm config for node toolforge-control-plane:
apiServer:
  certSANs:
  - localhost
  - 127.0.0.1
  extraArgs:
    enable-admission-plugins: NodeRestriction,PodSecurityPolicy
    feature-gates: TTLAfterFinished=true
    runtime-config: ""
apiVersion: kubeadm.k8s.io/v1beta3
clusterName: toolforge
controlPlaneEndpoint: toolforge-control-plane:6443
controllerManager:
  extraArgs:
    enable-hostpath-provisioner: "true"
    feature-gates: TTLAfterFinished=true
kind: ClusterConfiguration
kubernetesVersion: v1.24.17
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16
scheduler:
  extraArgs:
    feature-gates: TTLAfterFinished=true
---
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- token: abcdef.0123456789abcdef
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.18.0.2
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    node-ip: 172.18.0.2
    node-labels: ingress-ready=true,kubernetes.wmcloud.org/nfs-mounted=true
    provider-id: kind://docker/toolforge/toolforge-control-plane
---
apiVersion: kubeadm.k8s.io/v1beta3
controlPlane:
  localAPIEndpoint:
    advertiseAddress: 172.18.0.2
    bindPort: 6443
discovery:
  bootstrapToken:
    apiServerEndpoint: toolforge-control-plane:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
kind: JoinConfiguration
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    node-ip: 172.18.0.2
    node-labels: ""
    provider-id: kind://docker/toolforge/toolforge-control-plane
---
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
cgroupRoot: /kubelet
evictionHard:
  imagefs.available: 0%
  nodefs.available: 0%
  nodefs.inodesFree: 0%
failSwapOn: false
featureGates:
  TTLAfterFinished: true
imageGCHighThresholdPercent: 100
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
conntrack:
  maxPerCore: 0
featureGates:
  TTLAfterFinished: true
iptables:
  minSyncPeriod: 1s
kind: KubeProxyConfiguration
mode: iptables
 ✓ Writing configuration 📜 
DEBUG: kubeadminit/init.go:82] I0425 09:30:10.067722     172 initconfiguration.go:255] loading configuration from "/kind/kubeadm.conf"
W0425 09:30:10.068785     172 initconfiguration.go:332] [config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
[...]
<< proceeds normally with the bootstrap >>
[...]

For the record, the kind config file I'm using looks like this:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  TTLAfterFinished: true
kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        enable-admission-plugins: NodeRestriction,PodSecurityPolicy
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.5.15"]
    endpoint = ["http://192.168.5.15"]
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true,kubernetes.wmcloud.org/nfs-mounted=true"
    extraPortMappings:
      - containerPort: 6443
        hostPort: 6443
        protocol: TCP
      - containerPort: 30001
        hostPort: 30001
        protocol: TCP
      - containerPort: 30002
        hostPort: 30002
        protocol: TCP
      - containerPort: 30003
        hostPort: 30003
        protocol: TCP
    extraMounts:
      - hostPath: /var/lib/sss/pipes/
        containerPath: /var/lib/sss/pipes/
      - hostPath: /data/project/
        containerPath: /data/project/
      - hostPath: /data/scratch/
        containerPath: /data/scratch/
      - hostPath: /mnt/nfs/dumps-clouddumps1001.wikimedia.org/
        containerPath: /mnt/nfs/dumps-clouddumps1001.wikimedia.org/
      - hostPath: /mnt/nfs/dumps-clouddumps1002.wikimedia.org/
        containerPath: /mnt/nfs/dumps-clouddumps1002.wikimedia.org/
      - hostPath: /public/dumps/
        containerPath: /public/dumps/
      - hostPath: /etc/wmcs-project
        containerPath: /etc/wmcs-project
        propagation: Bidirectional
      - hostPath: /etc/ldap.yaml
        containerPath: /etc/ldap.yaml
        propagation: Bidirectional
      - hostPath: /etc/ldap.conf
        containerPath: /etc/ldap.conf
        propagation: Bidirectional
      - hostPath: /etc/novaobserver.yaml
        containerPath: /etc/novaobserver.yaml
        propagation: Bidirectional

In case it helps, this is the resulting kubeadm configmap:

user@debian:~$ kubectl -n kube-system get cm kubeadm-config -o yaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      certSANs:
      - localhost
      - 127.0.0.1
      extraArgs:
        authorization-mode: Node,RBAC
        enable-admission-plugins: NodeRestriction,PodSecurityPolicy
        feature-gates: TTLAfterFinished=true
        runtime-config: ""
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta3
    certificatesDir: /etc/kubernetes/pki
    clusterName: toolforge
    controlPlaneEndpoint: toolforge-control-plane:6443
    controllerManager:
      extraArgs:
        enable-hostpath-provisioner: "true"
        feature-gates: TTLAfterFinished=true
    dns: {}
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: registry.k8s.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.24.17
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/16
    scheduler:
      extraArgs:
        feature-gates: TTLAfterFinished=true
kind: ConfigMap
metadata:
  creationTimestamp: "2024-04-25T09:30:18Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "203"
  uid: 83b048f2-1399-4756-b2b2-aa15da982a8b

What you expected to happen:

No concerning warning would be printed.

Environment:

neolit123 commented 2 months ago

No concerning warning would be printed.

the warning is fine. kind passes all the configs to "kubeadm init" separated with --- so kubeadm complains that you should only pass JoinConfiguration to "join" and not "init".

BenTheElder commented 2 months ago

since kubeadm will ignore it with at most a warning we use a combined template for each api version.

it would be a lot messier to avoid this and it doesn't negatively impact running.