Enabling Feature Gates and kubernetes custom configuration on CAPA

[T-Kukawka]

Currently Vintage is a heavy user of k8s initiator app which allows customers to customize the settings directly on the nodes file systems/configuration. Following the investigation of the currently used features: https://github.com/giantswarm/giantswarm/issues/28558, we have to make the adjustments to CAPA such that the migration of given features is possible and there are 'features' enabled to migrate to.

Feature Gates

As a customer i would like to be able to enable the features gates available on the k8s. Currently on Vintage this part is done via the k8s-initiator-app.

There are several groups of feature gates used by customers:

1. The k8s api feature gates such as:

   • TTLAfterFinished=true
   • JobTrackingWithFinalizers=true
   • EphemeralContainers=true

2. k8s Controller Manager:

   • TTLAfterFinished=true
   • CSIMigrationAWS=true
   • JobTrackingWithFinalizers=true

3. Admission controller:

   • configure API to enable PodNodeSelector admission controller - https://github.com/giantswarm/k8s-initiator-app/blob/main/examples/values_enable_admission_controller.yaml

k8s API configuration

Additionally via k8s-initiator-app it is available to customers to edit the e.g. audit-policy for k8s api such as: https://github.com/giantswarm/k8s-initiator-app/blob/main/examples/values_audit_policy.yaml

This feature is extensively used by one customer, where several policies are applied per cluster.

kubelet max log size

This feature is also used by one customer:

command:
 - sh
 - -c
 - |
   KUBELETCONFIG=config/kubelet.yaml
   if ! grep -q containerLogMaxSize $KUBELETCONFIG; then
     echo "Kubelet containerLogMaxSize Settings not present, applying"
     echo "Previous Kubelet Config  yaml"
     cat $KUBELETCONFIG
     echo "containerLogMaxSize: 30Mi" >> $KUBELETCONFIG
     echo "New kubelet config:"
     cat $KUBELETCONFIG;
   else
     echo "Kubelet containerLogMaxSize Settings present"
     echo "Replacing the Kubelet containerLogMaxSize"
     cp $KUBELETCONFIG /tmp/tmp_api.yaml
     sed -i -r -e "s,containerLogMaxSize:(.+),containerLogMaxSize: 30Mi," /tmp/tmp_api.yaml
     mv /tmp/tmp_api.yaml $KUBELETCONFIG
     echo "New kubelet config:"
     cat $KUBELETCONFIG;
   fi
 extraHostPaths:
 - hostPath: /etc/kubernetes/config
   mountPath: /config
   name: kubefs
 initiators:
 - bash
 masterOnly: false
 nodeSelector:
   kubernetes.io/role: worker
 psp:
   privileged: true

[njuettner]

I checked the feature gates mentioned from above

k8s api feature gates

TTLAfterFinished=true - GA and active since 1.23, we do not need to add this JobTrackingWithFinalizers=true - Beta and active by default since 1.25, we do not need to add this EphemeralContainers=true - Beta and active by default 1.23, we do not need to add this

k8s Controller Manager

TTLAfterFinished=true - see above no action needed CSIMigrationAWS=true - Beta and active since 1.23, we do not need to add this JobTrackingWithFinalizers=true - see above

Regarding Feature Gates in general, I think it would make sense having a feature gate configuration for each component but only BETA:

• kubelet .Values.internal.advancedConfiguration.controlPlane.kubelet and .Values.internal.advancedConfiguration.workers.kubelet
• kube-apiserver .Values.internal.advancedConfiguration.controlPlane.apiServer
• kube-controller-manager .Values.internal.advancedConfiguration.controlPlane.controlerManager
• kube-scheduler .Values.internal.advancedConfiguration.controlPlane.kubeScheduler
• kube-proxy (is not needed due to Cilium)

[T-Kukawka]

For documentation purposes, i would ask to add each topic as a separate entry in directory such as: https://github.com/giantswarm/capi-migration-cli/pull/13

[Gacko]

I read https://github.com/giantswarm/giantswarm/issues/28558 and compared it to what's listed here and what @njuettner stated above.

@T-Kukawka: If I'm getting everything right, the only three features we need to implement to get this story done are the following:

• PodNodeSelector Admission Controller
• API Server Audit Policies
• Kubelet containerLogMaxSize

Is this correct?

[T-Kukawka]

correct @Gacko 👍

[Gacko]

We already have a quite extensive audit policy in place for CAPI: https://github.com/giantswarm/cluster/blob/main/helm/cluster/files/etc/kubernetes/policies/audit-policy.yaml

[T-Kukawka]

well, customers like to set their own as on Vintage, u can refer to the discovery ticket @Gacko

[Gacko]

Ok, fine. So I think we just need the possibility to add additional rules to the existing policy.

[T-Kukawka]

yeah, exactly 👍

[T-Kukawka]

@Gacko can u provide documentation for those features? or references so we can document as well as integrate that in the migration CLI?

[Gacko]

Sure, I'll put something in the README.md of this repository. IIRC the old way of using the k8s-initiator-app was a bit tricky as you needed to change stuff in the code to customize it, but now it's more about setting the right values, that's it.

[T-Kukawka]

@Gacko i have added this for the node-tainting - i believe we should do the same for your case: https://github.com/giantswarm/capi-migration-cli/blob/main/k8s-initiator-features/node-tainting.md

We can have a call if u like

[Gacko]

Added requested docs here.