[FEATURE REQUEST] Enable configuration of kubelet `enable-controller-attach-detach` argument via input manifest

[cicharka]

Is your feature request related to a problem? Please describe. Initialization of Kubernetes worker node in Epiphany with kubeadm join is setting enable-controller-attach-detach kubeletExtraArgs parameter to false while default value provided by kubernetes is true. That value, when set to false is blocking attach-detach controller from rook-ceph to attach volume to the pod on worker node, and therefore blocking pod from being created and used.

Describe the solution you'd like. While it can be changed manually, it should be possible to configure enable-controller-attach-detach parameter by users when creating the cluster with epicli.

Feature should include also change in documentation about that parameter, and information about its mandatory value for Block Device and Shared Filesystem storage type.

Note: this change is needed only if user want to setup rook-ceph storage type of Block Device and Shared Filesystem. If user wants to use Object Storage value of that flag doesn't have to be changed.

While testing ensure that change in enable-controller-attach-detach is not affecting any other features provided in Kubernetes by Epiphany.

Additional context See ansible/playbooks/roles/kubernetes_node/templates/kubeadm-join-node.yml.j2 for the template file used for kubeadm join. That file should include change to parse from input manifest value for enable-controller-attach-detach set by user.

Solves problem described here: https://github.com/epiphany-platform/epiphany/issues/3184

More about controller-attach-detach: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ https://www.alibabacloud.com/blog/getting-started-with-kubernetes-%7C-storage-architecture-and-plug-ins_596307

---

DoD checklist

• Changelog
  • [x] updated
  • [ ] not needed
• COMPONENTS.md
  • [ ] updated
  • [x] not needed
• Schema
  • [x] updated
  • [ ] not needed
• Backport tasks
  • [ ] created
  • [x] not needed
• Documentation
  • [x] added
  • [ ] updated
  • [ ] not needed
• [ ] Feature has automated tests
• [ ] Automated tests passed (QA pipelines)
  • [x] apply
  • [x] upgrade
  • [ ] backup/restore
• [x] Idempotency tested
• [x] All conversations in PR resolved
• [ ] Solution meets requirements and is done according to design doc
• [x] Usage compliant with license

[cicharka]

Important: setting value for kubelet enable-controller-attach-detach parameter shouldn't be done via modifying ansible/playbooks/roles/kubernetes_node/templates/kubeadm-join-node.yml.j2 file - see info from https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/: (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) Therefore we should move that flag to the configuration which is templated by ansible/playbooks/roles/kubernetes_master/templates/kubeadm-config.yml.j2 file.