[WIP] Release Proposal v2.27

[tico88612]

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required Change kubeadm_patches format to use an array of inline patch instead of patch files. See the example for new format. (#11521, @VannTen)
  • Action required Removes the generation of static tokens for every node in the cluster when kube_token_auth: true (#11567, @VannTen)
  • Action required The kubelet_node_{config_extra_args,custom_flags} are removed. Use kubelet_{config_extra_args,custom_flags} in <your_inventory>/group_vars/kube_node.yml. The {kube,system}_master_{cpu,memory,ephemeral-storage,pid} are removed. Use the {kube,system}_{cpu,memory,ephemeral-storage,pid} variables in <your_inventory>/group_vars/kube_control_plane.yml. kubelet_custom_flags can no longer be a string, an array is required. (#10643, @VannTen)
  • Action required k8s_cluster group is now automatically defined, it can be removed from your inventory if you're not using it for group_vars (#11559, @VannTen)
  • Action required kubeadm_ignore_preflight_errors is introduced to ignore specific preflight checks from kubeadm. The previous was effectively all, so some errors might surface during upgrade, in which cases, users should add the ones they choose to ignore to that variable. (#11710, @VannTen)

Changes by Kind

Container-Managers

• [containerd] Make containerd 1.7.22 default (#11554, @foobaar)

API Change

• If you use CRI-O and want to keep runc as your container default runtime when you upgrade cluster, you must set runc_enable: true and crio_default_runtime: "runc". Make CRI-O's default runtime configurable CRI-O v1.31 default runtime change to crun Crun upgrade to 1.17 Skopeo upgrade to v1.16.1 (#11601, @tico88612)

Feature

• Add CI for openeuler 24.03 Add CI Image for openeuler 24.03, 22.03 (#11689, @yankay)
• Add a new CRI-O crio_root variable (#11692, @toliger)
• Add external Oracle cloud infrastructure cloud controller manager (#11378, @tico88612)
• Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 (#11581, @DirkTheDaring)
• Add optional support for Host Firewall and PolicyAuditMode features in Cilium (#11230, @ledroide)
• Add support Fedora 39/40 (#11573, @tico88612)
• Add support to use existing fips with terraform OpenStack (#11558, @anders-elastisys)
• Add the support of network isolation configuration in Multus. (#11605, @Sispheor)
• Added support for using ntpsec (#11665, @davidumea)
• Adds ingress_nginx_service_annotations variable to allow setting annotations for ingress-nginx controller service (#11544, @ThisIsQasim)
• Adds nodelocaldns_additional_configs variable (#11657, @0x4c6565)
• Allow disabling cilium hubble-ui using cilium_enable_hubble_ui variable (#10939, @pedro-peter)
• Configuration can now be supplied to ImagePolicyWebhook and PodNodeSelector admission plugins (#11471, @VannTen)
• Default to containerd v1.7.23 (#11642, @robertvolkmann)
• Feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition (#11570, @mirwan)
• Feat: Kubeadm config API support v1beta4 (#11674, @tico88612)
• Make Kubernetes v1.31.2 default Add hashes for Kubernetes 1.30.6 and 1.29.10 (#11662, @robertvolkmann)
• Make Kubernetes v1.31.3 default Add hashes for Kubernetes 1.30.7 and 1.29.11 (#11737, @tico88612)
• Make cri-dockerd log level configurable (#11646, @mirwan)
• Make kubernetes v1.31.1 default Add hashes for Kubernetes 1.31.0 and 1.31.1 (#11533, @philipsabri)
• Partial Support of Cilium v1.16+ - kube-proxy replacement var changes Add optional support for configuring BGP Control Plane, IP Load Balancer Pools , Legacy BGP Peer Config v1 and BGP Config v2 features in Cilium (#11620, @logicsys)
• Remove support Fedora 37/38 (#11600, @tico88612)
• Reset operation: remove /var/log/containers and disable service auto-boot, make sure that multi-user.target.wants is deleted. (#11501, @leeonfu)
• Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 (#11757, @ErikJiang)
• Update crictl to version v1.31.1 for Kubernetes 1.31 Update crictl to version v1.30.1 for Kubernetes 1.30 (#11661, @robertvolkmann)
• Update multus to v4.1.0 (#11434, @ThisIsQasim)
• Upgrade CoreDNS version to v1.11.3 (#11653, @tico88612)
• Upgrade OpenStack Cloud Controller Manager to v1.31.1 (#11738, @tico88612)
• Upgrade pause container to 3.10 (#11695, @tico88612)
• [cert-manager] upgrade to v1.15.3 (#11668, @tico88612)
• [cilium] Make cilium 1.15.9 default (#11593, @foobaar)
• [cri-o] Switch binaries to libexecdir Update youki version to 0.4.1 to fix ci. (#11584, @yankay)
• [etcd] Default version to 3.5.16 for 1.28, 1.29, 1.30, 1.31 (#11572, @janosbabik)
• [need notice] update containerd max_container_log_line_size default value to 16384 (#11585, @KubeKyrie)
• [nerdctl] Default version to 1.7.7 (#11575, @janosbabik)
• [runc] Default version to v1.1.14 (#11574, @janosbabik)

Documentation

• No longer support in-tree cloud provider, please delete or write external to the cloud_provider variable. (#11633, @tico88612)
• Remove inventory_builder scripts and contrib/dind (#11748, @VannTen)
• Update dns-stack.md reference in docs/ansible/vars.md (#11745, @emmanuel-ferdman)

Failing Test

• Fix CI: elastx cleanup security groups (#11411, @yankay)
• Update rockylinux9 base image to 9.5 (#11744, @yankay)

Bug or Regression

• Action required Running kubespray with --limit without cached facts is no longer supported. Improves the scaling for large clusters. (#11598, @VannTen)
• Always copy cert generation script to first etcd to pick up fixes on existing clusters (#11612, @VannTen)
• Fix Cilium agent permission can't read loadbalancerippools and secrets (#11466, @foobaar)
• Fix collection usage for calico and other configuration depending on .sh and .conf files in Kubespray (#11707, @VannTen)
• Fix error in boostrap-os when git does not handle symlinks (#11508, @VannTen)
• Fix format of kubeadm-config v1beta4 (#11709, @VannTen)
• Fix kube-vip container securityContext (#11647, @KubeKyrie)
• Fix openEuler system packages installation (#11688, @VannTen)
• Fix pretty-printing (in kubectl) of nodelocaldns and coredns configmap when using dns_upstream_forward_extra_opts with an empty value option. (#11694, @VannTen)
• Fix task naming in bootstrap-os (#11714, @ErikJiang)
• Fix the check for cached data when using --limit (#11693, @VannTen)
• Fix the usage of --limit when using legacy groups (#11577, @VannTen)
• Fix: cannot stop & remove all cri containers via remove_node.yml (#11631, @tico88612)
• Fixed: VSphere CSI and CPI drivers and are now retrieved from registry.k8s.io instead of gcr.io, as they have been deleted from the latter. Only a few recent versions are available in the new repository; if you have pinned vsphere_csi_controller, vsphere_csi_driver_image_tag or vsphere_syncer_image_tag to a version older than v3.1.2, please check if that version is available from the new repository. The same goes for external_vsphere_cloud_controller_image_tag which can no longer be latest, and should align with the running version of Kubernetes. It now defaults to v1.31.0. (#11564, @luringens)
• HA etcd cluster keeps quorum during upgrades. (#11677, @VannTen)
• Kubeadm images (kube-controller-manager,kube-scheduler,kube-apiserver,kube-proxy) are properly downloaded, including when using the download cache. (#11741, @VannTen)
• Make sure kubespray-defaults can be executed successfully by executing bootstrap-os first (#11441, @huangkevin404)
• Make upcloud csi_driver use the correct pull secret (#11597, @VannTen)
• Modifies Helm parameters wait and atomic to be set to false when using kube_network_plugin=cni to prevent deployment issues with kubelet-csr-approver. (#11704, @M-JavadHeydarpour)
• Remove invalid extraArgs entry and update template file reference (#11703, @agravgaard)
• Update calico-nopde template and remove flexvol-driver initContainer (#11634, @KubeKyrie)
• Use correct version for community.general collection (#11724, @VannTen)

Other (Cleanup or Flake)

• Cleanup older terminology, replace "master" with "control plane" (#11394, @bogd)
• Drop support for Kubernetes 1.28.x minimum version now is 1.29.x Drop support for CRI-O 1.28.x minimum version now is 1.29.x (#11609, @yankay)
• Fix roles/download/tasks/download_file.yml task name typo (#11684, @dmncmn)
• Remove pip install . support and rpm spec file (#11760, @VannTen)
• Replace deprecated unarchive.copy with unarchive.remote_src (#11207, @Payback159)
• Update KUBESPRAY_VERSION for v2.26.0 (#11511, @yankay)
• containerd_use_config_path is removed as kubespray now always use containerd config_path configuration. (#11755, @VannTen)

[VannTen]

Hum I need to fix some of my release notes it seems

[VannTen]

I've updated the release note of my PR, the rendering should be better now.