Release Proposal v2.21 - Githubissues

floryut commented 1 year ago

Deprecation / Removal

Drop calico v3.21 support (#9515, @oomichi)

Feature / Major Changes

Add Check resolv.conf is empty to avoid CoreDNS crash (#9502, @yankay)
Add XDG related Helm paths to be removed from reset tasks (#9561, @emiran-orange)
Add a parameter (disable_host_nameservers) to disable host nameservers (#9357, @eminaktas)
Add an option (populate_loadbalancer_apiserver_to_hosts_file) to skip adding load balancer name in the hosts file (#9331, @JRaver)
Add custom options to coredns kubernets plugin (coredns_kubernetes_extra_opts) (#9608, @mvandergiesen)
Add docker support for openEuler linux (#9498, @ErikJiang)
Add support for the OpenEuler Linux (#9494, @ErikJiang)
Add terraform script for Flatcar Linux on Hetzner (#9618, @florianow)
Add the ability to define options for DNS upstream servers (using new variable dns_upstream_forward_extra_opts) (#9311, @emiran-orange)
Add var (ingress_nginx_probe_initial_delay_seconds) for control initialDelaySeconds in ingress-nginx probes (#9405, @zvlb)
Add variable condition snapshot in vSphere CSI (vsphere_csi_block_volume_snapshot) (#9429, @yanggangtony)
Add variable in metrics_server deployment (metrics_server_replicas) to enable HA mode (#9539, @ugur99)
Change dns upstream condition for nodelocaldns when using host_resolvconf (#9378, @unai-ttxu)
Download coredns image to all hosts in k8s_cluster (#9316, @joes)
Enable check mode in DNS Cleanup tasks (#9472, @emiran-orange)
Etcd image has the same tag accross multiple archs (#9516, @hangscer8)
Fix a pre-upgrade node drain rescue task failure when kube_override_hostname is set (#9556, @chadswen)
Fix default value for kubelet_secure_addresses (#9355, @willtrnr)
Provides to change the timeout of first control-plane initialization (#9617, @tu1h)
Remove PodSecurityPolicies in MetalLB for kubernetes 1.25 (#9442, @yanggangtony)
Support Python 3.11 - ruamel.yaml.clib need to be updated to 0.2.7 (#9426, @olivierlemasle)
Support customize the additional sysctl variables using additional_sysctl (#9351, @yankay)
Support patches field in kubeadm v1beta3 in both InitConfiguration and JoinConfiguration (using new variable kubeadm_patches) (#9326, @titaneric)
Switch helm install (from synchronize to copy) to support password authentication (#9343, @ghostloda)
Update api version for pdb and batch (deprecated in 1.25) (#9369, @yankay)
Update dashboard image repo to remove arch flag (#9530, @tu1h)
Update etcd log-level parameter name (new name: ETCD_LOG_LEVEL) (#9540, @ErikJiang)
Update local-volume-provisioner to 2.5.0 + add documentation (#9463, @olivierlemasle)
Update the number of nofile limits in containerd to 65535 (#9507, @ErikJiang)
Upgrade metrics server to v0.6.2 (#9554, @mzaian)
Upgrade the load balancer ( nginx and haproxy ) image version. (#9506, @yankay)
Use kube_apiserver_port variable instead of hard-coding 6443 (#9620, @huangkevin404)
[etcd] Default version to 3.5.5 for k8s 1.25.x (#9419, @mzaian)
Update CoreDNS version to v1.9.3 (#9503, @yankay)
Add the possibility to specify extra domains for the coredns kubernets plugin (using coredns_kubernetes_extra_domains) (#9635, @mvandergiesen)
Streamline ansible_default_ipv4 gathering loop (#9281, @rptaylor)
Update kubernetes dashboard to 2.7.0 (k8s 1.25 support) (#9425, @mzaian)
Skip retry operation with containerd when etcd installed on host VM (#9560, @JRaver)
Update pause image version to v3.8 (#9668, @mzaian)
Enable kubelet_authorization_mode_webhook back by default and remove extra role (#9662, @MrFreezeex)
Terraform gcp can now have extra ingress firewall rules, using new variable extra_ingress_firewalls (#9658, @sathieu)
kubeadm/etcd: use config to download certificate (#9609, @MrFreezeex)

Applications

[argocd] update argocd to v2.5.5 (#9604, @mzaian)
Upcloud: Reclaim policy for PV is now delete (#9574, @robinAwallace)
[Exoscale] Add missing zone input variable (#9495, @ayoubeddafali)
[MetalLB] Avoid MetalLB speaker image download when MetalLB speaker is disabled (#9248, @unai-ttxu)
[Openstack] Replace deprecated "template" Terraform provider with supported "cloudinit" Terraform provider (#9536, @inflatador)
[OpenStack] Updated openstack cloud controller to version v1.25.3 (#9500, @robinAwallace)
[Openstack] Add bastion_allowed_ports to allow custom security group rules on bastion node (#9336, @bl0m1)
[Openstack] Upgrade 1.22.0 to 1.23.4 (#9332, @QcFe) (See Notes 1)
[Openstack] Added override variable, additional server groups and cloudinit config (#9452, @Xartos)
[cinder-csi-nodeplugin] Remove the pods-cloud-data volume (delete upstream) (#9362, @huangkevin404)
[vsphere-csi] Add missing defaults for externalvsphere* variables in the csi_driver/vsphere role (#9664, @rlacko58)
[hetzner] In config, rename ansible groups to use _ instead of - (#9569, @ym)
[kube-vip] Minor changes on Kube VIP configuration parameters (and fix wrong properties) (#9414, @woutergd)
[cert-manager] Upgrade to v1.10.1 (#9512, @rtsp) then v1.11.0 (#9661, @mzaian)
[helm] upgrade to 3.10.3 (#9605, @mzaian)
[ingress-nginx] upgrade to 1.5.1 (#9532, @mzaian)
[vSphere] Removing unneeded terraform dependencie & mark vsphere_password as sensitive (#9672, @sathieu)

Container-Managers

Optimize cgroups settings for node reserved (using new kube_reserved, see docs for more information) (#9209, @shelmingsong)
[Docker] Update docker package to 20.10.20 (partial fix for CVE-2022-39253) (#9410, @floryut)
[containerd] Add support for 1.6.11 (#9544, @yanggangtony)
[containerd] Added variables for unpriviledged ports and icmp (#9517, @Xartos)
[containerd] Allow containerd-common to execute multiple times per play (#9543, @chadswen)
[containerd] Newly started containers will be limited to 16384 open files. To change this number, set containerd_base_runtime_spec_rlimit_nofile, or remove base_runtime_spec from runc runtime to revert to previous behaviour. (#9319, @fungusakafungus)
[containerd] Support v1.6.13 and v1.6.14 (#9585, @yanggangtony)
[containerd] Add config_path var in config.toml.j2 file (#9566, @lengrongfu)
[containerd] Add hashes for containerd versions 1.5.14 , 1.5.15 , 1.5.16 (#9678, @yanggangtony)
[cri-o] Use cri-o from upstream instead of kubic/OBS (#9374, @cristicalin)
[nerdctl] upgrade to version 1.0.0 (#9424, @mzaian)

Network

Bump cni-plugins version to v1.2.0 (#9671, @cyclinder)
Fix remove Cilium CNI failed because the CNI bin dependency (#9563, @yankay)
[Calico] Add cni bin when installing (#9367, @ErikJiang)
[Calico] Add retry for start calico kube controller (#9450, @cleverhu)
[Calico] Adjust calico-kube-controller pod to non hostNetwork pod (#9465, @cyclinder)
[Calico] Adjust calico-kube-controller pod to use hostnetwork if using etcd (#9573, @JSpon)
[Calico] Disable 'Check that IP range is enough for the nodes' (#9491, @mzaian)
[Calico] Update the tag image to support multiple architectures with the same tag (#9529, @ErikJiang)
[Calico] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9395, @yankay)
|Calico] Allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml (using calico_felix_mtu_iface_pattern) (#9330, @shelmingsong)
[Calico] Replace node-role.kubernetes.io/master with control-plane (#9627, @my-git9)
[Calico] upgrade default calico version to v3.24.5 (#9580, @yankay)
[Calico] Add vxlan-v6.calico to the list of NetworkManager unmanaged interfaces (#9631, @cyclinder)
[Calico] Add retry to avoid 'unknown' state for calicoctl (#9633, @tu1h)
[Calico] Update Calico VXLAN offload docs because Calico changed the default value (#9639, @yankay)
[Calico] Add possibility to enable calico floatingIPs feature (using calico_felix_floating_ips) (#9680, @MatthieuFin)
[Cilium] Add download configuration for cilium hubble images (using cilium_enable_hubble variable) (#9376, @ErikJiang)
[Cilium] Add switch cilium_enable_bandwidth_manager (#9441, @dcwbq)
[Cilium] Cleanup cilium-init image from cilium template (#9508, @ErikJiang)
[Cilium] update cilium cli offline download url example (#9458, @cleverhu)
[Cilium] Install Cilium CLI alongside Cilium (#9436, @dcwbq)
[flannel] Initcontainer image now correctly support architecture suffix (#9461, @rollandf)
[flannel] Upgrade version to v0.20.1 (#9528, @ErikJiang)
[flannel] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9365, @yankay)
[flannel] Add wireguard encryption backend as option (#9583, @janaurka)
[flannel] Support dual stack IPv4 & IPv6 networking (#9564, @styshoo)
[flannel] Allow setting the DirectRouting option on VXLAN (#9438, @willtrnr)
[flannel] update to v0.20.2 & make it default (#9675, @mzaian)
[kube-ovn] Update version to v1.10.7 (#9527, @liupeng0518)
[kube-ovn] Remove kube-ovn log directories when reseting (#9625, @JochenFriedrich)
[kube-ovn] Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector (#9594, @JochenFriedrich)
[kube-ovn] Support OVN Interconnect (#9599, @JochenFriedrich)
[multus] added support for mixed type of container engine (#9224, @mr-yaky)

Bug or Regression

Change include to import_playbook in recover_control_plane playbook, to support ansible 2.12+ (#9576, @floryut)
Corrected vsphere directory in docs (#9534, @wojciehm)
Deleting worker nodes is now skipped if there is no kube_control_plane node. (#9430, @kerryeon)
Etcd arch can now support arm64 and amd64 (#9421, @yanggangtony)
Fix cert-manager deployment on hardening environments (#9404, @oomichi)
Fix checksum of ciliumcli v0.12.5 for arm64 (#9614, @oomichi)
Fix inconsistent handling of admission plugin list (kube_apiserver_enable_admission_plugins must be specified as a list of individual plugin names instead of a single item comma-separated list) (#9407, @willtrnr)
Fix kube token dir permissions (#9590, @C-Romeo)
Fix missing control plane taint in kubeadm (#9592, @yankay)
Fix regex for comments nameserver in resolv.conf (#9523, @yankay)
Fix reset for RedHat based distro with major version >=8 (#9537, @dougsland)
Fix wrong cri_socket path for containerd (#9401, @maxime1907)
Fix wrong rbac of the ClusterRole csi-snapshotter-role (#9610, @maxime1907)
Remove coredns_server from supersede_nameserver in dhclient.conf if nodelocaldns is enabled. (#9392, @JiffsMaverick)
Remove immutable flag from /var/lib/kubelet subdirs (#9597, @emiran-orange)
Skip the install of ping package in Fedora CoreOS & Flatcar (#9370, @yankay)
Fix OL9 setup - disable Centos Extras repo creation (#9483, @psvmcc)
Use hostname override in post-remove role, just as pre-remove role does (#9360, @JSpon)
[Calico] Install calico-kube-controller also when using kdd datastore (#9358, @wayfrro)
[Cilium] Fix the Hubble certificate being faulty because the cluster name has an hard coded value (#9340, @dcwbq)
[Cilium] Fix tls settings not being properly set (#9457, @charlychiu)
[Cilium] Remove trailing backslash and fix yaml indent (#9339, @reneluria)
[Openstack] Fix a race condition in terraform causing ports to not get an IP (#9345, @bl0m1)
[Openstack] Fix missing permissions for Openstack cloud-controller-manager (#9335, @bl0m1)
[gVisor] Allow installation on arm architecture systems (#9493, @ErikJiang)
[kube-ovn] Cluster support for ovn-central (#9596, @JochenFriedrich)
[upcloud] Fixed issue where DNS would be blocked while using allowlist (#9510, @Xartos)

Other (Cleanup or Flake)

Use the correct api version and resource type in secrets_encryption.yaml.j2 (#9575, @LukasNajman)
Minor cleanup of docs by rephrasing some unclear documentation (#9621, @anthonyeleven)
Add mirror doc to support mirror usage. (#9396, @yankay)
[CI] Add check_typo job (and fix a bunch of typos) (#9361, @oomichi)
[CI] Stop using python 'test' internal package (#9454, @olivierlemasle)
[CI] Update securityContext of netchecker (#9398, @oomichi)
[CI] Use agnhost instead of busybox for network test (#9390, @oomichi)
[CI] Add ubuntu20 hardening job (#9359, @oomichi)
[CI] Fix YAML format in hardening.md file (#9387, @oomichi)
[CI] Make vagrant-ubuntu20-flannel voting (by removing allow failure) (#9469, @oomichi)
[CI] Update sonobuoy version to a more recent one (#9485, @oomichi)
[CI] Increase the fedora memory at CI to fix the CI broken (#9640, @yankay)
[CI] Add CI for rockylinux9 and cilium (#9562, @yankay)

Component versions

Core
- kubernetes v1.25.6
- etcd v3.5.6
- docker v20.10 (cri_dockerd: v0.3.0)
- containerd v1.6.15
- cri-o v1.24
Network Plugin
- cni-plugins v1.2.0
- calico v3.24.5
- cilium v1.12.1
- flannel v0.20.2
- kube-ovn v1.10.7
- kube-router v1.5.1
- multus v3.8
- weave v2.8.1
- kube-vip v0.5.5
Application
- cert-manager v1.11.0
- coredns v1.9.3
- ingress-nginx v1.5.1
- krew v0.4.3
- argocd v2.5.7
- helm v3.10.3
- metallb v0.12.1
- registry v2.8.1
Storage Plugin
- cephfs-provisioner v2.1.0-k8s1.11
- rbd-provisioner v2.1.1-k8s1.11
- aws-ebs-csi-plugin v0.5.0
- azure-csi-plugin v1.10.0
- cinder-csi-plugin v1.22.0
- gcp-pd-csi-plugin v1.4.0
- local-path-provisioner v0.0.22
- local-volume-provisioner v2.5.0

Known issues

N/A

Notes

As stated in cloud-provider-openstack:1.23.0: Load balancers don't relate to a dedicated Service anymore, any scripts relying on that relationship previously need to change to use the load balancer tags instead

yankay commented 1 year ago

The Calico VXLAN Offload Issue has been fixed at the calico 3.25 & 3.24.5 by https://github.com/projectcalico/calico/pull/6842 , So I think the kubespray may not need the "note" to disable it. :-)

cyclinder commented 1 year ago

Can https://github.com/kubernetes-sigs/kubespray/pull/9631 be part of this RELEASE?

floryut commented 1 year ago

The Calico VXLAN Offload Issue has been fixed at the calico 3.25 & 3.24.5 by projectcalico/calico#6842 , So I think the kubespray may not need the "note" to disable it. :-)

Indeed I saw you PR, update done

floryut commented 1 year ago

Can #9631 be part of this RELEASE?

As soon as it's rebased and go through the CI 👍

mvandergiesen commented 1 year ago

Hi,

Can #9635 be part of the release? https://github.com/kubernetes-sigs/kubespray/pull/9635 https://github.com/kubernetes-sigs/kubespray/pull/9635

Thanks, Marijn

On 5 Jan 2023, at 04:40, Cyclinder @.***> wrote:

Can #9631 https://github.com/kubernetes-sigs/kubespray/pull/9631 be part of this RELEASE?

— Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/kubespray/issues/9638#issuecomment-1371748277, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANJIBRW7XDKUPKQB6RLQGJLWQY7ETANCNFSM6AAAAAATRHP6E4. You are receiving this because you were mentioned.

JRaver commented 1 year ago

Hi, please add this PR to the release #9560

floryut commented 1 year ago

Hi, Can #9635 be part of the release? #9635 <#9635> Thanks, Marijn … On 5 Jan 2023, at 04:40, Cyclinder @.***> wrote: Can #9631 <#9631> be part of this RELEASE? — Reply to this email directly, view it on GitHub <#9638 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANJIBRW7XDKUPKQB6RLQGJLWQY7ETANCNFSM6AAAAAATRHP6E4. You are receiving this because you were mentioned.

Done