Planning upgrade to EKS 1.28

[poornima-krishnasamy]

Go through the release notes of EKS 1.28 and create a plan to upgrade our clusters

Things to consider:

Review changelog & release notes ✅ EKS Module support at current version? ✅ Are there any API deprecations & removals? (Check insights) ✅ Are there new components being added? ✅ What changes are being introduced to current components? ✅ Are there changes to core infra of the CP required? i.e. Are all our current components compatible with ? Are there changes users need to make? ✅ Do we need to expand any of our smoke/integration testing? ✅ Create additional tickets needed for any findings specific to this upgrade ✅

Cluster upgrade Runbook: https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html

Related to: #5569

[jaskaransarkaria]

Deprecations and removals

Removals:

[Removal of CSI Migration for GCE PD](https://github.com/kubernetes/enhancements/issues/1488)

Deprecations:

[Ceph RBD in-tree plugin](https://github.com/kubernetes/kubernetes/pull/118303)
[Ceph FS in-tree plugin](https://github.com/kubernetes/kubernetes/pull/118143)

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#no-really-you-must-read-this-before-you-upgrade

CephFS volume plugin (kubernetes.io/cephfs) has been deprecated in this release and will be removed in a subsequent release. The alternative is to use the CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes cluster. (https://github.com/kubernetes/kubernetes/pull/118143, @humblec)

Deprecated support for CSI migration of Ceph RBD volumes. Users who were relying on Kubernetes' ability to migrate to an out-of-tree storage driver should complete that migration before the support for it is removed. (https://github.com/kubernetes/kubernetes/pull/118303, @carlory)

RBD volume plugin (kubernetes.io/rbd) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (https://github.com/kubernetes/kubernetes/pull/118552, @humblec)

Changes by Kind

Deprecation

• Changed kubectl version default output to be identical to what kubectl version --short printed, and removed --short flag entirely. (https://github.com/kubernetes/kubernetes/pull/116720, @soltysh)
• Kube-controller-manager deprecate --volume-host-cidr-denylist and --volume-host-allow-local-loopback flags. (https://github.com/kubernetes/kubernetes/pull/118128, @carlory) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Kubelet: The --azure-container-registry-config flag has been deprecated and will be removed in a future release, please use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. (https://github.com/kubernetes/kubernetes/pull/118596, @SataQiu) [SIG Node]
• Removed tracking annotation from validation and defaulting. (https://github.com/kubernetes/kubernetes/pull/117633, @kannon92)
• Removed withdrawn feature NetworkPolicyStatus. (https://github.com/kubernetes/kubernetes/pull/115843, @rikatz)
• The deprecated flag --lock-object-namespace and --lock-object-name have been removed from kube-scheduler. Please use --leader-elect-resource-namespace and --leader-elect-resource-name or ComponentConfig instead to configure those parameters. (https://github.com/kubernetes/kubernetes/pull/119130, @SataQiu) [SIG Scheduling]
• KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In a future release, Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (https://github.com/kubernetes/kubernetes/pull/119007, @aramase)

[jaskaransarkaria]

What's New

• Changes to supported skew between control plane and node versions:
  • Kubernetes v1.28 expands the supported skew between core node and control plane components by one minor version, from n-2 to n-3, so that node components (kubelet and kube-proxy) for the oldest supported minor version work with control plane components (kube-apiserver, kube-scheduler, kube-controller-manager, cloud-controller-manager) for the newest supported minor version.

• Beta support for enabling swap space on Linux

  • This adds swap support to nodes in a controlled, predictable manner so that Kubernetes users can perform testing and provide data to continue building cluster capabilities on top of swap.

  There are two distinct types of users for swap, who may overlap:

  • Node administrators, who may want swap available for node-level performance tuning and stability/reducing noisy neighbor issues.

  • Application developers, who have written applications that would benefit from using swap memory.

• Source code reorganization for control plane components

New components / features

Graduations to stable

This release includes a total of 12 enhancements promoted to Stable:

• kubectl events
• Retroactive default StorageClass assignment
• Non-graceful node shutdown
• Support 3rd party device monitoring plugins
• Auth API to get self-user attributes
• Proxy Terminating Endpoints
• Expanded DNS Configuration
• Cleaning up IPTables Chain Ownership
• Minimizing iptables-restore input size
• Graduate the kubelet pod resources endpoint to GA
• Extend podresources API to report allocatable resources
• Move EndpointSlice Reconciler into Staging

Notable features

• Graduated the LegacyServiceAccountTokenTracking feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key kubernetes.io/legacy-token-last-used). (https://github.com/kubernetes/kubernetes/pull/117591, @zshihang) [SIG API Machinery, Auth and Testing]
• Moved non-graceful node shutdown to GA. (https://github.com/kubernetes/kubernetes/pull/118228, @carlory)
• Renamed PodHasNetwork to PodReadyToStartContainers. (https://github.com/kubernetes/kubernetes/pull/117702, @kannon92) [SIG Node and Testing]
• The short names vwc and mwc were introduced for the resources validatingwebhookconfigurations and mutatingwebhookconfigurations. (https://github.com/kubernetes/kubernetes/pull/117535, @hysyeah)
• kube-proxy in iptables mode will now have separate sync_full_proxy_rules_duration_seconds\nand sync_partial_proxy_rules_duration_seconds (in addition to the existing\nsync_proxy_rules_duration_seconds), giving better information about the duration of each \nsync type, rather than only giving a weighted average of the two sync types together. (https://github.com/kubernetes/kubernetes/pull/117787, @danwinship)
• [Kube-proxy]: Implemented connection draining for terminating nodes. (https://github.com/kubernetes/kubernetes/pull/116470, @alexanderConstantinescu)
• Added the implementation for PodRecreationPolicy to wait for the creation of pods once the existing ones are fully terminated. (https://github.com/kubernetes/kubernetes/pull/117015, @kannon92)
• Added a container image for kubectl at registry.k8s.io/kubectl across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) (https://github.com/kubernetes/kubernetes/pull/116672, @dims) [SIG Architecture and Release]
• Added a new command line argument --interactive to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. (https://github.com/kubernetes/kubernetes/pull/114530, @ardaguclu) [SIG CLI and Testing]
• Added '--concurrent-cron-job-syncs' flag for kube-controller-manager to set the number of workers for cron job controller. (https://github.com/kubernetes/kubernetes/pull/117550, @borgerli)
• Added '--concurrent-job-syncs' flag for kube-controller-manager to set the number of job controller workers. (https://github.com/kubernetes/kubernetes/pull/117138, @tosi3k)
• Added DisruptionTarget condition to the pod preempted by kubelet to make room for a critical pod. (https://github.com/kubernetes/kubernetes/pull/117586, @mimowo)

Other features

• A ValidatingAdmissionPolicy now has its messageExpression field checked against resolved types. (https://github.com/kubernetes/kubernetes/pull/119209, @jiahuif) [SIG API Machinery]

• Added --concurrency flag to configure the concurrency of kubectl diff execution, defaults to 1. (https://github.com/kubernetes/kubernetes/pull/118810, @brancz)

• Added ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache. (https://github.com/kubernetes/kubernetes/pull/118508, @serathius)

• Added apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics. (https://github.com/kubernetes/kubernetes/pull/119311, @ivelichkovich)

• Added a new feature gate, SchedulerQueueingHints (enabled by default). The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins. In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. (https://github.com/kubernetes/kubernetes/pull/119328, @sanposhiho) [SIG Scheduling]

• Added full cgroup v2 swap support for both Limited and Unlimited swap.

• When LimitedSwap is enabled the swap limit would be automatically calculated for Burstable QoS pods. For Best-Effort/Guaranteed QoS pods, swap would be disabled.

• Containers with memory requests equal to their memory limits also won't have swap access, and it is a way to opt-out of swap for a single container.

• The formula for the swap limit for Burstable QoS pods is: (/)*.

• Support for cgroup v1 is removed. (https://github.com/kubernetes/kubernetes/pull/118764, @iholder101)

• Added handling for pods in podgc for PodReplacementPolicy or PodDisruption. (https://github.com/kubernetes/kubernetes/pull/118772, @kannon92)

• Added reason to metric attachdetach_controller_forced_detaches in the attach detach controller. (https://github.com/kubernetes/kubernetes/pull/119185, @xing-yang)

• Added support for pod hostNetwork field selector (https://github.com/kubernetes/kubernetes/pull/110477, @halfcrazy) [SIG Apps and Node]

• Added swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). (https://github.com/kubernetes/kubernetes/pull/118865, @iholder101)

• Allow to monitor client-go DNS resolver latencies via rest_client_dns_resolution_duration_seconds Prometheus metric. (https://github.com/kubernetes/kubernetes/pull/115357, @mfojtik)

• Apiserver adds two new metrics etcd_requests_total and etcd_request_errors_total that allow users to monitor requests to etcd storage, split by operation and resource type. (https://github.com/kubernetes/kubernetes/pull/117222, @iyear) [SIG API Machinery]

• Bumped distroless-iptables to 0.2.6 based on Go 1.20.6. (https://github.com/kubernetes/kubernetes/pull/119365, @xmudrii)

• Bumped metrics-server to v0.6.3. (https://github.com/kubernetes/kubernetes/pull/117120, @dgrisonnet)

• CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". (https://github.com/kubernetes/kubernetes/pull/118804, @benluddy) [SIG API Machinery]

• CRI: exposed commit memory bytes in container stats specific to Windows (https://github.com/kubernetes/kubernetes/pull/119238, @kiashok)

• Client-go now exposes two new metrics to monitor the client-go logic that generate http.Transports for the clients.

  • rest_client_transport_cache_entries is a gauge metric with the number of existing entries in the internal cache
  • rest_client_transport_create_calls_total is a counter that increments each time a new transport is created, storing the result of the operation needed to generate it: hit, miss or uncacheable. (https://github.com/kubernetes/kubernetes/pull/117295, @aojea)

• Cloud controller manager's node controller now emits timing metrics for initial Node synchronization. These metrics measure the delay between the creation of a new Node and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's --concurrent-node-syncs flag. (https://github.com/kubernetes/kubernetes/pull/119241, @cartermckinnon) [SIG Cloud Provider and Instrumentation]

• Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation." (https://github.com/kubernetes/kubernetes/pull/118936, @pohly)

• Enabled use of pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport. (https://github.com/kubernetes/kubernetes/pull/118691, @giuseppe)

• External credential provider plugins will now have their standard error output logged by kubelet upon failures. (https://github.com/kubernetes/kubernetes/pull/117448, @cartermckinnon)

• Faster scheduling when ResourceClaims are involved. (https://github.com/kubernetes/kubernetes/pull/119078, @pohly)

• Fixed the alpha CloudDualStackNodeIPs feature. (https://github.com/kubernetes/kubernetes/pull/118329, @danwinship)

• Graduated the ProbeTerminationGracePeriod feature gate to GA. (https://github.com/kubernetes/kubernetes/pull/114307, @rphillips)

• Hashing of KeyID in Logs

  • This release adds a feature to hash the KeyID values in the logs. The KeyID values are sensitive information that should not be exposed in plain text in the logs. By hashing the KeyID values, we can protect the confidentiality of the data while still being able to log the necessary information. (https://github.com/kubernetes/kubernetes/pull/118988, @nilekhc) [SIG API Machinery, Auth and Testing]

• Implemented alpha support for a drop-in kubelet configuration directory. (https://github.com/kubernetes/kubernetes/pull/119390, @sohankunkerkar)

• In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. (https://github.com/kubernetes/kubernetes/pull/116443, @benluddy) [SIG API Machinery and Testing]

• Introduce support for CEL optionals (see CEL spec proposal 246). This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable safe rollback on downgrade. (https://github.com/kubernetes/kubernetes/pull/118339, @jpbetz) [SIG API Machinery, Auth, Cloud Provider and Testing]

• Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. (https://github.com/kubernetes/kubernetes/pull/118209, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]

• Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim (https://github.com/kubernetes/kubernetes/pull/117718, @aojea) [SIG Network, Testing and Windows]

• Kube-proxy service health returns http header X-Load-Balancing-Endpoint-Weight with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. (https://github.com/kubernetes/kubernetes/pull/118999, @cezarygerard)

• Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. (https://github.com/kubernetes/kubernetes/pull/119012, @pohly)

• Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. (https://github.com/kubernetes/kubernetes/pull/116254, @pohly) [SIG Auth and Testing]

• Kubelet: un-deprecated --provider-id flag. (https://github.com/kubernetes/kubernetes/pull/116530, @pacoxu)

• Kubernetes is now built with Go 1.20.4. (https://github.com/kubernetes/kubernetes/pull/117744, @xmudrii) [SIG Release and Testing]

• Kubernetes is now built with Go 1.20.5. (https://github.com/kubernetes/kubernetes/pull/118507, @jeremyrickard)

• Kubernetes is now built with Go 1.20.6. (https://github.com/kubernetes/kubernetes/pull/119324, @xmudrii)

• Metric scheduler_scheduler_goroutines is removed. Use scheduler_goroutines instead. (https://github.com/kubernetes/kubernetes/pull/117727, @kerthcet) [SIG Scheduling]

• Migrated pkg/controller/endpoint to contextual logging. (https://github.com/kubernetes/kubernetes/pull/116755, @my-git9)

• Migrated pkg/scheduler/framework/preemption to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/116835, @mengjiao-liu)

• Migrated pod-security-admission to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/114471, @Namanl2001) [SIG Apps and Auth]

• Migrated controller functions to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/116930, @fatsheep9146) [SIG API Machinery, Apps, Network, Node, Storage and Testing]

• Migrated the Job controller (within kube-controller-manager) to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/116910, @fatsheep9146) [SIG API Machinery, Apps and Testing]

• Migrated the EndpointSlice and EndpointSliceMirroring controllers (within kube-controller-manager) to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/115295, @Namanl2001) [SIG API Machinery, Apps, Network and Testing]

• Migrated the certificate controller (within kube-controller-manager) to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/113994, @mengjiao-liu) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]

• Migrated the noderesources scheduler plugin to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/116748, @mengjiao-liu)

• Migrated the podtopologyspread scheduler plugins to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/116797, @mengjiao-liu) [SIG Instrumentation and Scheduling]

• New CEL Library functions to support Kubernetes Quantities. (https://github.com/kubernetes/kubernetes/pull/118803, @alexzielenski) [SIG API Machinery]

• New Metrics Added for Encryption Configuration Controller

• This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:

• apiserver_encryption_config_controller_automatic_reload_failures_total: Total number of failed automatic reloads of encryption configuration.

• apiserver_encryption_config_controller_automatic_reload_success_total: Total number of successful automatic reloads of encryption configuration.

• apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds: Timestamp of the last successful or failed automatic reload of encryption configuration.

• These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. (https://github.com/kubernetes/kubernetes/pull/119008, @nilekhc)

• New staging repo has been created for the EndpointSlice reconciler. (https://github.com/kubernetes/kubernetes/pull/118953, @mskrocki)

• Promoted ServiceNodePortStaticSubrange feature gate to beta, and it will be enabled by default. (https://github.com/kubernetes/kubernetes/pull/117877, @xuzhenglun)

• Promoted the following apiserver flowcontrol metrics to Beta:

• apiserver_flowcontrol_request_wait_duration_seconds

• apiserver_flowcontrol_current_executing_seats

• apiserver_flowcontrol_nominal_limit_seats

• apiserver_flowcontrol_rejected_requests_total

• apiserver_flowcontrol_dispatched_requests_total

• apiserver_flowcontrol_current_inqueue_requests

• apiserver_flowcontrol_current_executing_requests (https://github.com/kubernetes/kubernetes/pull/119110, @andrewsykim)

• Replaced apiserver_storage_db_total_size_in_bytes with apiserver_storage_size_bytes metric. (https://github.com/kubernetes/kubernetes/pull/118812, @serathius)

• Scheduler now waits for handlers to finish syncing before the scheduling cycles start. (https://github.com/kubernetes/kubernetes/pull/116729, @AxeZhan)

• Set metrics-server's metric-resolution to 15s. (https://github.com/kubernetes/kubernetes/pull/117121, @dgrisonnet) [SIG Cloud Provider and Instrumentation]

• SubjectAccessReview requests sent to webhook authorizers now default spec.resourceAttributes.version to * if unset. (https://github.com/kubernetes/kubernetes/pull/116937, @AxeZhan) [SIG Apps and Auth]

• Supported specifying a custom retry period for cloud load-balancer operations. (https://github.com/kubernetes/kubernetes/pull/94021, @timoreimann)

• The "value" part in the wait --for=jsonpath='{expression}'[=value] is now optional. If the value is not provided i.e., the command looks like wait --for=jsonpath='{expression}' then the wait condition is interpreted as matched when the expression returns any single JSON value like object or a literal. (https://github.com/kubernetes/kubernetes/pull/118160, @minherz)

• The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. (https://github.com/kubernetes/kubernetes/pull/117025, @sanposhiho) [SIG Scheduling]

• The AdvancedAuditing feature gate that graduated to GA in v1.12 (and was unconditionally enabled) has been removed. (https://github.com/kubernetes/kubernetes/pull/118763, @Shubham82)

• The ExpandedDNSConfig feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. (https://github.com/kubernetes/kubernetes/pull/116741, @gjkim42) [SIG Apps, Network and Node]

• The apiserver debug endpoint /debug/api_priority_and_fairness/dump_requests has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. (https://github.com/kubernetes/kubernetes/pull/119009, @MikeSpreitzer) [SIG API Machinery]

• The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of kubectl --help ...

• Changed kubectl create secret --help description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. (https://github.com/kubernetes/kubernetes/pull/117930, @LronDC)

• The scheduler skips the InterPodAffinity Score plugin when nothing to do with the Pod. It will affect some metrics values related to the InterPodAffinity Score plugin. (https://github.com/kubernetes/kubernetes/pull/117794, @utam0k) [SIG Scheduling]

• The scheduler skips the PodTopologySpread Filter plugin if no spread constraints. It will affect some metrics values related to the PodTopologySpread Filter plugin. (https://github.com/kubernetes/kubernetes/pull/117683, @utam0k)

• The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod. It will affect some metrics values related to the PodTopologySpread Score plugin. (https://github.com/kubernetes/kubernetes/pull/118608, @utam0k)

• Updated etcd image to 3.5.9-0. (https://github.com/kubernetes/kubernetes/pull/117999, @kkkkun) [SIG API Machinery]

• Updated cAdvisor to v0.47.2 and fixed metrics in cri-o when a container restarts. (https://github.com/kubernetes/kubernetes/pull/118774, @harche)

• Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 (https://github.com/kubernetes/kubernetes/pull/118541, @jeremyrickard) [SIG Testing]

• Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.4 (https://github.com/kubernetes/kubernetes/pull/117746, @xmudrii) [SIG Testing]

• Updated the scheduler interface and cache methods to use contextual logging. (https://github.com/kubernetes/kubernetes/pull/116849, @mengjiao-liu)

• ValidatingAdmissionPolicy type checking now correctly handles authorizer variable. (https://github.com/kubernetes/kubernetes/pull/118540, @jiahuif) [SIG API Machinery]

• When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. (https://github.com/kubernetes/kubernetes/pull/118817, @pohly)

• With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating the need to specify the cgroupDriver configuration option (or --cgroup-driver flag) of kubelet. (https://github.com/kubernetes/kubernetes/pull/118770, @marquiz)

• --version=v1.X.Y... can now be used to set the prerelease and buildID portions of the version reported by components (https://github.com/kubernetes/kubernetes/pull/117688, @liggitt) [SIG API Machinery, Architecture and Release]

• RetroactiveDefaultStorageClass feature made stable and enabled by default. (https://github.com/kubernetes/kubernetes/pull/118102, @RomanBednar)

• TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. (https://github.com/kubernetes/kubernetes/pull/118816, @PiotrProkop)

• force_delete_pods_total and force_delete_pod_errors_total metrics count all pod deletion behaviors. (https://github.com/kubernetes/kubernetes/pull/118480, @carlory)

• klog text output now uses JSON as encoding for structs, maps and slices. (https://github.com/kubernetes/kubernetes/pull/117687, @pohly)

• kubeadm: added a new "kubeadm config validate" command that can be used to validate any input config file. Use the --config flag to pass a config file to it. See the command --help screen for more information. As a result of adding this new command, enhance the validation capabilities of the existing "kubeadm config migrate" command. For both commands unknown APIs or fields will throw errors. (https://github.com/kubernetes/kubernetes/pull/118013, @neolit123)

• kubeadm: added the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP/experimental APIs in the future. (https://github.com/kubernetes/kubernetes/pull/118866, @neolit123)

• kubeadm: generate CA certificates with a start time that is offset 5 minutes in the past relative to the current system time to workaround cases of clock desync. (https://github.com/kubernetes/kubernetes/pull/118922, @champtar)

• plugin_evaluation_total metric supports prescore/score extension point. The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming pod. (https://github.com/kubernetes/kubernetes/pull/118025, @AxeZhan)

[jaskaransarkaria]

API Changes

Notable changes

• ACTION_REQUIRED When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. (https://github.com/kubernetes/kubernetes/pull/118420, @alculquicondor) [SIG Apps]
• Added IP mode field to load balancer status ingress. (https://github.com/kubernetes/kubernetes/pull/118895, @RyanAoh)
• Added new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (https://github.com/kubernetes/kubernetes/pull/118137, @helayoty)
• Graduated AdmissionWebhookMatchCondition feature to beta. (https://github.com/kubernetes/kubernetes/pull/119380, @a-hilaly)
• If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via memory.oom.group . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. (https://github.com/kubernetes/kubernetes/pull/117793, @tzneal) [SIG Apps, Node and Testing]
• Indexed Job pods now have the pod completion index set as a pod label. (https://github.com/kubernetes/kubernetes/pull/118883, @danielvegamyhre) [SIG Apps]
• NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. (https://github.com/kubernetes/kubernetes/pull/115398, @tangwz) [SIG Scheduling]
• PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. (https://github.com/kubernetes/kubernetes/pull/116469, @RomanBednar)
• Pods which set hostNetwork: true and declare ports, get the hostPort field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now hostPort will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. (https://github.com/kubernetes/kubernetes/pull/117696, @thockin) [SIG Apps]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (https://github.com/kubernetes/kubernetes/pull/119232, @danielvegamyhre) [SIG Apps]
• • Supported BackoffLimitPerIndex in Jobs. (https://github.com/kubernetes/kubernetes/pull/118009, @mimowo)
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer creates the KUBE-MARK-DROP chain (which has been unused for several releases) or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (https://github.com/kubernetes/kubernetes/pull/119374, @danwinship)
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (https://github.com/kubernetes/kubernetes/pull/116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Updated the comment about the feature-gate level for PodFailurePolicy from alpha to beta (https://github.com/kubernetes/kubernetes/pull/118278, @mimowo)
• kube-controller-manager: The LegacyServiceAccountTokenCleanUp feature gate is now available as alpha (off by default). When enabled, the legacy-service-account-token-cleaner controller loop removes service account token secrets that have not been used in the time specified by --legacy-service-account-token-clean-up-period (defaulting to one year), and are referenced from the .secrets list of a ServiceAccount object, and are not referenced from pods. (https://github.com/kubernetes/kubernetes/pull/115554, @yt2985)
• kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (https://github.com/kubernetes/kubernetes/pull/117649, @SataQiu)
• Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta. (https://github.com/kubernetes/kubernetes/pull/117802, @kerthcet) [SIG API Machinery and Apps]

Other changes

• A CDIDevice field is included in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (https://github.com/kubernetes/kubernetes/pull/118254, @elezar) [SIG Node and Testing]

• Added ServedVersions field to StorageVersion API. (https://github.com/kubernetes/kubernetes/pull/118386, @Richabanker)

• Added podReplacementPolicy and terminating field to job api. (https://github.com/kubernetes/kubernetes/pull/119301, @kannon92)

• Added a new namespaceParamRef field to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy. (https://github.com/kubernetes/kubernetes/pull/119215, @alexzielenski) [SIG API Machinery and Testing]

• Added a warning that TLS 1.3 ciphers are not configurable. (https://github.com/kubernetes/kubernetes/pull/115399, @3u13r) [SIG API Machinery and Node]

• Added error handling for seccomp localhost configurations that do not properly set a localhostProfile. (https://github.com/kubernetes/kubernetes/pull/117020, @cji)

• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. (https://github.com/kubernetes/kubernetes/pull/118041, @cici37) [SIG API Machinery]

• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject variable with expressions. (https://github.com/kubernetes/kubernetes/pull/118267, @cici37) [SIG API Machinery and Testing]

• Added new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (https://github.com/kubernetes/kubernetes/pull/118990, @alexzielenski)

• Added new config option delayCacheUntilActive to KubeSchedulerConfiguration that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in kube-scheduler (https://github.com/kubernetes/kubernetes/pull/115754, @linxiulei) [SIG API Machinery and Scheduling]

• Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the KMSv2KDF feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. (https://github.com/kubernetes/kubernetes/pull/118828, @enj)

• Exposed rest.DefaultServerUrlFor function. (https://github.com/kubernetes/kubernetes/pull/118055, @timofurrer)

• Extended the Job API for alpha version of BackoffLimitPerIndex. (https://github.com/kubernetes/kubernetes/pull/119294, @mimowo)

• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (https://github.com/kubernetes/kubernetes/pull/118782, @MikeSpreitzer) [SIG API Machinery]

• Kube-proxy: added --logging-format flag to support structured logging. (https://github.com/kubernetes/kubernetes/pull/117800, @cyclinder)

• Promoted API groups ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding to v1beta1. (https://github.com/kubernetes/kubernetes/pull/118644, @alexzielenski) [SIG API Machinery, Apps and Testing]

• Promoted the feature gate ValidtaingAdmissionPolicy to beta, and it is turned off by default. (https://github.com/kubernetes/kubernetes/pull/119409, @alexzielenski)

• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (https://github.com/kubernetes/kubernetes/pull/119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]

• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus. (https://github.com/kubernetes/kubernetes/pull/116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]

• Removed WindowsHostProcessContainers feature-gate. (https://github.com/kubernetes/kubernetes/pull/117570, @marosset) [SIG API Machinery, Apps, Auth, Node and Windows]

• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver (https://github.com/kubernetes/kubernetes/pull/117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]

• The SelfSubjectReview API is promoted to authentication.k8s.io/v1 and the kubectl auth whoami command is GA. (https://github.com/kubernetes/kubernetes/pull/117713, @nabokihms) [SIG API Machinery, Architecture, Auth, CLI and Testing]

• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still -, but a random suffix will avoid name collisions. (https://github.com/kubernetes/kubernetes/pull/117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

• client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently. (https://github.com/kubernetes/kubernetes/pull/113362, @sxllwx)

• component-base/logs is now stricter about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. (https://github.com/kubernetes/kubernetes/pull/117108, @pohly)

[jaskaransarkaria]

kubent output

→  kubent --target-version 1.28
11:01AM INF >>> Kube No Trouble `kubent` <<<
11:01AM INF version 0.7.2 (git sha 25eb8a3757d1db39a04e94bb97a3f099fb5c9fb6)
11:01AM INF Initializing collectors and retrieving data
11:01AM INF Target K8s version is 1.28.0
11:01AM INF Retrieved 5643 resources from collector name=Cluster
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for autoscaling/v2beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for autoscaling/v2beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:01AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:09AM WRN failed to discover supported resources for policy/v1beta1: the server could not find the requested resource
11:12AM WRN failed to discover supported resources for networking.k8s.io/v1beta1: the server could not find the requested resource
11:12AM WRN failed to discover supported resources for networking.k8s.io/v1beta1: the server could not find the requested resource
11:21AM WRN failed to discover supported resources for batch/v1beta1: the server could not find the requested resource
11:30AM INF Retrieved 9890 resources from collector name="Helm v3"
11:30AM INF Loaded ruleset name=custom.rego.tmpl
11:30AM INF Loaded ruleset name=deprecated-1-16.rego
11:30AM INF Loaded ruleset name=deprecated-1-22.rego
11:30AM INF Loaded ruleset name=deprecated-1-25.rego
11:30AM INF Loaded ruleset name=deprecated-1-26.rego
11:30AM INF Loaded ruleset name=deprecated-1-27.rego
11:30AM INF Loaded ruleset name=deprecated-1-29.rego
11:30AM INF Loaded ruleset name=deprecated-future.rego
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.22 <<<
------------------------------------------------------------------------------------------
KIND      NAMESPACE                       NAME                                        API_VERSION                 REPLACE_WITH (SINCE)
Ingress   <undefined>                     hmpps-interventions-onboarding              networking.k8s.io/v1beta1   networking.k8s.io/v1 (1.19.0)
Ingress   <undefined>                     hmpps-delius-interventions-event-listener   networking.k8s.io/v1beta1   networking.k8s.io/v1 (1.19.0)
Ingress   polygraph-offender-management   poms-ingress                                networking.k8s.io/v1beta1   networking.k8s.io/v1 (1.19.0)
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.25 <<<
------------------------------------------------------------------------------------------
KIND                      NAMESPACE                     NAME                                                              API_VERSION           REPLACE_WITH (SINCE)
PodDisruptionBudget       <undefined>                   hmpps-community-accommodation-wiremock                            policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   create-and-vary-a-licence-wiremock-hmpps-community-api-wiremock   policy/v1beta1        policy/v1 (1.21.0)
HorizontalPodAutoscaler   <undefined>                   court-list-splitter                                               autoscaling/v2beta1   autoscaling/v2 (1.23.0)
HorizontalPodAutoscaler   <undefined>                   court-hearing-event-receiver                                      autoscaling/v2beta1   autoscaling/v2 (1.23.0)
CronJob                   c100-application-production   c100-application-cronjob-production                               batch/v1beta1         batch/v1 (1.21.0)
CronJob                   c100-application-production   c100-application-cronjob-payments-production                      batch/v1beta1         batch/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   pre-sentence-service-gotenberg                                    policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   pre-sentence-service-wproofreader                                 policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   pre-sentence-service                                              policy/v1beta1        policy/v1 (1.21.0)
CronJob                   <undefined>                   dlq-transfer-cronjob                                              batch/v1beta1         batch/v1 (1.21.0)
PodDisruptionBudget       c100-application-production   c100-application-pdb-production                                   policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   court-list-splitter                                               policy/v1beta1        policy/v1 (1.21.0)
PodDisruptionBudget       <undefined>                   court-hearing-event-receiver                                      policy/v1beta1        policy/v1 (1.21.0)
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.26 <<<
------------------------------------------------------------------------------------------