Evaluate Kubernetes 1.31 changes

[lfrancke]

I would like to know if there are any changes in Kubernetes 1.31 that the SDP could potentially make use of (e.g. sidecars), will require changes or will break things on our end.

This should be timeboxed and take up at most 4h of research and reading. The result of this should be a comment on this issue or follow-up issues listing the high level points of things we could, should and should not do.

Must read:

• https://kubernetes.io/blog/2024/08/13/kubernetes-v1-31-release/

And the Kubernetes blog has dedicated blog post for certain topics: https://kubernetes.io/blog/

[razvan]

Elli

Link: https://kubernetes.io/blog/2024/08/13/kubernetes-v1-31-release/

Possible actions

• Consider if and where OCI image volumes could be useful for the SDP.
• Long shot: consider using pod failure policies for Spark applications.

Summary

• 45 total enhancements
  • 11 stable
  • 22 beta
  • 12 alpha

Highlights - Stable Features

AppArmor

• previously used labels but now a securityContext field is recommended.
• appArmorProfile.type in the container's securityContex

Better kube-proxy connectivity reliability

• ensures connections are drained before node exposed via load balancers are terminated

Persistent Volume last phase transision timestamp

• PVs now have a new status field status.lastTransitionTime
• allows to measure the time required to move a PV to new phase (Pending, Bound, Released).

Highlights - Beta Features

nftables back-end for kube-proxy

• successor of iptables mode
• faster packet routing in the linux kernel
• there is a migration guide from iptables

PV reclaim policy

• ensures PV reclaim policy is respected even after the associated PVC is deleted
• prevents PV leaks

Bound SA token improvement

• feature name ServiceAccountTokenNodeBinding
• request a token for a specific node instead of a pod

Multiple service CIDRs

• disabled by default
• prevents IP exhaustion for service objects in large clusters
• allows admins to dynamically modify service CIDRs without downtime

Traffic distribution for services

• new field trafficDistribution in Service objects
• provides guidelines for the implementations

VolumeAttributesClass ModifyVolume

• this API allows workload to vertically scale volumes on-line to balance cost and performance.

Highlights - Alpha Features

New DRA api

• dynamic resource allocation api enables new features like cluster autoscaling

Image volumes

• support OCI objects (images and artefacts) as native volume sources
• behind ImageVolume feature gate
• targets AI/ML workflows

Device health exposed via pod status

• container status now has allocatedResourcesStatus field
• reports health of devices assigned to the container

Finer grained authorization with selectors

• it is now possible for an authorizer to express: this user cannot list all pods, but can list all pods where .spec.nodeName matches some specific value
• allow a user to watch all Secrets in a namespace that are not labelled as confidential: true.

Restrictions on anonymous api access

• restrict enspoints that can be accessed with anonymous requests

Deprecations and Removals

• cgroup v1 in maintenance mode
• SHA-1 signature support must be explicitely enabled with GODEBUG=x509sha1=1
• .status.nodeInfo.kubeProxyVersion is deprecated
• all remaining in-tree cloud providers are removed
• all remaining in-tree provider feature gates are removed
• --keep-terminated-pod-volumes flag for the kubelet is new removed
• CephFS volume plugin removed
• Ceph RBD volume plugin removed
• Deprecation of non-CSI volume limit plugins in kube-scheduler

[razvan]

Read only volumes based on OCI artifacts

Link: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/

Summary

Allows users to focus on OCI standards as well as enables them to store and distribute any content using OCI registries.

Example:

…
kind: Pod
spec:
  containers:
    - …
      volumeMounts:
        - name: my-volume
          mountPath: /path/to/directory
  volumes:
    - name: my-volume
      image:
        reference: my-image:tag

Mounts the contents of my-image:tag at /path/to/directory in the container.

Use cases

• distribute configuration files
• distribute large datasets
• distribute binaries
• security engineers can distribute malware signatures.

See link above for more details.

[sbernauer]

LGTM, especially the Possible actions sections 👍