DaemonSet UpdatedNumberScheduled Status Not Right with RollingUpdate

[shihaohems]

What happened?

DaemonSet, with update strategy that will cause new Pod being created before old Pod terminated on a Node (surge > 0), does not count scheduled updated Pods in "UpdatedNumberScheduled" status until old Pod on the same Node is terminated.

What did you expect to happen?

With rolling update, "UpdatedNumberScheduled" should count an updated Pod once it is scheduled, instead of being blocked by old Pod on the same Node.

How can we reproduce it (as minimally and precisely as possible)?

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-daemonset
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["/bin/sh"]
        args: ["-c", "trap 'sleep 30' TERM; while true; do sleep 1; done"]
  updateStrategy:
    type: RollingUpdate                                                                                                                                                                                                 
    rollingUpdate:                                                                                                                                                                                               
      maxSurge: 1                                                                                                                                                                                        
      maxUnavailable: 0                                                                                                                                                                                   

• Create a DaemonSet with above spec.
• Update/restart the DaemonSet to trigger the rolling update.
• Check "UpdatedNumberScheduled" status of this DaemonSet after the new Pod is scheduled and before the old Pod is terminated.

Anything else we need to know?

The root cause should be status update only checks the oldest pod on each Node, which should not be the case with surge>0 rolling update?

Kubernetes version

1.28.2

Cloud provider

Docker Desktop

OS version

No response

Install tools

No response

Container runtime (CRI) and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[liangyuanpeng]

/sig apps

[Ithrael]

I think this seems to be normal logic.

This is the process I reproduced, I used minikube k8s server 1.27.4 with 3 nodes.

daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-daemonset
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["/bin/sh"]
        args: ["-c", "trap 'sleep 30' TERM; while true; do sleep 1; done"]

kubectl rollout restart ds my-daemonset

The observed situation is: Pods on the node are first deleted and then created. When the pod is in the 'terminating' state, 'UpdatedNumberScheduled' does not change. It only increases when the pod is in the 'creating' state.

Look the doc，https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/daemon-set-v1/

I believe this description is consistent with the actual logic being carried out

[shihaohems]

Hi @Ithrael, thanks for the reply. I missed "RollingUpdate as the updateStrategy" in the reproduce steps, sorry for that (editted it in).

Try the following and see the issue.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-daemonset
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["/bin/sh"]
        args: ["-c", "trap 'sleep 30' TERM; while true; do sleep 1; done"]
  updateStrategy:
    type: RollingUpdate                                                                                                                                                                                                 
    rollingUpdate:                                                                                                                                                                                               
      maxSurge: 1                                                                                                                                                                                        
      maxUnavailable: 0                                                                                                                                                                                   

Edit: To be more precise, I should say "with RollingUpdate strategy that will result in new Pod being created before old one terminated on a Node".

[Ithrael]

Hi @shihaohems , thanks for the reply.

I can reproduce this issue now.

    rollingUpdate:                                                                                                                                                                                               
      maxSurge: 1                                                                                                                                                                                        
      maxUnavailable: 0   

This configuration, during the rolling update process, will first create a new version of the Pod then delete the old version of the Pod. This will result in both new and old versions of the Pods coexist for a period of time. Due to the sort operation of sort.Sort(podByCreationTimestampAndPhase(daemonPods))line , the line of code pod := daemonPods[0]line will only check the oldest Pod. This means that only when the old version of Pod is deleted, will it begin to check the new version of the Pod.

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[atiratree]

/triage accepted /priority important-longterm /remove-lifecycle stale

[atiratree]

There are some considerations that should be taken into account when fixing this: https://github.com/kubernetes/kubernetes/pull/122587#discussion_r1565532534