[CONTINT-3950] Fix rollout blocked on canary failure

AliDatadog commented 3 months ago

What does this PR do?

This PR changes the way we count unready pods during the rollout. Instead of looking at an annotation that was created with the new replicaset, we look at the number of unready pods among the old pods.

Motivation

As part of https://github.com/DataDog/extendeddaemonset/pull/166, we enhanced the EDS to keep track of unhealthy pods before a deployment by adding an annotation on the new ERS. However, it does not cover the case of canary failures. The annotation is never updated which means that the number of unreadyPods could be 0.

Additional Notes

N/A

Describe your test plan

Deploy the Agent using the operator on a 15 nodes clusters with the arguments attached to the datadog-operator command:

    - -supportExtendedDaemonset=true
    - -edsMaxPodUnavailable=1%
    - -edsCanaryDuration=15m
    - -edsCanaryReplicas=12
    - -edsCanaryAutoPauseEnabled=true
    - -edsCanaryAutoPauseMaxRestarts=5
    - -edsCanaryAutoPauseMaxSlowStartDuration=3m
    - -edsCanaryAutoFailEnabled=true
    - -edsCanaryAutoFailMaxRestarts=0

Deploy an agent with an image that does not exist:

❯ k -n datadog-agent-operator get pods
NAME                                     READY   STATUS              RESTARTS   AGE
datadog-agent-j6864-2glnv                5/5     Running             0          34m
datadog-agent-j6864-bgkwl                5/5     Running             0          35m
datadog-agent-j6864-qxvd7                5/5     Running             0          34m
datadog-agent-j6864-xh48g                5/5     Running             0          35m
datadog-agent-qrvhh-7bnxb                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-8dfd8                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-cm2vh                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-dsppk                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-h7jtr                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-pq4bb                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-t99vg                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-tcscz                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-v9fxm                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-vkg7f                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-vmw8z                0/1     Init:ErrImagePull   0          45s
datadog-agent-qrvhh-zpkfv                0/1     Init:ErrImagePull   0          45s
datadog-cluster-agent-7fff98774d-7gbpz   1/1     Running             0          17m
datadog-operator-56d659b494-gzxwp        1/1     Running             0          2m46s
extendeddaemonset-5c7788c9cf-8tfn6       1/1     Running             0          28m

Once the canary pods are created, fail the canary manually with k eds canary fail datadog-agent -n datadog-agent-operator and make sure pods are being replaced:

❯ k -n datadog-agent-operator get pods
NAME                                     READY   STATUS                  RESTARTS   AGE
datadog-agent-j6864-2glnv                5/5     Running                 0          35m
datadog-agent-j6864-4lbl2                0/5     Init:0/3                0          0s
datadog-agent-j6864-bgkwl                5/5     Running                 0          36m
datadog-agent-j6864-qxvd7                5/5     Running                 0          34m
datadog-agent-j6864-wrtnv                0/5     Init:0/3                0          0s
datadog-agent-j6864-xh48g                5/5     Running                 0          35m
datadog-agent-j6864-xs5gc                0/5     Init:0/3                0          0s
datadog-agent-qrvhh-7bnxb                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-cm2vh                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-h7jtr                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-pq4bb                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-t99vg                0/1     Init:ErrImagePull       0          80s
datadog-agent-qrvhh-tcscz                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-v9fxm                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-vkg7f                0/1     Init:ImagePullBackOff   0          80s
datadog-agent-qrvhh-zpkfv                0/1     Init:ImagePullBackOff   0          80s
datadog-cluster-agent-7fff98774d-7gbpz   1/1     Running                 0          18m
datadog-operator-56d659b494-gzxwp        1/1     Running                 0          3m21s
extendeddaemonset-5c7788c9cf-8tfn6       1/1     Running                 0          28m

AliDatadog commented 2 months ago

/merge

dd-devflow[bot] commented 2 months ago

:x: MergeQueue

You are not allowed to use the merge queue towards main.

If you need support, contact us on Slack #devflow with those details!

DataDog / extendeddaemonset