Keep track of pods already unhealthy before deployment

What does this PR do?

This PR enhances the extended daemon set to keep track of pods that were already unhealthy before deployment.

Motivation

The EDS controller like the Daemonset controller is configured to limit the number of daemonset pod un-ready during a rollout. For example the rollout strategy can be not more than 5% of pod unready. But if before a rollout start 4% of the existing pods are not ready, it means that the controller can also update 1% at the same time (5%-4%= 1%) which slow down the deployment. and even in some case block the deployment (if more than 5% of the pods are unready before the rollout).

If the EDS could keep track of the number of pods that were initially unhealthy before deployment, it would be able to keep the rollout strategy unaffected (and thus not blocked or slowed-down) by the pods that were previously unhealthy.

Changes Made

The number of unhealthy pods is added to the extendeddaemonset replicaset upon creation as an annotation unready-pods.
When calculating the number of pods that can be deleted in limits.go, the number of initially unhealthy pods (which is stored in the annotations of the ERS) is used to compensate for this difference in the calculation.

Additional Notes

The image k8s.gcr.io/pause:latest was used in the integration tests. However this image doesn't work on arm architecture, this is why I changed it to k8s.gcr.io/pause:3.0.
In case the unready pods annotation doesn't exist for some reason, the reconciler will fallback to the default behaviour (i.e the rollout will be slowed down / blocked in case there were many pods unready before rollout)

Describe your test plan

Unit Testing:
- The function CalculatePodToCreateAndDelete in limits.go was unit-tested in limits_test.go to ensure that the pods that were already unhealthy are taken into consideration.
- We can run the unit tests by executing make test
E2E Testing:
- Integration testing was implemented
- Deploy a kind cluster with a few nodes
- Run the E2E test using make e2e
- The E2E test of this change includes the following:
  - Creating an EDS with a corrupted image
  - This results in all pods of the EDS being unhealthy
  - Edit the EDS by setting a valid container image in the pod template (e.g. k8s.gcr.io/pause:3.0)
  - Launch the deployment
  - Ensure that the rollout is not blocked and that the EDS becomes finally in the Running state, with 1 active ERS and all pods are healthy and running
QA Testing:
- Create a kind cluster with few nodes (at least 3)
- First create a configmap by running kubectl create configmap example-configmap
- Run make install to install the CRDs
- Run the controller (e.g by running make run, attention you should add LeaderElectionNamespace: "kube-system" to the created manager in main.go for this to work)
- Create an EDS with the following description:
```
      kind: ExtendedDaemonSet
      metadata:
           name: foo
      spec:
        strategy:
          canary:
            replicas: 1
            duration: 5m
            autoFail:
              enabled: true
              canaryTimeout: 10m
            autoPause:
              enabled: true
              maxRestarts: 5
          rollingUpdate:
            maxParallelPodCreation: 1
            maxUnavailable: 2
            slowStartIntervalDuration: 1m
        template:
          spec:
            containers:
              - name: daemon
                image: k8s.gcr.io/pause:3.0
                envFrom:
                  - configMapRef:
                      name: example-configmap
            tolerations:
              - operator: Exists
```
- When the pods are up and running, delete the configmap by running kubectl delete configmap example-configmap
- Delete some pods (at least 3)
- The ERS will try to recreate the pods, but the created pods will become unhealthy because the configmap is deleted
- Launch a deployment using the same definition of the EDS, but remove the envFrom property from the pod template
- Check the status of the deployment process by running kubectl get ers --watch. You should see 1 canary and the active replicaset
- After 5 minutes, the rollout should start and the deployment should not be blocked by the pods that are already unhealthy.
- You should end up with 1 active ERS having all pods healthy and running

Files Changed	Coverage Δ
...endeddaemonsetreplicaset/strategy/limits/limits.go	`57.14% <40.00%> (-7.57%)`	:arrow_down:
...endeddaemonsetreplicaset/strategy/rollingupdate.go	`78.07% <41.66%> (-2.50%)`	:arrow_down:
controllers/extendeddaemonset/controller.go	`75.53% <57.14%> (-0.44%)`	:arrow_down:

DataDog / extendeddaemonset