feat(probe): add startup probe

[wdhif]

What this PR does / why we need it:

Adds Kubernetes startup probe support for:

• Agent
• Cluster Agent
• Cluster Check Runner

Special notes for your reviewer:

• Deploy the Helm chart on a Kubernetes cluster, with the Cluster Check Runner active to validate the startup probe on it

  
  ➜  datadog-dev git:(main) ✗ helm install --set datadog.apiKey=$DD_API_KEY datadog-agent --set clusterChecksRunner.enabled=true ../helm-charts/charts/datadog
  NAME: datadog-agent
  LAST DEPLOYED: Wed Jun 12 14:03:51 2024
  NAMESPACE: default
  STATUS: deployed
  REVISION: 1
  TEST SUITE: None
  NOTES:
  Datadog agents are spinning up on each node in your cluster. After a few
  minutes, you should see your agents starting in your event stream:
  https://app.datadoghq.com/event/explorer

###################################################################################

WARNING: Cluster-Agent should be deployed in high availability mode

###################################################################################

The Cluster-Agent should be in high availability mode because the following features are enabled:

• Admission Controller

To run in high availability mode, our recommendation is to update the chart configuration with:

• set clusterAgent.replicas value to 2 replicas .

• set clusterAgent.createPodDisruptionBudget to true.

• Verify the resources

  ➜  datadog-dev git:(main) ✗ k get pod
  NAME                                          READY   STATUS    RESTARTS   AGE
  datadog-agent-cluster-agent-ccf87d958-ldbbs   1/1     Running   0          110s
  datadog-agent-clusterchecks-b6bc9b6bf-b9vcb   1/1     Running   0          110s
  datadog-agent-clusterchecks-b6bc9b6bf-bvjbp   1/1     Running   0          110s
  datadog-agent-g8n5r                           3/3     Running   0          110s

• Verify the Agent startup settings and curl the endpoint

  ➜  datadog-dev git:(main) ✗ k describe pods datadog-agent-g8n5r | grep startup
  Startup:        http-get http://:5555/startup delay=15s timeout=5s period=15s #success=1 #failure=6

  ➜  datadog-dev git:(main) ✗ kubectl exec -it daemonsets/datadog-agent -- curl http://localhost:5555/startup
  Defaulted container "agent" out of: agent, trace-agent, process-agent, init-volume (init), init-config (init)
  {"Healthy":["healthcheck","aggregator","tagger-store","tagger-workloadmeta","dogstatsd-main","collector-queue-20s","ad-config-provider-kubernetes-container-allinone","ad-servicelistening","workloadmeta-puller","workloadmeta-store","ad-kubeletlistener","collector-queue-15s","workloadmeta-docker","collector-queue-900s"],"Unhealthy":null}%

• Verify the Cluster Agent startup settings and curl the endpoint

  ➜  datadog-dev git:(main) ✗ k describe pods datadog-agent-cluster-agent-ccf87d958-ldbbs | grep startup
  Startup:        http-get http://:5556/startup delay=15s timeout=5s period=15s #success=1 #failure=6

  ➜  datadog-dev git:(main) ✗ k exec -it deployments/datadog-agent-cluster-agent -- curl http://localhost:5556/startup
  Defaulted container "cluster-agent" out of: cluster-agent, init-volume (init)
  {"Healthy":["healthcheck","ad-servicelistening","workloadmeta-puller","collector-queue-15s","clusterchecks-leadership","aggregator","workloadmeta-store","tagger-store","tagger-workloadmeta","ad-config-provider-kubernetes-endpoints","ad-config-provider-kubernetes-services","clusterchecks-dispatch"],"Unhealthy":null}

• Verify the Cluster Check Runner startup settings and curl the endpoint

  ➜  datadog-dev git:(main) ✗ k describe pods datadog-agent-clusterchecks-b6bc9b6bf-bvjbp | grep startup
  Startup:        http-get http://:5557/startup delay=15s timeout=5s period=15s #success=1 #failure=6

  ➜  datadog-dev git:(main) ✗ k exec -it deployments/datadog-agent-clusterchecks -- curl http://localhost:5557/startup
  Defaulted container "agent" out of: agent, init-volume (init), init-config (init)
  {"Healthy":["healthcheck","ad-config-provider-cluster-checks","ad-servicelistening","aggregator","workloadmeta-store","workloadmeta-puller"],"Unhealthy":null}

Note that currently the /startup endpoint is returning the default /health endpoint. Support for the dedicated /startup endpoint has been added in Agent 7.55. No components currently register a startup health check.

• With 7.55.0-rc.4

  ➜  datadog-dev git:(main) ✗ k get pod
  NAME                                           READY   STATUS    RESTARTS   AGE
  datadog-agent-cluster-agent-c97895b79-jlw84    1/1     Running   0          2m52s
  datadog-agent-clusterchecks-5df5488d59-4sm7l   1/1     Running   0          2m52s
  datadog-agent-clusterchecks-5df5488d59-c9d8d   1/1     Running   0          2m52s
  datadog-agent-mzdq8                            3/3     Running   0          2m52s

  ➜  datadog-dev git:(main) ✗ k exec -it daemonsets/datadog-agent -- curl http://localhost:5555/live
  Defaulted container "agent" out of: agent, trace-agent, process-agent, init-volume (init), init-config (init)
  {"Healthy":["healthcheck","workloadmeta-store","collector-queue-900s","tagger-workloadmeta","workloadmeta-docker","collector-queue-15s","collector-queue-60s","ad-servicelistening","workloadmeta-puller","ad-kubeletlistener","collector-queue-20s","aggregator","tagger-store","dogstatsd-main","ad-config-provider-kubernetes-container-allinone"],"Unhealthy":null}%
  ➜  datadog-dev git:(main) ✗ k exec -it daemonsets/datadog-agent -- curl http://localhost:5555/startup
  Defaulted container "agent" out of: agent, trace-agent, process-agent, init-volume (init), init-config (init)
  {"Healthy":null,"Unhealthy":null}%

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• [x] Chart Version bumped
• [x] Documentation has been updated with helm-docs (run: .github/helm-docs.sh)
• [x] CHANGELOG.md has been updated
• [x] Variables are documented in the README.md
• [ ] For Datadog Operator chart or value changes update the test baselines (run: make update-test-baselines)

[wdhif]

/merge

[dd-devflow[bot]]

:steam_locomotive: MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals. Note: if you pushed new commits since the last approval, you may need additional approval. You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

[dd-devflow[bot]]

:steam_locomotive: MergeQueue: pull request added to the queue

The median merge time in main is 38m.

Use /merge -c to cancel this operation!