Add readiness probe to check webhook cert ready

[nishantapatil3]

Q	A
Bug fix?	no	yes
New feature?	yes (/readyz as readiness probe)
API breaks?	yes (/metrics as readiness probe)
Deprecations?	yes (/metrics as readiness probe)
Related tickets
License	Apache 2.0

What's in this PR?

Add readiness probe to mark cluster-registry as "ready" only after webhook configuration certificate is updated successfully

Why?

cluster-registry generates self signed certificate to inject into webhook which take approx ~30 sec after deploy.

If we are running an automation script that waits for cluster-registrty to be ready then /metrics end point marks container as ready even before caBundle in injected into ValidatingWebhookConfig.

As a result, the automation script would fail with

x509: certificate is not valid for any names, but wanted to match cluster-registry-controller.cluster-registry.svc

so we need to wait until the webhook is ready, then mark cluster-registry as "ready"

Additional context

This PR changes the readiness probe from /metrics to /readyz which would potentially break API that depends on readiness probe

Checklist

• [x] Implementation tested
• [x] User guide and development docs updated (if needed)

To Do

• [x] If the PR is not complete but you want to discuss the approach

Tests

• Webhook enabled (takes ~30 sec to be ready) (webhook ready health checker) Init "16:34:22" -> Ready "16:35:02" (40 seconds)

  status:
  conditions:
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:34:22Z"
  status: "True"
  type: Initialized
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:35:02Z"
  status: "True"
  type: Ready
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:35:02Z"
  status: "True"
  type: ContainersReady
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:34:22Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
  - containerID: docker://7c5063a148ea36dc0d7f4fa8451710ae34f304b1b4e05475e607d7b7bffacf20
  image: nishantapatil3/cluster-registry-controller:latest
  imageID: docker-pullable://nishantapatil3/cluster-registry-controller@sha256:1a672de686f0e0c2419be9194a2bb55cdd1cf0437ff2c72a840f3b70aafb2504
  lastState: {}
  name: manager
  ready: true
  restartCount: 0
  started: true
  state:
    running:
      startedAt: "2022-12-12T16:34:24Z"

• Webhook disabled (takes ~5 sec to be ready) (ping health checker) Init "16:29:06" -> Ready "16:29:10" (4 seconds)

  status:
  conditions:
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:29:06Z"
  status: "True"
  type: Initialized
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:29:10Z"
  status: "True"
  type: Ready
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:29:10Z"
  status: "True"
  type: ContainersReady
  - lastProbeTime: null
  lastTransitionTime: "2022-12-12T16:29:06Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
  - containerID: docker://b12445fe5a486db6227e1d9b2b76fb6dcacadff25c1ff96e5a6a62fd70a74c40
  image: nishantapatil3/cluster-registry-controller:latest
  imageID: docker-pullable://nishantapatil3/cluster-registry-controller@sha256:1a672de686f0e0c2419be9194a2bb55cdd1cf0437ff2c72a840f3b70aafb2504
  lastState: {}
  name: manager
  ready: true
  restartCount: 0
  started: true
  state:
    running:
      startedAt: "2022-12-12T16:29:08Z"

[nishantapatil3]

CC: @pregnor @waynz0r @Laci21 @tiswanso (I couldnt assign reviewers)

[nishantapatil3]

@pregnor thanks for the review.

Added manual test cases in PR summary to verify pod readiness after deploy using kubectl get pod -n cluster-registry cluster-registry-controller-controller-xxx -o yaml from status field

[nishantapatil3]

Thanks for the review guys, this is my first pull request so i am unable to merge.. please merge when possible, I'll create another PR to update charts and image versions

[nishantapatil3]

Thanks for the review and merge @Laci21 @pregnor @waynz0r