emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.33k stars 682 forks source link

apiext: add crd ca bundle check to ready probe #5497

Closed LanceEa closed 7 months ago

LanceEa commented 7 months ago

Description

Although the apiext server has CA bundle it might not have been injected into the CRD. Unfortunatelly, there is no good Condition/Readiness check on the CRD to ensure it has been patched correctly.

This causes a race condition when using something like Helm because the apiext pod will say it is ready because it has a CA cert but the CA bundle might not have been picked up by the k8s api-extension server.

This adds an additional check to the Ready Probe to validate both that we have a CA Cert and in fact it matches the CA bundle in the CRD's. Since we are using the controller-runtime Manager client which caches this List this will be a low latency way to ensure the CRD's are patched and ready as well.

For example, the test now takes a little longer for the apiext replicas to come ready but we can be confident that when we try to apply the Listener and Mapping that both are ready.

2024/01/04 17:40:12 waiting for Emissary-ingress apiext deployment to be ready...
2024/01/04 17:40:14 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:16 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:18 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:20 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:22 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:24 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:26 AvailableReplicas == 3, ReadyReplicas == 3
2024/01/04 17:40:26 creating getambassador.io resources in cluster
2024/01/04 17:40:26 creating getambassadorio item-0 --> Listener:edge-stack-listener-8080
2024/01/04 17:40:26 creating getambassadorio item-1 --> Listener:edge-stack-listener-8443
2024/01/04 17:40:26 creating getambassadorio item-2 --> Mapping:quote-backend
2024/01/04 17:40:28 assert getambassadorio item: Listener-edge-stack-listener-8080
2024/01/04 17:40:33 assert getambassadorio item: Listener-edge-stack-listener-8443
2024/01/04 17:40:38 assert getambassadorio item: Mapping-quote-backend

If not previously due to the race condition, we would see a few retries on the Mapping due to CABundle not being available and unable to validate self-signed server certificate.

2024/01/04 17:40:12 waiting for Emissary-ingress apiext deployment to be ready...
2024/01/04 17:40:14 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:16 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:18 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:20 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:22 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:24 AvailableReplicas == 0, ReadyReplicas == 0
2024/01/04 17:40:26 AvailableReplicas == 3, ReadyReplicas == 3
2024/01/04 17:40:26 creating getambassador.io resources in cluster
2024/01/04 17:40:26 creating getambassadorio item-0 --> Listener:edge-stack-listener-8080
2024/01/04 17:40:26 creating getambassadorio item-1 --> Listener:edge-stack-listener-8443
2024/01/04 17:40:26 creating getambassadorio item-2 --> Mapping:quote-backend
2024/01/04 17:40:26 creating getambassadorio item-2 --> Mapping:quote-backend
2024/01/04 17:40:26 creating getambassadorio item-2 --> Mapping:quote-backend
2024/01/04 17:40:26 creating getambassadorio item-2 --> Mapping:quote-backend
2024/01/04 17:40:28 assert getambassadorio item: Listener-edge-stack-listener-8080
2024/01/04 17:40:33 assert getambassadorio item: Listener-edge-stack-listener-8443
2024/01/04 17:40:38 assert getambassadorio item: Mapping-quote-backend

Related Issues

N/A

Testing

No additional testing added, current e2e test coverage already exercises it.

Checklist