Open zsedem opened 3 weeks ago
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"no Rollout with BlueGreen strategy specified, delegating to DataPlaneReconciler","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"reconciling DataPlane resource","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"validating DataPlane configuration","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"exposing DataPlane deployment admin API via headless service","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"exposing DataPlane deployment via service","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"ensuring mTLS certificate","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"checking readiness of DataPlane service","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"dataplane-ingress-test-gateway-jsmvn-dj2k5"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.555Z","logger":"controlplane","msg":"ensuring DataPlane has service addesses in status","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"dataplane-ingress-test-gateway-jsmvn-dj2k5"}
2024-08-26 11:16:05 {"level":"Level(-2)","ts":"2024-08-26T11:16:05.559Z","logger":"controlplane.deployment_builder","msg":"unexpected type processed for trace logging: string, this is a bug!","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48"}
2024-08-26 11:16:05 {"level":"debug","ts":"2024-08-26T11:16:05.600Z","logger":"controlplane","msg":"Resource modified","controller":"controlplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"ControlPlane","ControlPlane":{"name":"test-gateway-kg5f9","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-kg5f9","reconcileID":"85c2580e-96fa-469c-8a60-9f7b8040c33c","namespace":"test-gateway","name":"test-gateway-kg5f9","Deployment":"controlplane-test-gateway-kg5f9-5cht2"}
2024-08-26 11:16:05 {"level":"debug","ts":"2024-08-26T11:16:05.634Z","logger":"controlplane.deployment_builder","msg":"Resource modified","controller":"dataplane","controllerGroup":"gateway-operator.konghq.com","controllerKind":"DataPlane","DataPlane":{"name":"test-gateway-jsmvn","namespace":"test-gateway"},"namespace":"test-gateway","name":"test-gateway-jsmvn","reconcileID":"59389878-cbf3-410e-9806-59d919e28f48","namespace":"test-gateway","name":"test-gateway-jsmvn","Deployment":"dataplane-test-gateway-jsmvn-vwz4n"}
Turned on the trace level
unexpected type processed for trace logging: string, this is a bug!
Noticed this in the logs
So it turns out the root cause is probably the same as for #500 .
Our company uses mutating webhooks to enforce plenty of best practices in our cluster. Unfortunately if one of these webhooks are patching the deployment created by the Kong Gateway Operator, it will stuck in a loop, where the owner resource (controlplane or dataplane) cannot become ready.
The workaround is, that we create a podTemplateSpec, which is eventually not mutated by any mutating webhooks.
AFAIK generally a Kubernetes Operator like this, should only look at managed fields, when considering the diff.
kubectl get deployment .... -o yaml --show-managed-fields
Current Behavior
We tried to deploy Kong Gateway operator to our test kubernetes cluster, but I am unable to get the HttpRoutes working. Based on these logs, I assume the problem is, because the Gateway does not turn
Programmed
.Based on some reading in the codes, I found, that the possible reason for that is Dataplane and Controlplane also not becoming ready:
Both the controlplane and the dataplane pods are ready and I can also send request to the dataplane (and get 404, since no httproute is configured)
Expected Behavior
Gateway becomes ready and HttpRoute added to the kong gateway.
Steps To Reproduce
Not sure how to reproduce, because locally in KinD, the same configuration works, so I assume we have some environment related issue. I tried turning off the admissionwebhook, but no success. these are some of the logs, that the operator keeps emitting in this stuck state:
Any tips how to debug further, which resource should become ready first, the controlplane or the dataplane?
Operator Version
1.3.0
kubectl version
Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.3