Migration and CLI improvements

[T-Kukawka]

After first migration of test cluster with a customer we have found few points that we can improve to make the migration more robust and stable. The acceptance lists consists of the points to improve.

Acceptance criteria:

• [x] @calvix migration-cli should scale down kyverno before the workloads migration such that the webhooks do not interfere with the workloads https://github.com/giantswarm/capi-migration-cli/pull/93/
• [x] @paurosello instead of cordoning nodes (that triggers nginx to become unhealthy), we should be adding tainting all nodes and draining them such that the health of services offered are still in check
• [x] rename k8s-node-cache into k8s-node-cache-app in the default apps not to hinder the helm app migration on new cluster
• [x] deploy migration ready Vault to all MCs
  • [x] garfish
  • [x] giraffe
  • [x] gaia
  • [x] akita
  • [x] alpaca
  • [x] anchovy
  • [x] anteater
  • [x] antelope
  • [x] argali
  • [x] eagle
  • [x] fox
  • [x] potato
  • [x] valkyrie
  • [x] viking
  • [x] viper
  • [x] visitor
• [x] improve servicepriority migration as it does not seem to be working properly on CAPA
• [x] @calvix disable healtchecks (ELB/ASG) for control-plane nodes in vintage during capi-migration-cli run - https://github.com/giantswarm/capi-migration-cli/pull/88
• [ ] avoid IRSA operator removing OIDC provider by checking if the aws-operator label exists, monitoring, kubeconfig
• [x] Migrate the service-account-issuer with the right order
• [x] @calvix Change API server DNS to point to the new loadbalancer instead of registering machines to the LB to avoid API downtime during master renewal, https://github.com/giantswarm/capi-migration-cli/pull/90
• [x] @calvix update vintage wildcard domain to point to capi wildcard to ensure ingress will work even after ELB recreation
• [x] Description of cluster
• [x] @AndiDog Remove tag kubernetes.io/cluster/%s from vintage node pool SG so that aws-load-balancer-controller uses the CAPA-created security group instead of failing because there are two with this tag => https://github.com/giantswarm/capi-migration-cli/pull/86
• [x] @bdehri https://github.com/giantswarm/adidas/issues/1237
• [x] @calvix for some reason externaldns, clusterautoscaler and loadbalancercontroller have permissions issues until restarted, I think it would be a good idea to restart them as soon as the 3 Vitnage CPs nodes are deleted to make sure they have proper credentials (with the DNS change we can cause a full ingress outage)

[paurosello]

AWS loadbalancer logs after migration

{"level":"info","ts":"2024-05-24T10:46:15Z","msg":"version","GitVersion":"v2.6.1","GitCommit":"5a5885bfbd3c7237ac190859fe0f20e96c95f1f2","BuildDate":"2023-09-11T17:49:28+0000"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"setup","msg":"adding health check for controller"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-v1-pod"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-v1-service"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-ingressclassparams"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-networking-v1-ingress"}
{"level":"info","ts":"2024-05-24T10:46:15Z","logger":"setup","msg":"starting podInfo repo"}
{"level":"info","ts":"2024-05-24T10:46:17Z","logger":"controller-runtime.webhook.webhooks","msg":"Starting webhook server"}
{"level":"info","ts":"2024-05-24T10:46:17Z","msg":"Starting server","kind":"health probe","addr":":61779"}
{"level":"info","ts":"2024-05-24T10:46:17Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":":8080"}
{"level":"info","ts":"2024-05-24T10:46:17Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-05-24T10:46:17Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443}
{"level":"info","ts":"2024-05-24T10:46:17Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
I0524 10:46:17.201464       1 leaderelection.go:248] attempting to acquire leader lease co/aws-load-balancer-controller-leader...
I0524 10:46:34.079416       1 leaderelection.go:258] successfully acquired lease co/aws-load-balancer-controller-leader
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc0004e0230"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc0004e02d0"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","source":"kind source: *v1beta1.TargetGroupBinding"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"kind source: *v1.Ingress"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc0004e0320"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc0004e0370"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"kind source: *v1beta1.IngressClassParams"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"ingress","source":"kind source: *v1.IngressClass"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting Controller","controller":"ingress"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","source":"kind source: *v1.Endpoints"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","source":"kind source: *v1.Node"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting Controller","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting EventSource","controller":"service","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting Controller","controller":"service"}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting workers","controller":"ingress","worker count":3}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting workers","controller":"service","worker count":3}
{"level":"info","ts":"2024-05-24T10:46:34Z","msg":"Starting workers","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","worker count":3}
{"level":"error","ts":"2024-05-24T10:47:27Z","msg":"Reconciler error","controller":"service","object":{"name":"ingress-nginx-app-controller","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"ingress-nginx-app-controller","reconcileID":"d9ffbbf5-c9b3-4b45-af0e-5e4cbb33554b","error":"WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements\n\tstatus code: 400, request id: db9f00d1-c504-44a3-95e5-e36b3c2b53ad"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"675187a8-b4d3-4cdf-891f-8792274e9411","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"a5006956-199e-41b6-982b-8c0a6f495a99","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"610f2cda-c2ad-454f-8a2c-c589213e07ce","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"5b59f6eb-c55c-4f9f-9576-b2a5e992ab43","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"36ae8cb1-08bb-4d1d-9a04-f43650cf0835","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"info","ts":"2024-05-24T10:48:03Z","logger":"controllers.service","msg":"successfully built model","model":"{\"id\":\"ingress-controllers/ingress-nginx-app-controller\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-ingressc-ingressn-8655d7b43d\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":80,\"toPort\":80,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]},{\"ipProtocol\":\"tcp\",\"fromPort\":443,\"toPort\":443,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"443\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":443,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/ingress-controllers/ingress-nginx-app-controller:443/status/targetGroupARN\"}}]}}]}},\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/ingress-controllers/ingress-nginx-app-controller:80/status/targetGroupARN\"}}]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-ingressc-ingressn-e9cdea2837\",\"type\":\"network\",\"scheme\":\"internet-facing\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-017285ebed91572e4\"},{\"subnetID\":\"subnet-0716684e036872b10\"},{\"subnetID\":\"subnet-0a1a77d56b9176708\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"},\"sg-0ab8888f4ee05deed\"],\"loadBalancerAttributes\":[{\"key\":\"load_balancing.cross_zone.enabled\",\"value\":\"true\"}]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"ingress-controllers/ingress-nginx-app-controller:443\":{\"spec\":{\"name\":\"k8s-ingressc-ingressn-96020ef4c5\",\"targetType\":\"instance\",\"port\":30443,\"protocol\":\"TCP\",\"ipAddressType\":\"ipv4\",\"healthCheckConfig\":{\"port\":30080,\"protocol\":\"HTTP\",\"path\":\"/healthz\",\"matcher\":{\"httpCode\":\"200-399\"},\"intervalSeconds\":6,\"timeoutSeconds\":6,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2},\"targetGroupAttributes\":[{\"key\":\"deregistration_delay.connection_termination.enabled\",\"value\":\"true\"},{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"true\"}]}},\"ingress-controllers/ingress-nginx-app-controller:80\":{\"spec\":{\"name\":\"k8s-ingressc-ingressn-1994e2eafe\",\"targetType\":\"instance\",\"port\":30080,\"protocol\":\"TCP\",\"ipAddressType\":\"ipv4\",\"healthCheckConfig\":{\"port\":30080,\"protocol\":\"HTTP\",\"path\":\"/healthz\",\"matcher\":{\"httpCode\":\"200-399\"},\"intervalSeconds\":6,\"timeoutSeconds\":6,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2},\"targetGroupAttributes\":[{\"key\":\"deregistration_delay.connection_termination.enabled\",\"value\":\"true\"},{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"true\"}]}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"ingress-controllers/ingress-nginx-app-controller:443\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-ingressc-ingressn-96020ef4c5\",\"namespace\":\"ingress-controllers\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/ingress-controllers/ingress-nginx-app-controller:443/status/targetGroupARN\"},\"targetType\":\"instance\",\"serviceRef\":{\"name\":\"ingress-nginx-app-controller\",\"port\":443},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":\"sg-0ab8888f4ee05deed\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":30443},{\"protocol\":\"TCP\",\"port\":30080}]}]},\"ipAddressType\":\"ipv4\"}}}},\"ingress-controllers/ingress-nginx-app-controller:80\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-ingressc-ingressn-1994e2eafe\",\"namespace\":\"ingress-controllers\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/ingress-controllers/ingress-nginx-app-controller:80/status/targetGroupARN\"},\"targetType\":\"instance\",\"serviceRef\":{\"name\":\"ingress-nginx-app-controller\",\"port\":80},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":\"sg-0ab8888f4ee05deed\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":30080}]}]},\"ipAddressType\":\"ipv4\"}}}}}}}"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"3d8f4bda-6d2b-444f-a36a-c1fef8bcef3b","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"bdf2cd49-8583-48b3-87ae-e26a84bb861d","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-034f63741246d044f, got: [sg-00086b7c0f688b4a0 sg-011dc1ebd5b160dd3] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"f67b2452-5521-48d7-baf7-9b65850ce5d6","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"ba028252-f4c0-4f90-ba2b-02d0c9e5a2af","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"7d9c7e0a-d26b-4ac6-a88d-ed67d4d8f9cc","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"c13fdea3-b845-47ed-9876-05774e6448c4","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"e7d9208a-99ff-4b45-ad3b-230288cfe926","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"040f9319-21f4-4bda-a53d-d998f850216c","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:03Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"8e17ef8c-c8c3-43d0-9de5-b7e58dc378d7","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:04Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"ea87021f-f3d9-4795-8239-87d8f8cb7a8e","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:04Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"8afb66ca-aae5-4853-b3b0-00bd31e67ad3","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:04Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"ab86639c-5d11-489c-a8cb-4551d6088943","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:04Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"e18db7a3-23a5-44e5-b3ee-579d1cd9b197","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"info","ts":"2024-05-24T10:48:05Z","logger":"controllers.service","msg":"successfully deployed model","service":{"namespace":"ingress-controllers","name":"ingress-nginx-app-controller"}}
{"level":"error","ts":"2024-05-24T10:48:06Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"a8d0a089-17b3-4c4a-882f-06376f275c3f","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-034f63741246d044f, got: [sg-00086b7c0f688b4a0 sg-011dc1ebd5b160dd3] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:06Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"533577a0-0a64-4fc5-a103-a31ddcf5f92f","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-034f63741246d044f, got: [sg-00086b7c0f688b4a0 sg-011dc1ebd5b160dd3] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:08Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"7673f9db-c9f1-4177-b8e2-51de7eaa76c4","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:08Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"0c1d4482-bfd5-4e8b-b378-92b910fb1380","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-034f63741246d044f, got: [sg-00086b7c0f688b4a0 sg-011dc1ebd5b160dd3] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:13Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"8ae2d5cf-357a-4f4d-a9e0-a12b81054293","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:13Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"b4051f59-2b42-4be4-b195-b3a7054ee6e1","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-034f63741246d044f, got: [sg-00086b7c0f688b4a0 sg-011dc1ebd5b160dd3] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:24Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"f4a5394c-40ea-414a-88a2-d85fe5afa19d","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:24Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"c3831ff5-01ee-42f7-8a4b-3d868ee7d66a","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:44Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"1651890f-4e47-4593-8305-f0cad108abc2","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:44Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"543e15b2-dc06-4f6f-be0d-f2f2b12d591f","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:55Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"6d04345f-f495-44c8-9e6a-4b1a9a24ae0a","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:55Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"c271a748-7be3-42fe-a498-b2a6cdaa1895","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-034f63741246d044f, got: [sg-00086b7c0f688b4a0 sg-011dc1ebd5b160dd3] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:56Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"4ce453f2-1b5d-4b78-8e17-d0fad77acd52","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:48:56Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"0cf789b3-3fff-4ca6-8b78-a7f06bfa08ec","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:49:25Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-96020ef4c5","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-96020ef4c5","reconcileID":"5ad26b27-0cbe-4c76-b119-a11882c79251","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0dd3bb90d831a3d7b, got: [sg-011dc1ebd5b160dd3 sg-09ae341cf1878a6b6] (clusterName: vtc02)"}
{"level":"error","ts":"2024-05-24T10:49:25Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-ingressc-ingressn-1994e2eafe","namespace":"ingress-controllers"},"namespace":"ingress-controllers","name":"k8s-ingressc-ingressn-1994e2eafe","reconcileID":"8816a865-151c-41eb-a3cf-0485c333af1f","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/vtc02 for eni eni-0aebb5a58e829066b, got: [sg-011dc1ebd5b160dd3 sg-0688876b9ee634257] (clusterName: vtc02)"}

sg-00086b7c0f688b4a0

sg-0688876b9ee634257

cannot delete the old SG

[paurosello]

CoreDNS

17.94.100.in-addr.arpa.:1053
18.94.100.in-addr.arpa.:1053
19.94.100.in-addr.arpa.:1053
192.18.172.in-addr.arpa.:1053
193.18.172.in-addr.arpa.:1053
194.18.172.in-addr.arpa.:1053
195.18.172.in-addr.arpa.:1053
196.18.172.in-addr.arpa.:1053
197.18.172.in-addr.arpa.:1053
198.18.172.in-addr.arpa.:1053
199.18.172.in-addr.arpa.:1053
2.94.100.in-addr.arpa.:1053
20.94.100.in-addr.arpa.:1053
200.18.172.in-addr.arpa.:1053
201.18.172.in-addr.arpa.:1053
202.18.172.in-addr.arpa.:1053
203.18.172.in-addr.arpa.:1053
204.18.172.in-addr.arpa.:1053
205.18.172.in-addr.arpa.:1053
206.18.172.in-addr.arpa.:1053
207.18.172.in-addr.arpa.:1053
21.94.100.in-addr.arpa.:1053
22.94.100.in-addr.arpa.:1053
23.94.100.in-addr.arpa.:1053
24.94.100.in-addr.arpa.:1053
25.94.100.in-addr.arpa.:1053
26.94.100.in-addr.arpa.:1053
27.94.100.in-addr.arpa.:1053
28.94.100.in-addr.arpa.:1053
29.94.100.in-addr.arpa.:1053
3.94.100.in-addr.arpa.:1053
30.94.100.in-addr.arpa.:1053
31.94.100.in-addr.arpa.:1053
32.94.100.in-addr.arpa.:1053
33.94.100.in-addr.arpa.:1053
34.94.100.in-addr.arpa.:1053
35.94.100.in-addr.arpa.:1053
36.94.100.in-addr.arpa.:1053
37.94.100.in-addr.arpa.:1053
38.94.100.in-addr.arpa.:1053
39.94.100.in-addr.arpa.:1053
4.94.100.in-addr.arpa.:1053
40.94.100.in-addr.arpa.:1053
41.94.100.in-addr.arpa.:1053
42.94.100.in-addr.arpa.:1053
43.94.100.in-addr.arpa.:1053
44.94.100.in-addr.arpa.:1053
45.94.100.in-addr.arpa.:1053
46.94.100.in-addr.arpa.:1053
47.94.100.in-addr.arpa.:1053
48.94.100.in-addr.arpa.:1053
49.94.100.in-addr.arpa.:1053
5.94.100.in-addr.arpa.:1053
50.94.100.in-addr.arpa.:1053
51.94.100.in-addr.arpa.:1053
52.94.100.in-addr.arpa.:1053
53.94.100.in-addr.arpa.:1053
54.94.100.in-addr.arpa.:1053
55.94.100.in-addr.arpa.:1053
56.94.100.in-addr.arpa.:1053
57.94.100.in-addr.arpa.:1053
58.94.100.in-addr.arpa.:1053
59.94.100.in-addr.arpa.:1053
6.94.100.in-addr.arpa.:1053
60.94.100.in-addr.arpa.:1053
61.94.100.in-addr.arpa.:1053
62.94.100.in-addr.arpa.:1053
63.94.100.in-addr.arpa.:1053
7.94.100.in-addr.arpa.:1053
8.94.100.in-addr.arpa.:1053
9.94.100.in-addr.arpa.:1053
cluster.local.:1053
[INFO] plugin/reload: Running configuration SHA512 = 7a6b23efa2a65c455cfb2e4fc0c139512aa5498da1aa16636f7a8d22a6a05488ee1511f9e21b586e82d3d7d7f68753b91837c4e1254737fa4a4c3eb17937095b
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Service: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Pod: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Namespace: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Namespace: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Service: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Pod: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Pod: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Namespace: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Service: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized

same thing for some kyverno admission controller pods

kyverno-pre E0524 11:47:07.896395       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:47:50.376784       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:47:50.376819       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:48:32.261272       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:48:32.261301       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:49:31.386000       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:49:31.386035       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:50:05.245282       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:50:05.245311       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:50:59.011596       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:50:59.011633       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:51:54.870924       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:51:54.870955       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:52:49.754870       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:52:49.754903       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:53:42.776356       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:53:42.776388       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:54:25.184721       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:54:25.184750       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:55:23.228305       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:55:23.228349       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:56:06.867378       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:56:06.867411       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:56:59.502428       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:56:59.502463       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:57:41.987123       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:57:41.987155       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
stream logs failed container "kyverno" in pod "kyverno-admission-controller-6b4994fc4f-h244n" is waiting to start: PodInitializing for kyverno/kyverno-admission-controller-6b4994fc4f-h244n (kyverno)
kyverno-pre W0524 11:58:29.569991       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:58:29.570023       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:59:01.919636       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:59:01.919669       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 11:59:37.779532       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 11:59:37.779566       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:00:08.669101       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:00:08.669131       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:00:46.474455       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:00:46.474487       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:01:40.147667       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:01:40.147723       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:02:19.710623       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:02:19.710653       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:03:14.265710       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:03:14.265742       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:04:13.701066       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:04:13.701095       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:05:01.799804       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:05:01.799837       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:05:55.578880       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:05:55.578913       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:06:29.899515       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:06:29.899555       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:07:08.529351       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:07:08.529388       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:07:40.014966       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:07:40.014997       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:08:21.357859       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:08:21.357896       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:09:11.028734       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:09:11.028761       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:10:05.222584       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:10:05.222618       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:10:53.738436       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:10:53.738476       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:11:40.899510       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:11:40.899543       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:12:18.415435       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:12:18.415469       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:13:03.355727       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:13:03.355774       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:13:42.015307       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:13:42.015338       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:14:22.238374       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:14:22.238406       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:15:03.070039       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:15:03.070073       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:15:59.416909       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:15:59.416942       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:16:37.016919       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:16:37.016953       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:17:28.379942       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:17:28.379972       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:18:20.111206       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:18:20.111237       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:18:53.903214       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:18:53.903256       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:19:38.914466       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:19:38.914495       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:20:36.511378       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:20:36.511407       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:21:24.888821       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:21:24.888851       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:22:10.218025       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:22:10.218057       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:22:55.051532       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:22:55.051569       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:23:37.443717       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:23:37.443759       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre W0524 12:24:08.424011       1 reflector.go:539] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Unauthorized
kyverno-pre E0524 12:24:08.424057       1 reflector.go:147] k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Unauthorized
stream logs failed container "kyverno" in pod "kyverno-admission-controller-6b4994fc4f-h244n" is waiting to start: PodInitializing for kyverno/kyverno-admission-controller-6b4994fc4f-h244n (kyverno)
stream logs failed container "kyverno" in pod "kyverno-admission-controller-6b4994fc4f-h244n" is waiting to start: PodInitializing for kyverno/kyverno-admission-controller-6b4994fc4f-h244n (kyverno)

This is what I can see on the API server

E0524 12:25:39.778260       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:25:42.083913       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:42.083943       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:42.445594       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:42.445696       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:42.585397       1 watcher.go:229] watch chan error: etcdserver: mvcc: required revision has been compacted
W0524 12:25:43.086167       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:43.086251       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:44.371236       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:25:44.779665       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:25:44.823061       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:44.823085       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:49.332371       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:49.332401       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:49.371122       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:25:49.778816       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:25:50.704473       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:50.704502       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:51.793162       1 dispatcher.go:188] Failed calling webhook, failing open vpa.k8s.io: failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
E0524 12:25:51.793197       1 dispatcher.go:192] failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
W0524 12:25:51.802191       1 dispatcher.go:188] Failed calling webhook, failing open vpa.k8s.io: failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
E0524 12:25:51.802220       1 dispatcher.go:192] failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
W0524 12:25:51.804299       1 dispatcher.go:188] Failed calling webhook, failing open vpa.k8s.io: failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
E0524 12:25:51.804329       1 dispatcher.go:192] failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
W0524 12:25:51.859453       1 dispatcher.go:182] Failed calling webhook, failing open namespace-admission-controller.adidas.com: failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:25:51.859505       1 dispatcher.go:190] failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
W0524 12:25:51.870099       1 dispatcher.go:188] Failed calling webhook, failing open vpa.k8s.io: failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
E0524 12:25:51.870190       1 dispatcher.go:192] failed calling webhook "vpa.k8s.io": failed to call webhook: Post "https://vpa-webhook.kube-system.svc:443/?timeout=5s": dial tcp 172.18.200.68:443: connect: connection refused
W0524 12:25:53.284814       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:53.284842       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:53.624454       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:53.624491       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:54.371305       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:25:54.778675       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:25:57.827958       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:57.828009       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:58.445930       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:58.445967       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:59.076156       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:59.076191       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:59.089969       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:59.090026       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:25:59.104137       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:59.104227       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:25:59.372574       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:25:59.781612       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:03.151837       1 dispatcher.go:182] Failed calling webhook, failing open namespace-admission-controller.adidas.com: failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:03.151870       1 dispatcher.go:190] failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:04.376719       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:04.783589       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:07.365438       1 dispatcher.go:182] Failed calling webhook, failing open namespace-admission-controller.adidas.com: failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:07.365473       1 dispatcher.go:190] failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
W0524 12:26:08.211728       1 dispatcher.go:182] Failed calling webhook, failing open namespace-admission-controller.adidas.com: failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:08.211756       1 dispatcher.go:190] failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
W0524 12:26:09.107570       1 dispatcher.go:182] Failed calling webhook, failing open namespace-admission-controller.adidas.com: failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:09.107599       1 dispatcher.go:190] failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:09.371136       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:09.781962       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:09.971037       1 dispatcher.go:182] Failed calling webhook, failing open namespace-admission-controller.adidas.com: failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
E0524 12:26:09.971066       1 dispatcher.go:190] failed calling webhook "namespace-admission-controller.adidas.com": failed to call webhook: Post "https://namespace-admission-controller.admission.svc:443/update?timeout=5s": dial tcp 172.18.194.111:443: connect: operation not permitted
W0524 12:26:11.048156       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:11.048184       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:26:13.360798       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:13.360823       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:26:13.522561       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:13.522586       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:26:13.968331       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:13.968356       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:14.371121       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:14.779159       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:16.008162       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:16.008193       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:26:16.412928       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:16.412960       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:26:17.310556       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:17.310583       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:19.371676       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:19.779116       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:21.533965       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:21.534015       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
W0524 12:26:21.870866       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:21.870894       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:24.371257       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:24.779337       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:29.371820       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:29.778788       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:32.505568       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:32.505602       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:34.371147       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:34.778598       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
W0524 12:26:37.936577       1 dispatcher.go:188] Failed calling webhook, failing open known-exploited-vulnr-admission-controller.adidas.com: failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:37.936607       1 dispatcher.go:192] failed calling webhook "known-exploited-vulnr-admission-controller.adidas.com": failed to call webhook: Post "https://known-exploited-vulnr-admission-controller.admission.svc:443/mutate?timeout=5s": dial tcp 172.18.201.14:443: connect: operation not permitted
E0524 12:26:39.371368       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:39.778410       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:44.370954       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:44.778663       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:49.371156       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:49.778785       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:54.370694       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:54.778749       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:59.370719       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:26:59.778946       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:27:04.371114       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:27:04.780109       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:27:09.371542       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"
E0524 12:27:09.779183       1 authentication.go:73] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=8999641150854164249, SKID=, AKID=8A:66:D4:0F:F5:72:33:3C:D4:5B:67:D9:4C:D3:49:42:B7:9F:B0:F2 failed: x509: certificate specifies an incompatible key usage]"

this happens in 2 of the 3 nodes that are on this cluster

I figured out 1 of the nodes did not have the proper service-account-issuer and the apiserver pod could not be seen from kubectl but was running inside the node. Terminated the node and:

![Uploading image.png…]()

[paurosello]

[calvix]

done