search

fluxcd / flux2

Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.
https://fluxcd.io
Apache License 2.0
6.48k stars 600 forks source link

AWS CNI Network policies broke Flux on EKS #4207

Closed albertschwarzkopf closed 1 year ago

albertschwarzkopf commented 1 year ago

Hi,

Amazon VPC CNI now supports Kubernetes Network Policies. I have enabled this in our DEV cluster, but the helm-controller, kustomize-controller and source-controller are crashing then (CrashLoopBackOff). Notification-controller is running.

│ {"level":"info","ts":"2023-08-30T14:25:11.878Z","logger":"setup","msg":"starting manager"}                                                                                                                                               │
│ {"level":"info","ts":"2023-08-30T14:25:11.879Z","msg":"Starting server","kind":"health probe","addr":":9440"}                                                                                                                            │
│ {"level":"info","ts":"2023-08-30T14:25:11.979Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/helm-controller-leader-election...\n"}                                                                          │
│ {"level":"info","ts":"2023-08-30T14:25:11.980Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                               │
│ {"level":"info","ts":"2023-08-30T14:25:11.992Z","logger":"runtime","msg":"successfully acquired lease flux-system/helm-controller-leader-election\n"}                                                                                    │
│ {"level":"info","ts":"2023-08-30T14:25:11.992Z","msg":"Starting EventSource","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","source":"kind source: *v2beta1.HelmRelease"}          │
│ {"level":"info","ts":"2023-08-30T14:25:11.992Z","msg":"Starting EventSource","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","source":"kind source: *v1beta2.HelmChart"}            │
│ {"level":"info","ts":"2023-08-30T14:25:11.992Z","msg":"Starting Controller","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease"}                                                        │
│ {"level":"info","ts":"2023-08-30T14:25:12.095Z","msg":"Starting workers","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","worker count":4}                                          │
│ {"level":"info","ts":"2023-08-30T14:25:12.101Z","msg":"HelmChart 'kube-system/kube-system-descheduler' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease": │
│ {"name":"descheduler","namespace":"kube-system"},"namespace":"kube-system","name":"descheduler","reconcileID":"a914f34a-13d0-4368-9a2a-ae59def50109"}                                                                                    │
│ {"level":"info","ts":"2023-08-30T14:25:12.102Z","msg":"HelmChart 'monitoring/monitoring-prometheus-operator' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRel │
│ ease":{"name":"prometheus-operator","namespace":"monitoring"},"namespace":"monitoring","name":"prometheus-operator","reconcileID":"6d551dc5-8deb-47ee-a5aa-e0f5d73ca52e"}                                                                │
│ {"level":"info","ts":"2023-08-30T14:25:12.102Z","msg":"HelmChart 'flux-system/kube-system-metrics-server' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmReleas │
│ e":{"name":"metrics-server","namespace":"kube-system"},"namespace":"kube-system","name":"metrics-server","reconcileID":"6d31c296-7ba9-4f8e-bcf7-1f308d8e2b5f"}                                                                           │
│ {"level":"info","ts":"2023-08-30T14:25:12.103Z","msg":"HelmChart 'kube-system/kube-system-node-problem-detector' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","Hel │
│ mRelease":{"name":"node-problem-detector","namespace":"kube-system"},"namespace":"kube-system","name":"node-problem-detector","reconcileID":"9d66b319-8580-425e-8240-4401d2d4f120"}                                                      │
│ {"level":"info","ts":"2023-08-30T14:25:12.163Z","msg":"reconcilation finished in 67.28591ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"nam │
│ e":"node-problem-detector","namespace":"kube-system"},"namespace":"kube-system","name":"node-problem-detector","reconcileID":"9d66b319-8580-425e-8240-4401d2d4f120"}                                                                     │
│ {"level":"info","ts":"2023-08-30T14:25:12.166Z","msg":"reconcilation finished in 70.625584ms, next run in 1h0m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"n │
│ ame":"descheduler","namespace":"kube-system"},"namespace":"kube-system","name":"descheduler","reconcileID":"a914f34a-13d0-4368-9a2a-ae59def50109"}                                                                                       │
│ {"level":"info","ts":"2023-08-30T14:25:12.167Z","msg":"reconcilation finished in 72.109918ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"na │
│ me":"metrics-server","namespace":"kube-system"},"namespace":"kube-system","name":"metrics-server","reconcileID":"6d31c296-7ba9-4f8e-bcf7-1f308d8e2b5f"}                                                                                  │
│ {"level":"info","ts":"2023-08-30T14:25:12.173Z","msg":"HelmChart 'opencost/opencost-opencost' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":" │
│ opencost","namespace":"opencost"},"namespace":"opencost","name":"opencost","reconcileID":"450071fe-4da4-4652-9e1d-01ed7fa495aa"}                                                                                                         │
│ {"level":"info","ts":"2023-08-30T14:25:12.174Z","msg":"HelmChart 'stakater-reloader/stakater-reloader-stakater-reloader' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelea │
│ se","HelmRelease":{"name":"stakater-reloader","namespace":"stakater-reloader"},"namespace":"stakater-reloader","name":"stakater-reloader","reconcileID":"31e3ca9c-cdbb-41d8-9f25-528f749aa6a6"}                                          │
│ {"level":"info","ts":"2023-08-30T14:25:12.201Z","msg":"reconcilation finished in 33.974647ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"na │
│ me":"stakater-reloader","namespace":"stakater-reloader"},"namespace":"stakater-reloader","name":"stakater-reloader","reconcileID":"31e3ca9c-cdbb-41d8-9f25-528f749aa6a6"}                                                                │
│ {"level":"info","ts":"2023-08-30T14:25:12.262Z","msg":"reconcilation finished in 97.897237ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"na │
│ me":"opencost","namespace":"opencost"},"namespace":"opencost","name":"opencost","reconcileID":"450071fe-4da4-4652-9e1d-01ed7fa495aa"}                                                                                                    │
│ {"level":"info","ts":"2023-08-30T14:25:12.264Z","msg":"reconcilation finished in 168.473958ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"n │
│ ame":"prometheus-operator","namespace":"monitoring"},"namespace":"monitoring","name":"prometheus-operator","reconcileID":"6d551dc5-8deb-47ee-a5aa-e0f5d73ca52e"}                                                                         │
│ {"level":"info","ts":"2023-08-30T14:25:12.266Z","msg":"HelmChart 'kube-system/kube-system-aws-ebs-csi-snapshot-controller' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRel │
│ ease","HelmRelease":{"name":"aws-ebs-csi-snapshot-controller","namespace":"kube-system"},"namespace":"kube-system","name":"aws-ebs-csi-snapshot-controller","reconcileID":"1bd81ebb-9c13-4439-bec1-2597c99d3ec8"}                        │
│ {"level":"info","ts":"2023-08-30T14:25:12.268Z","msg":"HelmChart 'kube-system/kube-system-vertical-pod-autoscaler' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","H │
│ elmRelease":{"name":"vertical-pod-autoscaler","namespace":"kube-system"},"namespace":"kube-system","name":"vertical-pod-autoscaler","reconcileID":"48849661-6822-434d-bbcc-95e78a7ffb21"}                                                │
│ {"level":"info","ts":"2023-08-30T14:25:12.295Z","msg":"reconcilation finished in 93.094191ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"na │
│ me":"aws-ebs-csi-snapshot-controller","namespace":"kube-system"},"namespace":"kube-system","name":"aws-ebs-csi-snapshot-controller","reconcileID":"1bd81ebb-9c13-4439-bec1-2597c99d3ec8"}                                                │
│ {"level":"info","ts":"2023-08-30T14:25:12.296Z","msg":"reconcilation finished in 31.561329ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"na │
│ me":"vertical-pod-autoscaler","namespace":"kube-system"},"namespace":"kube-system","name":"vertical-pod-autoscaler","reconcileID":"48849661-6822-434d-bbcc-95e78a7ffb21"}                                                                │
│ {"level":"info","ts":"2023-08-30T14:25:12.299Z","msg":"HelmChart 'traefik/traefik-traefik' is not ready","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"tra │
│ efik","namespace":"traefik"},"namespace":"traefik","name":"traefik","reconcileID":"0235fc94-7e8e-4561-9f76-341b3318e29e"}                                                                                                                │
│ {"level":"info","ts":"2023-08-30T14:25:12.383Z","msg":"reconcilation finished in 86.334997ms, next run in 14m0s","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"na │
│ me":"traefik","namespace":"traefik"},"namespace":"traefik","name":"traefik","reconcileID":"0235fc94-7e8e-4561-9f76-341b3318e29e"}                                                                                                        │
│ {"level":"info","ts":"2023-08-30T14:25:40.926Z","msg":"Stopping and waiting for non leader election runnables"}                                                                                                                          │
│ {"level":"info","ts":"2023-08-30T14:25:40.926Z","msg":"shutting down server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                          │
│ {"level":"info","ts":"2023-08-30T14:25:40.926Z","msg":"Stopping and waiting for leader election runnables"}                                                                                                                              │
│ {"level":"info","ts":"2023-08-30T14:25:40.926Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease"}

We are using EKS 1.27 and Flux v2.0.1.

No other Networkpolicies are deployed yet. Which additional permits needs Flux else? E.g. Api-Server, ...

Steps to reproduce

Create AWS EKS 1.27 Cluster (with Bottlerocket OS 1.14.3) and enable Networkpolicies in the VPC CNI Add-On. Deploy Flux.

Expected behavior

Flux should not crashes with default enabled Networkpolicies.

OS / Distro

Bottlerocket OS 1.14.3 (aws-k8s-1.27)

Flux version

v2.0.1

Flux check

N/A

Code of Conduct

stefanprodan commented 1 year ago

Can you please post here the logs from a crash looping controller, the startup part? I guess AWS CNI wrongly blocks access to the API server, we do have an egress allow all policy.

albertschwarzkopf commented 1 year ago

@stefanprodan Hi thanks for the reply. Here are the logs.

helm-controller:

┌──────────────────────────────────────────────────────────────────────────────────── Logs(flux-system/helm-controller-58d6476fdf-scx46:manager)[tail] ────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                    Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:Off                                                                                     │
│ {"level":"info","ts":"2023-08-31T12:56:15.568Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}                                                                                       │
│ {"level":"info","ts":"2023-08-31T12:56:15.570Z","logger":"setup","msg":"starting manager"}                                                                                                                                               │
│ {"level":"info","ts":"2023-08-31T12:56:15.570Z","msg":"Starting server","kind":"health probe","addr":":9440"}                                                                                                                            │
│ {"level":"info","ts":"2023-08-31T12:56:15.671Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                               │
│ {"level":"info","ts":"2023-08-31T12:56:15.671Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/helm-controller-leader-election...\n"}                                                                          │
│ {"level":"info","ts":"2023-08-31T12:56:15.680Z","logger":"runtime","msg":"successfully acquired lease flux-system/helm-controller-leader-election\n"}                                                                                    │
│ {"level":"info","ts":"2023-08-31T12:56:15.681Z","msg":"Starting EventSource","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","source":"kind source: *v2beta1.HelmRelease"}          │
│ {"level":"info","ts":"2023-08-31T12:56:15.681Z","msg":"Starting EventSource","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","source":"kind source: *v1beta2.HelmChart"}            │
│ {"level":"info","ts":"2023-08-31T12:56:15.681Z","msg":"Starting Controller","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease"}                                                        │
│ {"level":"info","ts":"2023-08-31T12:56:15.784Z","msg":"Starting workers","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","worker count":4}                                          │
│ {"level":"info","ts":"2023-08-31T12:56:44.587Z","msg":"Stopping and waiting for non leader election runnables"}                                                                                                                          │
│ {"level":"info","ts":"2023-08-31T12:56:44.587Z","msg":"shutting down server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                          │
│ {"level":"info","ts":"2023-08-31T12:56:44.587Z","msg":"Stopping and waiting for leader election runnables"}                                                                                                                              │
│ {"level":"info","ts":"2023-08-31T12:56:44.587Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease"}

kustomize-controller:

┌───────────────────────────────────────────────────────────────────────────────── Logs(flux-system/kustomize-controller-54987bd455-2448x:manager)[tail] ──────────────────────────────────────────────────────────────────────────────────┐
│                                                                                    Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:Off                                                                                     │
│ {"level":"info","ts":"2023-08-31T12:55:45.989Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}                                                                                       │
│ {"level":"info","ts":"2023-08-31T12:55:45.991Z","logger":"setup","msg":"starting manager"}                                                                                                                                               │
│ {"level":"info","ts":"2023-08-31T12:55:45.991Z","msg":"Starting server","kind":"health probe","addr":":9440"}                                                                                                                            │
│ {"level":"info","ts":"2023-08-31T12:55:46.091Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                               │
│ {"level":"info","ts":"2023-08-31T12:55:46.092Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/kustomize-controller-leader-election...\n"}                                                                     │
│ {"level":"info","ts":"2023-08-31T12:55:46.119Z","logger":"runtime","msg":"successfully acquired lease flux-system/kustomize-controller-leader-election\n"}                                                                               │
│ {"level":"info","ts":"2023-08-31T12:55:46.164Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1.Kustomization"}    │
│ {"level":"info","ts":"2023-08-31T12:55:46.165Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1beta2.OCIRepository │
│ {"level":"info","ts":"2023-08-31T12:55:46.165Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1.GitRepository"}    │
│ {"level":"info","ts":"2023-08-31T12:55:46.166Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1beta2.Bucket"}      │
│ {"level":"info","ts":"2023-08-31T12:55:46.166Z","msg":"Starting Controller","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization"}                                               │
│ {"level":"info","ts":"2023-08-31T12:55:46.269Z","msg":"Starting workers","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","worker count":4}                                 │
│ {"level":"info","ts":"2023-08-31T12:55:46.269Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.270Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.270Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.270Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.290Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.291Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.296Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.303Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.362Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:55:46.371Z","msg":"Dependencies do not meet ready condition, retrying in 30s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomi │
│ {"level":"info","ts":"2023-08-31T12:56:14.943Z","msg":"Stopping and waiting for non leader election runnables"}                                                                                                                          │
│ {"level":"info","ts":"2023-08-31T12:56:14.943Z","msg":"shutting down server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                          │
│ {"level":"info","ts":"2023-08-31T12:56:14.943Z","msg":"Stopping and waiting for leader election runnables"}                                                                                                                              │
│ {"level":"info","ts":"2023-08-31T12:56:14.943Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization"}       │
│ {"level":"info","ts":"2023-08-31T12:56:44.944Z","msg":"Stopping and waiting for caches"}                                                                                                                                                 │
│ {"level":"info","ts":"2023-08-31T12:56:44.944Z","msg":"Stopping and waiting for webhooks"}                                                                                                                                               │
│ {"level":"info","ts":"2023-08-31T12:56:44.944Z","msg":"Wait completed, proceeding to shutdown the manager"}                                                                                                                              │
│ {"level":"error","ts":"2023-08-31T12:56:44.952Z","logger":"setup","msg":"problem running manager","error":"failed waiting for all runnables to end within grace period of 30s: context deadline exceeded"}                               │
│ Stream closed EOF for flux-system/kustomize-controller-54987bd455-2448x (manager)

source-controlle:

┌───────────────────────────────────────────────────────────────────────────────────── Logs(flux-system/source-controller-f46df59c-gbjnb:manager)[1m] ─────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                    Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:On                                                                                      │
│ {"level":"info","ts":"2023-08-31T13:00:34.581Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}                                                                                       │
│ {"level":"info","ts":"2023-08-31T13:00:34.581Z","logger":"setup","msg":"caching of Helm index files is disabled"}                                                                                                                        │
│ {"level":"info","ts":"2023-08-31T13:00:34.585Z","logger":"setup","msg":"starting manager"}                                                                                                                                               │
│ {"level":"info","ts":"2023-08-31T13:00:34.585Z","msg":"Starting server","kind":"health probe","addr":":9440"}                                                                                                                            │
│ {"level":"info","ts":"2023-08-31T13:00:34.685Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                               │
│ {"level":"info","ts":"2023-08-31T13:00:34.686Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/source-controller-leader-election...\n"}                                                                        │
│ {"level":"info","ts":"2023-08-31T13:01:03.364Z","msg":"Stopping and waiting for non leader election runnables"}                                                                                                                          │
│ {"level":"info","ts":"2023-08-31T13:01:03.364Z","msg":"shutting down server","path":"/metrics","kind":"metrics","addr":":8080"}                                                                                                          │
│ {"level":"info","ts":"2023-08-31T13:01:03.364Z","msg":"Stopping and waiting for leader election runnables"}                                                                                                                              │
│ {"level":"info","ts":"2023-08-31T13:01:03.365Z","msg":"Starting EventSource","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","source":"kind source: *v1.GitRepository"}       │
│ {"level":"info","ts":"2023-08-31T13:01:03.365Z","msg":"Starting Controller","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository"}                                                  │
│ {"level":"info","ts":"2023-08-31T13:01:03.368Z","msg":"Starting EventSource","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket","source":"kind source: *v1beta2.Bucket"}                       │
│ {"level":"info","ts":"2023-08-31T13:01:03.369Z","msg":"Starting Controller","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket"}                                                                │
│ {"level":"info","ts":"2023-08-31T13:01:03.369Z","msg":"Starting EventSource","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","source":"kind source: *v1beta2.HelmRepository │
│ "}                                                                                                                                                                                                                                       │
│ {"level":"info","ts":"2023-08-31T13:01:03.370Z","msg":"Starting Controller","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository"}                                                │
│ {"level":"info","ts":"2023-08-31T13:01:03.371Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1beta2.HelmChart"}              │
│ {"level":"info","ts":"2023-08-31T13:01:03.371Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1beta2.HelmRepository"}         │
│ {"level":"info","ts":"2023-08-31T13:01:03.372Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1.GitRepository"}               │
│ {"level":"info","ts":"2023-08-31T13:01:03.372Z","msg":"Starting EventSource","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","source":"kind source: *v1beta2.Bucket"}                 │
│ {"level":"info","ts":"2023-08-31T13:01:03.372Z","msg":"Starting Controller","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart"}                                                          │
│ {"level":"info","ts":"2023-08-31T13:01:03.374Z","msg":"Starting EventSource","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","source":"kind source: *v1beta2.HelmRepository │
│ "}                                                                                                                                                                                                                                       │
│ {"level":"info","ts":"2023-08-31T13:01:03.374Z","msg":"Starting Controller","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository"}                                                │
│ {"level":"info","ts":"2023-08-31T13:01:03.376Z","msg":"Starting EventSource","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository","source":"kind source: *v1beta2.OCIRepository"}  │
│ {"level":"info","ts":"2023-08-31T13:01:03.376Z","msg":"Starting Controller","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository"}                                                  │
│ {"level":"error","ts":"2023-08-31T13:01:03.379Z","logger":"controller-runtime.source.EventHandler","msg":"failed to get informer from cache","error":"Timeout: failed waiting for *v1.GitRepository Informer to sync"}                   │
│ {"level":"info","ts":"2023-08-31T13:01:03.379Z","msg":"Starting workers","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket","worker count":2}                                                  │
│ {"level":"info","ts":"2023-08-31T13:01:03.380Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket"}                        │
│ {"level":"info","ts":"2023-08-31T13:01:03.381Z","msg":"Starting workers","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","worker count":2}                                            │
│ {"level":"info","ts":"2023-08-31T13:01:03.382Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart"}                  │
│ {"level":"info","ts":"2023-08-31T13:01:03.382Z","msg":"Starting workers","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","worker count":2}                                  │
│ {"level":"info","ts":"2023-08-31T13:01:03.383Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository"}        │
│ {"level":"info","ts":"2023-08-31T13:01:03.384Z","msg":"Starting workers","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","worker count":2}                                    │
│ {"level":"info","ts":"2023-08-31T13:01:03.384Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository"}          │
│ {"level":"info","ts":"2023-08-31T13:01:03.384Z","msg":"Starting workers","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","worker count":2}                                  │
│ {"level":"info","ts":"2023-08-31T13:01:03.386Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository"}        │
│ {"level":"info","ts":"2023-08-31T13:01:03.388Z","msg":"Starting workers","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository","worker count":2}                                    │
│ {"level":"info","ts":"2023-08-31T13:01:03.388Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository"}          │
│ {"level":"info","ts":"2023-08-31T13:01:03.388Z","msg":"All workers finished","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket"}                                                               │
│ {"level":"info","ts":"2023-08-31T13:01:03.389Z","msg":"All workers finished","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart"}                                                         │
│ {"level":"info","ts":"2023-08-31T13:01:03.390Z","msg":"All workers finished","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository"}                                                 │
│ {"level":"info","ts":"2023-08-31T13:01:03.390Z","msg":"All workers finished","controller":"ocirepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"OCIRepository"}                                                 │
│ {"level":"error","ts":"2023-08-31T13:01:03.402Z","logger":"controller-runtime.source.EventHandler","msg":"failed to get informer from cache","error":"Timeout: failed waiting for *v1beta2.Bucket Informer to sync"}                     │
│ {"level":"error","ts":"2023-08-31T13:01:03.404Z","logger":"controller-runtime.source.EventHandler","msg":"failed to get informer from cache","error":"Timeout: failed waiting for *v1beta2.Bucket Informer to sync"}                     │
│ {"level":"error","ts":"2023-08-31T13:01:03.462Z","logger":"controller-runtime.source.EventHandler","msg":"failed to get informer from cache","error":"Timeout: failed waiting for *v1beta2.OCIRepository Informer to sync"}              │
│ {"level":"error","ts":"2023-08-31T13:01:03.465Z","msg":"Reconciler error","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"karpenter","namespace": │
│ "flux-system"},"namespace":"flux-system","name":"karpenter","reconcileID":"77323dd0-7c48-45b2-9ed4-9b09b4b5e505","error":"context canceled","errorCauses":[{"error":"context canceled","errorCauses":[{"error":"context canceled"},{"err │
│ or":"context canceled"}]}]}                                                                                                                                                                                                              │
│ {"level":"info","ts":"2023-08-31T13:01:03.468Z","msg":"All workers finished","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository"}                                               │
│ {"level":"info","ts":"2023-08-31T13:01:03.881Z","msg":"artifact up-to-date with remote revision: 'sha256:b29bb5693fc36b56aadcc53b2ab6020ac9f3707ffe7365e27b8132070207e51f'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"incubator","namespace":"flux-system"},"namespace":"flux-system","name":"incubator","reconcileID":"f9ecb312-89d5-4159-919b-f8a7fe420ef5"}                      │
│ {"level":"info","ts":"2023-08-31T13:01:04.062Z","msg":"artifact up-to-date with remote revision: 'sha256:1df959b442f64a7e148eacfe72b64358cda5c0c73b9d9f4b52c842e639cadb96'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"external-secrets","namespace":"external-secrets"},"namespace":"external-secrets","name":"external-secrets","reconcileID":"dfc7f964-f42b-4241-af24-f2acd80a570 │
│ 5"}                                                                                                                                                                                                                                      │
│ {"level":"info","ts":"2023-08-31T13:01:04.270Z","msg":"artifact up-to-date with remote revision: 'sha256:92273dedebeaa6c78831a238dba1b59e6643ff64bee1258822d67f0f4674ad38'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"traefik","namespace":"traefik"},"namespace":"traefik","name":"traefik","reconcileID":"624ed1cc-fee8-4ee6-a72c-0919386c3128"}                                  │
│ {"level":"info","ts":"2023-08-31T13:01:04.275Z","msg":"artifact up-to-date with remote revision: 'sha256:b61b15d27295678e1f0f55ce2252e59d0d5eefcfd9378438987f0c7d22b8874b'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"termination-handler","namespace":"kube-system"},"namespace":"kube-system","name":"termination-handler","reconcileID":"3eb73cd9-64c8-449c-880c-efa43f804ac5"}  │
│ {"level":"info","ts":"2023-08-31T13:01:04.367Z","msg":"artifact up-to-date with remote revision: 'sha256:fc2366fb61e7d6a0e3885ad3a9dbb4603483bab7fb7be193d7a19a32531ab33b'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"keda","namespace":"keda"},"namespace":"keda","name":"keda","reconcileID":"a680a92c-8fce-40d8-b1bb-a00b2ea3f93b"}                                              │
│ {"level":"info","ts":"2023-08-31T13:01:04.462Z","msg":"artifact up-to-date with remote revision: 'sha256:fc8b77772f25bd9ee80e623ca9dae6d75d9e12d7a0881f8e6569698c77e39c95'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"arvato","namespace":"flux-system"},"namespace":"flux-system","name":"arvato","reconcileID":"76ad94ec-061b-4a76-930f-7622eaea47cc"}                            │
│ {"level":"info","ts":"2023-08-31T13:01:04.471Z","msg":"artifact up-to-date with remote revision: 'sha256:6a6d60d7687d169ef6a98773250ade991b2666bd2b139b35585031a76ec54a1d'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"aws-ebs-csi-snapshot-controller","namespace":"kube-system"},"namespace":"kube-system","name":"aws-ebs-csi-snapshot-controller","reconcileID":"388a54da-bd5e-4 │
│ b3b-9c71-f9ea09c42e2a"}                                                                                                                                                                                                                  │
│ {"level":"info","ts":"2023-08-31T13:01:04.573Z","msg":"artifact up-to-date with remote revision: 'sha256:bcf4a99bdd95c734e79f335eace5c60ae98499379b87a4e91c4eab5c417f3edb'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"descheduler","namespace":"kube-system"},"namespace":"kube-system","name":"descheduler","reconcileID":"da94742b-c190-48f0-a20e-b8228c611477"}                  │
│ {"level":"info","ts":"2023-08-31T13:01:04.662Z","msg":"artifact up-to-date with remote revision: 'sha256:30ff2c24cdf104e8a8c1d4cabd29d67b1be26ff7fa30ee7c81fd697cd5f6c0f8'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"cert-manager","namespace":"cert-manager"},"namespace":"cert-manager","name":"cert-manager","reconcileID":"901feeed-d2fc-4b87-af8f-741cfc422371"}              │
│ {"level":"info","ts":"2023-08-31T13:01:04.674Z","msg":"artifact up-to-date with remote revision: 'sha256:b61b15d27295678e1f0f55ce2252e59d0d5eefcfd9378438987f0c7d22b8874b'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"aws-alb-ingress-controller","namespace":"kube-system"},"namespace":"kube-system","name":"aws-alb-ingress-controller","reconcileID":"aea57543-95d1-4b4e-a4cf-7 │
│ 9020320e863"}                                                                                                                                                                                                                            │
│ {"level":"info","ts":"2023-08-31T13:01:04.865Z","msg":"artifact up-to-date with remote revision: 'sha256:c019a926b73fac9181437efdf22a3dd42c58192be83c6731ae3ed578661b51bb'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"vertical-pod-autoscaler","namespace":"kube-system"},"namespace":"kube-system","name":"vertical-pod-autoscaler","reconcileID":"319d398c-2e9b-4e1d-bfb6-eaca77b │
│ 1a448"}                                                                                                                                                                                                                                  │
│ {"level":"error","ts":"2023-08-31T13:01:05.263Z","msg":"failed to patch","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"opencost","namespace":"o │
│ pencost"},"namespace":"opencost","name":"opencost","reconcileID":"953b434f-2f02-4050-9b21-4e1ad7311270","error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}                            │
│ {"level":"error","ts":"2023-08-31T13:01:05.265Z","msg":"Reconciler error","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"opencost","namespace":" │
│ opencost"},"namespace":"opencost","name":"opencost","reconcileID":"953b434f-2f02-4050-9b21-4e1ad7311270","error":"context canceled","errorCauses":[{"error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"con │
│ text canceled"}]},{"error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}]}                                                                                                               │
│ {"level":"error","ts":"2023-08-31T13:01:05.664Z","msg":"failed to patch","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"node-problem-detector"," │
│ namespace":"kube-system"},"namespace":"kube-system","name":"node-problem-detector","reconcileID":"c5d6e0d6-b837-421a-87ae-3472b39a0e58","error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context cancele │
│ d"}]}                                                                                                                                                                                                                                    │
│ {"level":"error","ts":"2023-08-31T13:01:05.666Z","msg":"Reconciler error","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"node-problem-detector", │
│ "namespace":"kube-system"},"namespace":"kube-system","name":"node-problem-detector","reconcileID":"c5d6e0d6-b837-421a-87ae-3472b39a0e58","error":"context canceled","errorCauses":[{"error":"context canceled","errorCauses":[{"error":" │
│ context canceled"},{"error":"context canceled"}]},{"error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}]}                                                                               │
│ {"level":"info","ts":"2023-08-31T13:01:05.808Z","msg":"artifact up-to-date with remote revision: 'sha256:6c904d7c78ccddcd015974879b2ac15e2de1154a72df2e77efce445a9994bde1'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"reloader","namespace":"stakater-reloader"},"namespace":"stakater-reloader","name":"reloader","reconcileID":"d3afe587-5b66-4855-97c7-e9dbc45a3876"}            │
│ {"level":"error","ts":"2023-08-31T13:01:06.174Z","msg":"failed to patch","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"trivy-server","namespace │
│ ":"security-system"},"namespace":"security-system","name":"trivy-server","reconcileID":"a4bc5e72-c4be-4229-b251-b978b1714662","error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}      │
│ {"level":"error","ts":"2023-08-31T13:01:06.177Z","msg":"Reconciler error","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"trivy-server","namespac │
│ e":"security-system"},"namespace":"security-system","name":"trivy-server","reconcileID":"a4bc5e72-c4be-4229-b251-b978b1714662","error":"context canceled","errorCauses":[{"error":"context canceled","errorCauses":[{"error":"context ca │
│ nceled"},{"error":"context canceled"}]},{"error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}]}                                                                                         │
│ {"level":"error","ts":"2023-08-31T13:01:07.268Z","msg":"failed to patch","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"deliveryhero","namespace │
│ ":"flux-system"},"namespace":"flux-system","name":"deliveryhero","reconcileID":"9e5e1fcc-20f7-4092-9079-497218991dc2","error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}              │
│ {"level":"error","ts":"2023-08-31T13:01:07.273Z","msg":"Reconciler error","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"deliveryhero","namespac │
│ e":"flux-system"},"namespace":"flux-system","name":"deliveryhero","reconcileID":"9e5e1fcc-20f7-4092-9079-497218991dc2","error":"context canceled","errorCauses":[{"error":"context canceled","errorCauses":[{"error":"context canceled"} │
│ ,{"error":"context canceled"}]},{"error":"context canceled","errorCauses":[{"error":"context canceled"},{"error":"context canceled"}]}]}                                                                                                 │
│ {"level":"info","ts":"2023-08-31T13:01:10.363Z","msg":"artifact up-to-date with remote revision: 'sha256:a97a6db135ad1913280ad0b2c6e6b205eddcf9e24ce2096b1385a81a30c281b6'","controller":"helmrepository","controllerGroup":"source.tool │
│ kit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"stable","namespace":"flux-system"},"namespace":"flux-system","name":"stable","reconcileID":"9468eb52-945c-448f-b064-8dd46f9a5f32"}                            │
│ {"level":"info","ts":"2023-08-31T13:01:10.572Z","logger":"runtime","msg":"successfully acquired lease flux-system/source-controller-leader-election\n"}                                                                                  │
│ {"level":"info","ts":"2023-08-31T13:01:10.573Z","logger":"setup","msg":"starting file server"}                                                                                                                                           │
│ Stream closed EOF for flux-system/source-controller-f46df59c-gbjnb (manager)
stefanprodan commented 1 year ago

So context canceled tells me that the AWS CNI blocks the access to Kubernetes API. IMO this is a bug in the CNI policy as we have an allow all egress that works with Calico, Cilium, Weave Net, etc. The policy is here: https://github.com/fluxcd/flux2/blob/main/manifests/policies/allow-egress.yaml

albertschwarzkopf commented 1 year ago

I have opened a support case in the affected AWS account. I will update this issue here.

albertschwarzkopf commented 1 year ago

AWS Support was able to reproduce this issue. The Calico setup is working. I'm waiting for further information.

teshsharma commented 1 year ago

Ran into this exact issue. Were you able to get any further info from AWS? @albertschwarzkopf

albertschwarzkopf commented 1 year ago

@teshsharma AWS is still working on this

felipesere commented 1 year ago

We are seeing a similar failure mode. SourceController comes up in EKS, but KustomizationController fails with these logs:

{"level":"info","ts":"2023-09-13T10:11:51.116Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2023-09-13T10:11:51.117Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-09-13T10:11:51.117Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}
{"level":"info","ts":"2023-09-13T10:11:51.218Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2023-09-13T10:11:51.218Z","logger":"runtime","msg":"attempting to acquire leader lease flux2/kustomize-controller-leader-election...\n"}
{"level":"info","ts":"2023-09-13T10:12:16.627Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2023-09-13T10:12:16.627Z","msg":"shutting down server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2023-09-13T10:12:16.627Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1.Kustomization"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1beta2.OCIRepository"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1.GitRepository"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1beta2.Bucket"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Starting Controller","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization"}
{"level":"error","ts":"2023-09-13T10:12:16.628Z","msg":"Could not wait for Cache to sync","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","error":"failed to wait for kustomization caches to sync: context canceled"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Stopping and waiting for caches"}
{"level":"error","ts":"2023-09-13T10:12:16.628Z","msg":"error received after stop sequence was engaged","error":"failed to wait for kustomization caches to sync: context canceled"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2023-09-13T10:12:16.628Z","msg":"Wait completed, proceeding to shutdown the manager"}
jdn5126 commented 1 year ago

As a side note, it looks like the KustomizationController and AWS EKS Node Agent can conflict on the port used for metrics (8080). Note that the latest AWS VPC CNI release: https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.14.1 changed the default metrics and health probe ports to avoid this conflict

tl-alex-nicot commented 1 year ago

we also have this problem running aws vpc cni 1.14.1

wc-s commented 1 year ago

@jdn5126 I don't think the port conflict there really matters. You'd only run into conflict if both node-agent and kustomize-controller run on hostNetwork: true mode. But while node-agent does do so, there's no reason for kustomize-controller to run on the hostNetwork.

tl-alex-nicot commented 1 year ago

we managed to get the pods to stop crashing by setting the ingress rule to allow traffic from 0.0.0.0/0 or {} but yeah not ideal

albertschwarzkopf commented 1 year ago

https://github.com/aws/amazon-vpc-cni-k8s/issues/2571

So let us wait for the next release.