Unable to receive webhook alerts

[nomanyaqub]

I have created provider and alert like below but unable to receive notifications.

Provider spec:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: agent-ny-hr-alerts-provider
  namespace: aval-ny # <- Tried creating under flux-system ns as well
spec:
  address: http://agent-ny.aval-ny:3000/api/v1/state-tracking/flux
  type: generic

Alert spec:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: agent-ny-hr-alerts
  namespace: aval-ny # <- Tried creating under flux-system ns as well
spec:
  eventSeverity: info
  eventSources:
    - kind: HelmRelease
      name: '*'
      namespace: warehouses-ny # <-- there is where my HelmReleases live
  providerRef:
    name: agent-ny-hr-alerts-provider
  summary: Cluster helmrelease events for resource state tracking.

Moreover, I am running notification-controller in debug mode, it does not show any information at all about alerts and providers.

{"level":"info","ts":"2024-04-09T22:07:53.755Z","logger":"setup","msg":"starting event server","addr":":9090"}
{"level":"info","ts":"2024-04-09T22:07:53.756Z","logger":"setup","msg":"starting webhook receiver server","addr":":9292"}
{"level":"info","ts":"2024-04-09T22:07:53.756Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-04-09T22:07:53.756Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-04-09T22:07:53.757Z","msg":"starting server","kind":"health probe","addr":"[::]:9440"}
{"level":"info","ts":"2024-04-09T22:07:53.757Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-04-09T22:07:53.857Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/notification-controller-leader-election..."}
{"level":"info","ts":"2024-04-09T22:08:02.385Z","logger":"runtime","msg":"successfully acquired lease flux-system/notification-controller-leader-election"}
{"level":"debug","ts":"2024-04-09T22:08:02.385Z","logger":"events","msg":"notification-controller-5f8bfdb6d-m2425_5c19d6ca-a53b-431e-9c55-213658505625 became leader","type":"Normal","object":{"kind":"Lease","namespace":"flux-system","name":"notification-controller-leader-election","uid":"6039bf8f-86ac-4f71-8469-480b626eb0c4","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1491051337"},"reason":"LeaderElection"}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting EventSource","controller":"provider","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Provider","source":"kind source: *v1beta3.Provider"}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting Controller","controller":"provider","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Provider"}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting EventSource","controller":"receiver","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Receiver","source":"kind source: *v1.Receiver"}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting Controller","controller":"receiver","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Receiver"}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting EventSource","controller":"alert","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Alert","source":"kind source: *v1beta3.Alert"}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting workers","controller":"receiver","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Receiver","worker count":4}
{"level":"info","ts":"2024-04-09T22:08:02.386Z","msg":"Starting Controller","controller":"alert","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Alert"}
{"level":"info","ts":"2024-04-09T22:08:02.490Z","msg":"Starting workers","controller":"provider","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Provider","worker count":4}
{"level":"info","ts":"2024-04-09T22:08:02.490Z","msg":"Starting workers","controller":"alert","controllerGroup":"notification.toolkit.fluxcd.io","controllerKind":"Alert","worker count":4}

I also tried getting logs of alerts but there is no much information.

❯ flux logs --kind=Alert --follow --level=info -A
2024-04-09T22:08:02.386Z info Alert - Starting EventSource
2024-04-09T22:08:02.386Z info Alert - Starting Controller
2024-04-09T22:08:02.490Z info Alert - Starting workers

Kubernetes events does have events about the HelmReleases objects. Can someone please help? Thanks.

[stefanprodan]

Please post flux check and mention how did you installed Flux.

[nomanyaqub]

❯ flux check ► checking prerequisites ✗ flux 2.2.3 <2.3.0 (new CLI version is available, please upgrade) ✔ Kubernetes 1.28.8-gke.1095000 >=1.26.0-0 ► checking version in cluster ✔ distribution: flux-v2.2.3 ✔ bootstrapped: false ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.37.4 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.2.4 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.2.4 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta3 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta3 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

[nomanyaqub]

helm and source controller was installed using flux install --export | kubectl apply --dry-run=client -f- some months ago, whereas notification-controller was installed just few days ago using flux install command.

[stefanprodan]

Rerun install for all controllers so they know about notification endpoint.

[nomanyaqub]

Thanks @stefanprodan that did the trick. Now I can see events coming from source and helm controllers.

BTW, is there a way to disable notifications from some controller. I would like notification-controller to not receive events from source controller, just to keep its logs clean.

[stefanprodan]

You can patch the source-controller deployment and remove the notification endpoint from there, but it will be overrriten the moment you update Flux. The flux install command is for testing, in real clusters you should be using bootstrap that allows customising Flux is various ways.