Setting serviceAccount.rbac.clusterRole: false generates the errors

[nalshamaajc]

Setting serviceAccount.rbac.clusterRole: false generates the below errors Version: 1.0.259

W0812 18:38:07.585472 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:07.585517 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:07.585605 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:07.585643 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:08.580997 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:08.581033 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:08.583978 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:08.584010 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:10.321915 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:10.321943 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:10.913108 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:10.913168 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:14.950220 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:14.950247 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:15.804675 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:15.804712 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:25.853381 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:25.853424 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:26.232697 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:26.232731 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:43.229619 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:43.229679 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0812 18:38:45.947139 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0812 18:38:45.947181 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope

[moshloop]

Hi @nalshamaajc

You can workaround this error with the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: canary-checker-cluster-list
rules:
  - apiGroups:
      - canaries.flanksource.com
    resources:
      - canaries
      - topologies
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - canaries.flanksource.com
    resources:
      - canaries/status
      - topologies/status
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: canary-checker-cluster-listrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: canary-checker-cluster-list
subjects:
  - kind: ServiceAccount
    name: canary-checker-sa
    namespace: canary-checker

and

extraArgs:
  - include-namespace=NAMESPACE

@adityathebe I think we need set controller-runtime/cache/options.Namespace = WatchNamespace

[moshloop]

I also think we might need a serviceAccount.rbac.disabled=true

[nalshamaajc]

@moshloop Thank you, will try that.

[nalshamaajc]

@moshloop

Forgot to mention that I'm using version 1.0.259 After testing the above workaround I hit the below error,

2024-08-14T12:20:06.096 ERROR operator.controllers.canary failed to update status {"canary": "tcp-check-flanksource-website", "error": "canaries.canaries.flanksource.com \"tcp-check-flanksource-website\" is forbidden: User \"system:serviceaccount:canary-checker:canary-checker-sa\" cannot patch resource \"canaries/status\" in API group \"canaries.flanksource.com\" in the namespace \"default\""}
github.com/flanksource/canary-checker/pkg/controllers.(*CanaryReconciler).Report
/app/pkg/controllers/canary_controller.go:250
2024-08-14T12:20:06.166 ERROR operator.controllers.canary failed to update status {"canary": "dns-pass-consul", "error": "canaries.canaries.flanksource.com \"dns-pass-consul\" is forbidden: User \"system:serviceaccount:canary-checker:canary-checker-sa\" cannot patch resource \"canaries/status\" in API group \"canaries.flanksource.com\" in the namespace \"default\""}
github.com/flanksource/canary-checker/pkg/controllers.(*CanaryReconciler).Report
/app/pkg/controllers/canary_controller.go:250
2024-08-14T12:20:06.244 ERROR operator.controllers.canary failed to update status {"canary": "dns-pass", "error": "canaries.canaries.flanksource.com \"dns-pass\" is forbidden: User \"system:serviceaccount:canary-checker:canary-checker-sa\" cannot patch resource \"canaries/status\" in API group \"canaries.flanksource.com\" in the namespace \"default\""}
github.com/flanksource/canary-checker/pkg/controllers.(*CanaryReconciler).Report
/app/pkg/controllers/canary_controller.go:250
2024-08-14T12:21:06.037 ERROR operator.controllers.canary failed to update status {"canary": "dns-pass-consul", "error": "canaries.canaries.flanksource.com \"dns-pass-consul\" is forbidden: User \"system:serviceaccount:canary-checker:canary-checker-sa\" cannot patch resource \"canaries/status\" in API group \"canaries.flanksource.com\" in the namespace \"default\""}`

then I added the patch verb as shown below

  - apiGroups:
      - canaries.flanksource.com
    resources:
      - canaries/status
      - topologies/status
    verbs:
      - get
      - patch

but then I hit the following error

E0814 12:26:14.646296 1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"dns-pass-consul.17eb97f8316ce9b1", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Canary", Namespace:"default", Name:"dns-pass-consul", UID:"40aec649-f557-4d8c-a13a-4f4cc28435e1", APIVersion:"canaries.flanksource.com/v1", ResourceVersion:"220713882", FieldPath:""}, Reason:"Failed", Message:"dns-A/consul.service.stg01-us-east-2.consul: ", Source:v1.EventSource{Component:"canary-checker", Host:""}, FirstTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 636251569, time.Local), LastTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 636251569, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"canary-checker", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
14/Aug/2024:12:26:14 +0000: Schema cache loaded
2024-08-14T12:26:14.664 INFO Canary[default/pod-check] scheduled @every 5m
2024-08-14T12:26:14.666 WARN pod check is deprecated. Please use the kubernetes resource check
E0814 12:26:14.680542 1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"pod-check.17eb97f833ec6b13", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Canary", Namespace:"default", Name:"pod-check", UID:"1602c04b-4223-4c0b-8e12-81836f1d45ab", APIVersion:"canaries.flanksource.com/v1", ResourceVersion:"220713887", FieldPath:""}, Reason:"Failed", Message:"pod-golang: ", Source:v1.EventSource{Component:"canary-checker", Host:""}, FirstTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 678162195, time.Local), LastTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 678162195, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"canary-checker", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
2024-08-14T12:26:14.682 INFO Canary[default/pod-scheduling-check] scheduled @every 5m
E0814 12:26:14.696759 1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ingress-test.17eb97f834cb4635", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Canary", Namespace:"default", Name:"ingress-test", UID:"f044433e-ef33-4bf4-be13-06f604a57519", APIVersion:"canaries.flanksource.com/v1", ResourceVersion:"220713884", FieldPath:""}, Reason:"Failed", Message:"kubernetes_resource-ingress-accessibility-check: ", Source:v1.EventSource{Component:"canary-checker", Host:""}, FirstTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 692767285, time.Local), LastTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 692767285, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"canary-checker", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
2024-08-14T12:26:14.707 INFO Canary[default/redis-check] scheduled @every 5m
E0814 12:26:14.744596 1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"pod-scheduling-check.17eb97f83783a282", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Canary", Namespace:"default", Name:"pod-scheduling-check", UID:"a3ab5875-910f-43e8-b912-7f7fa8099524", APIVersion:"canaries.flanksource.com/v1", ResourceVersion:"220713890", FieldPath:""}, Reason:"Failed", Message:"kubernetes_resource-pod exit code: ", Source:v1.EventSource{Component:"canary-checker", Host:""}, FirstTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 738403970, time.Local), LastTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 738403970, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"canary-checker", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
2024-08-14T12:26:14.747 INFO Canary[default/scheduling-check] scheduled @every 5m
2024-08-14T12:26:14.766 INFO Canary[default/tcp-check-flanksource-website] scheduled @every 1m
E0814 12:26:14.771406 1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"scheduling-check.17eb97f8395b7e03", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Canary", Namespace:"default", Name:"scheduling-check", UID:"63cbd4d7-fee0-4ade-bd27-35f3089df69d", APIVersion:"canaries.flanksource.com/v1", ResourceVersion:"220713895", FieldPath:""}, Reason:"Failed", Message:"kubernetes_resource-pod exit code: ", Source:v1.EventSource{Component:"canary-checker", Host:""}, FirstTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 769327619, time.Local), LastTimestamp:time.Date(2024, time.August, 14, 12, 26, 14, 769327619, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"canary-checker", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
2024-08-14T12:26:14.789 INFO Canary[default/dns-pass] scheduled @every 1m
E0814 12:26:26.798506 1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"redis-check.17eb97fb061fce6c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Canary", Namespace:"default", Name:"redis-check", UID:"7f7670e8-db64-4256-b60d-28f27ebe7675", APIVersion:"canaries.flanksource.com/v1", ResourceVersion:"220714182", FieldPath:""}, Reason:"Failed", Message:"redis-replica.dev-usw2-p03-argocd.otuvyh.usw2.cache.amazonaws.com:6379: ", Source:v1.EventSource{Component:"canary-checker", Host:""}, FirstTimestamp:time.Date(2024, time.August, 14, 12, 26, 26, 794679916, time.Local), LastTimestamp:time.Date(2024, time.August, 14, 12, 26, 26, 794679916, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"canary-checker", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)

and it seems that adding the following rule fixed it

  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch

full cluster role

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: canary-checker-cluster-list
rules:
  - apiGroups:
      - canaries.flanksource.com
    resources:
      - canaries
      - topologies
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - canaries.flanksource.com
    resources:
      - canaries/status
      - topologies/status
    verbs:
      - get
      - patch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch

[nalshamaajc]

@moshloop I was testing v1.0.260-beta.179, but got similar errors was the fix merged in a later release?

{"time":"2024-09-04T20:50:54.640600607Z","level":"DEBUG-1","msg":"CREATE TYPE \"public\".\"playbook_run_status\" AS ENUM ('scheduled', 'running', 'cancelled', 'completed', 'failed', 'pending', 'sleeping')","logger":"migrate"}
{"time":"2024-09-04T20:50:54.640906596Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"check_statuses_1h\" (\"check_id\" uuid NOT NULL, \"created_at\" timestamptz NOT NULL, \"duration\" integer NOT NULL, \"total\" integer NOT NULL, \"passed\" integer NOT NULL, \"failed\" integer NOT NULL, PRIMARY KEY (\"check_id\", \"created_at\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.640935365Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"check_statuses_1d\" (\"check_id\" uuid NOT NULL, \"created_at\" timestamptz NOT NULL, \"duration\" integer NOT NULL, \"total\" integer NOT NULL, \"passed\" integer NOT NULL, \"failed\" integer NOT NULL, PRIMARY KEY (\"check_id\", \"created_at\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:55.434488162Z","level":"DEBUG-1","msg":"initalized CleanupCanaries{schedule=@every 12h, timeout=0s, history=true, singleton=true, retention=(success=3, failed=3)}","logger":"cleanupcanaries"}
{"time":"2024-09-04T20:50:55.434730812Z","level":"DEBUG-1","msg":"initalized SyncCanaryJobs{schedule=@every 5m, timeout=0s, history=true, singleton=true, retention=(success=1, failed=3)}","logger":"synccanaryjobs"}
{"time":"2024-09-04T20:50:55.434729333Z","level":"DEBUG-1","msg":"initalized CleanupMetricsGauges{schedule=@every 1h, timeout=0s, history=true, singleton=true, retention=(success=3, failed=3)}","logger":"cleanupmetricsgauges"}
{"time":"2024-09-04T20:50:55.434778767Z","level":"DEBUG-1","msg":"initalized SyncTopology{schedule=@every 5m, timeout=0s, history=true, singleton=true, retention=(success=1, failed=3)}","logger":"synctopology"}
W0904 20:50:55.445362 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:50:55.445410 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:50:55.445554 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:50:55.445609 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:50:56.757928 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:50:56.757967 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:50:56.934144 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:50:56.934172 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:50:59.839516 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:50:59.839579 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:50:59.889706 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:50:59.889740 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:51:03.434808 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:51:03.434837 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:51:04.620844 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:51:04.620873 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:51:10.205954 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:51:10.206003 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:51:13.372321 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:51:13.372349 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:51:24.602128 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:51:24.602174 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Topology: failed to list *v1.Topology: topologies.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "topologies" in API group "canaries.flanksource.com" at the cluster scope
W0904 20:51:37.045374 1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
E0904 20:51:37.045400 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.8/tools/cache/reflector.go:229: Failed to watch *v1.Canary: failed to list *v1.Canary: canaries.canaries.flanksource.com is forbidden: User "system:serviceaccount:canary-checker:canary-checker-sa" cannot list resource "canaries" in API group "canaries.flanksource.com" at the cluster scope
{"time":"2024-09-04T20:51:55.001063471Z","level":"DEBUG-1","msg":"initalized RefreshCheckStatusSummary{schedule=@every 1m, timeout=1m0s, history=true, singleton=true, retention=(success=1, failed=3)}","logger":"refreshcheckstatussummary"}

{"time":"2024-09-04T20:50:54.288392929Z","level":"INFO","msg":"Connecting to url=postgres://postgres:xxxxx@localhost:40433/embedded?sslmode=disable migrate=true log=error postgrest=(version:v10.0.0 port=3000 log-level=info, jwt=md5(5ae5132e),length=32)"}
{"time":"2024-09-04T20:50:54.640766044Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"check_statuses\" (\"check_id\" uuid NOT NULL, \"details\" jsonb NULL, \"duration\" integer NULL, \"error\" text NULL, \"time\" timestamptz NOT NULL, \"created_at\" timestamptz NOT NULL DEFAULT now(), \"invalid\" boolean NULL, \"message\" text NULL, \"status\" boolean NULL, \"severity\" text NULL, \"is_pushed\" boolean NOT NULL DEFAULT false, PRIMARY KEY (\"check_id\", \"time\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.642077445Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"notifications\" (\"id\" uuid NOT NULL DEFAULT generate_ulid(), \"events\" text[] NOT NULL, \"error\" text NULL, \"title\" text NULL, \"template\" text NULL, \"filter\" text NULL, \"properties\" jsonb NULL, \"person_id\" uuid NULL, \"team_id\" uuid NULL, \"repeat_interval\" text NULL, \"group_by\" text[] NULL, \"custom_services\" jsonb NULL, \"created_by\" uuid NULL, \"created_at\" timestamptz NOT NULL DEFAULT now(), \"source\" \"public\".\"source\" NULL, \"updated_at\" timestamptz NOT NULL DEFAULT now(), \"deleted_at\" timestamptz NULL, PRIMARY KEY (\"id\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.642164847Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"notification_send_history\" (\"id\" uuid NOT NULL DEFAULT generate_ulid(), \"notification_id\" uuid NOT NULL, \"body\" text NOT NULL, \"status\" text NULL, \"source_event\" text NOT NULL, \"resource_id\" uuid NOT NULL, \"person_id\" uuid NULL, \"error\" text NULL, \"duration_millis\" integer NULL, \"created_at\" timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (\"id\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.642283189Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"playbook_runs\" (\"id\" uuid NOT NULL DEFAULT generate_ulid(), \"playbook_id\" uuid NOT NULL, \"status\" text NOT NULL DEFAULT 'pending', \"created_at\" timestamptz NOT NULL DEFAULT now(), \"start_time\" timestamptz NULL, \"scheduled_time\" timestamptz NOT NULL DEFAULT now(), \"end_time\" timestamptz NULL, \"created_by\" uuid NULL, \"check_id\" uuid NULL, \"config_id\" uuid NULL, \"component_id\" uuid NULL, \"parameters\" jsonb NULL, \"request\" jsonb NULL, \"agent_id\" uuid NULL DEFAULT '00000000-0000-0000-0000-000000000000', \"error\" text NULL, PRIMARY KEY (\"id\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.642405156Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"playbook_run_actions\" (\"id\" uuid NOT NULL DEFAULT generate_ulid(), \"name\" text NOT NULL, \"status\" text NOT NULL DEFAULT 'running', \"playbook_run_id\" uuid NULL, \"start_time\" timestamptz NULL, \"scheduled_time\" timestamptz NOT NULL DEFAULT now(), \"end_time\" timestamptz NULL, \"result\" jsonb NULL, \"is_pushed\" boolean NOT NULL DEFAULT false, \"agent_id\" uuid NULL DEFAULT '00000000-0000-0000-0000-000000000000', \"error\" text NULL, PRIMARY KEY (\"id\"), CONSTRAINT \"playbook_action_not_null_run_id\" CHECK ( (playbook_run_id IS NULL AND agent_id IS NOT NULL) OR\n (playbook_run_id IS NOT NULL)\n))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.642492633Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"event_queue\" (\"id\" uuid NOT NULL DEFAULT generate_ulid(), \"name\" text NOT NULL, \"properties\" jsonb NULL, \"error\" text NULL, \"created_at\" timestamptz NOT NULL DEFAULT now(), \"last_attempt\" timestamptz NULL, \"attempts\" integer NULL DEFAULT 0, \"priority\" integer NOT NULL DEFAULT 100, PRIMARY KEY (\"id\"))","logger":"migrate"}
{"time":"2024-09-04T20:50:54.642571892Z","level":"DEBUG-1","msg":"CREATE TABLE \"public\".\"job_history\" (\"id\" uuid NOT NULL DEFAULT generate_ulid(), \"agent_id\" uuid NOT NULL DEFAULT '00000000-0000-0000-0000-000000000000', \"name\" text NULL, \"success_count\" integer NULL, \"error_count\" integer NULL, \"details\" jsonb NULL, \"hostname\" text NULL, \"duration_millis\" integer NULL, \"resource_type\" text NULL, \"resource_id\" text NULL, \"status\" text NULL, \"time_start\" timestamptz NULL, \"time_end\" timestamptz NULL, \"created_at\" timestamptz NOT NULL DEFAULT now(), \"is_pushed\" boolean NOT NULL DEFAULT false, PRIMARY KEY (\"id\"))","logger":"migrate"}
{"time":"2024-09-04T20:52:55.435845045Z","level":"ERROR","msg":"error received after stop sequence was engaged","err":"failed to wait for canary caches to sync: timed out waiting for cache to be synced for Kind *v1.Canary"}
04/Sep/2024:20:52:55 +0000: {"code":"PGRST000","details":"connection to server at \"localhost\" (::1), port 40433 failed: FATAL: the database system is shutting down\n","hint":null,"message":"Database connection error. Retrying the connection."}
{"time":"2024-09-04T20:52:55.603600975Z","level":"DEBUG-1","msg":"finished duration=603.273ms, error=FATAL: terminating connection due to administrator command (SQLSTATE 57P01)","logger":"job. refresh check status summary"}

PS: I removed the changes I mentioned in https://github.com/flanksource/canary-checker/issues/2036#issuecomment-2288642157

And I'm using the below helm chart values (as a subchart).

image:
  tag: "v1.0.260-beta.179"
replicas: 1
debug: false
# -v, -vv, -vvv
logLevel: "-vvv"
jsonLogs: true
ingress:
  enabled: false
flanksource-ui:
  enabled: true
  ingress:
    enabled: false
resources:
  requests:
    cpu: 200m
    memory: 200Mi
  limits:
    memory: 2Gi
serviceAccount:
  rbac:
    # When set to false will not install Roles & Bindings.
    enable: true
    # Whether to create cluster-wide or namespaced roles
    clusterRole: false # set to true until https://github.com/flanksource/canary-checker/issues/2036 is resolved
podAnnotations: