Errors after helm install, "Failed to list *v1.Pod: pods is forbidden"

[danfinn]

Just recently installed your helm chart, pretty much as is but had to add a nodeSelector because we also have windows nodes. Other than that I didn't change any values. I'm getting the following errors:

kubectl logs -f kube-cleanup-operator-6c4747d7cb-6bdrn
2023/09/29 18:02:40 Starting the application. Version: , CommitTime:
2023/09/29 18:02:40 Provided options:
    namespace:
    dry-run: false
    delete-successful-after: 15m0s
    delete-failed-after: 0s
    delete-pending-after: 0s
    delete-orphaned-after: 1h0m0s
    delete-evicted-after: 15m0s
    ignore-owned-by-cronjobs: false

    legacy-mode: true
    keep-successful: 0
    keep-failures: -1
    keep-pending: -1
    label-selector:

2023/09/29 18:02:40
!!! DEPRECATION WARNING !!!
     Operator is running in `legacy` mode. Using old format of arguments. Please change the settings.
    `keep-successful` is deprecated, use `delete-successful-after` instead
    `keep-failures` is deprecated, use `delete-failed-after` instead
    `keep-pending` is deprecated, use `delete-pending-after` instead
 These fields are going to be removed in the next version

W0929 18:02:40.798716       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2023/09/29 18:02:40 Controller started...
2023/09/29 18:02:40 Listening at 0.0.0.0:7000
2023/09/29 18:02:41 Listening for changes...
E0929 18:02:41.835542       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope
E0929 18:02:43.121566       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope
E0929 18:02:46.228673       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope
E0929 18:02:51.557849       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope
E0929 18:03:00.762475       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope
E0929 18:03:19.001154       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope
E0929 18:04:02.187343       1 reflector.go:178] pkg/controller/controller_legacy.go:135: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-cleanup:kube-cleanup-operator" cannot list resource "pods" in API group "" at the cluster scope

Couple of questions:

1) What is missing that's not allowing it to list pods? 2) Why is the default to run in legacy mode if it's deprecated? 3) How do I tell it to monitor all namespaces, it's not clear from the docs?

[lwolf]

1. most likely RBAC is missing, verify that it got created
2. backwards compatibility with existing setups
3. --namespace should not be set and cluster-wide RBAC is deployed

[danfinn]

Here is what got created by the helm chart:

kubectl get sa kube-cleanup-operator -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    meta.helm.sh/release-name: kube-cleanup-operator
    meta.helm.sh/release-namespace: kube-cleanup
  creationTimestamp: "2023-09-29T17:53:24Z"
  labels:
    app.kubernetes.io/instance: kube-cleanup-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kube-cleanup-operator
    helm.sh/chart: kube-cleanup-operator-1.0.4
  name: kube-cleanup-operator
  namespace: kube-cleanup
  resourceVersion: "333606738"
  uid: d9e9abe4-8b74-493c-a0e4-dce4616b3cc0

That seems to be it. I don't see a clusterrole or clusterrolebinding. Is something needed to tell the helm chart to create those?

[lwolf]

how do you install the app, which chart do you use with which values?

[danfinn]

closing this, we decided not to use it.