"cannot list resource "secrets" in API group "" at the cluster scope" => can't perform "INSERT INTO `annotation`

[tectumopticum]

The icinga-kubernetes-service fetches data but stops after this:

Oct 10 15:37:07 testnode icinga-kubernetes[15247]: I1010 15:37:07.389598   15247 main.go:69] Starting Icinga for Kubernetes (v0.2.0)
Oct 10 15:37:07 testnode icinga-kubernetes[15247]: I1010 15:37:07.392306   15247 database.go:286] "Connecting to database" logger="database"
Oct 10 15:37:07 testnode icinga-kubernetes[15247]: W1010 15:37:07.456376   15247 reflector.go:561] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list *v1.Secret: secrets is forbidden: User "testuser" cannot list resource "secrets" in API group "" at the cluster scope
Oct 10 15:37:07 testnode icinga-kubernetes[15247]: E1010 15:37:07.456426   15247 reflector.go:158] "Unhandled Error" err="k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"testuser\" cannot list resource \"secrets\" in API group \"\" at the cluster scope" logger="UnhandledError"
Oct 10 15:37:07 testnode icinga-kubernetes[15247]: W1010 15:37:07.458653   15247 reflector.go:561] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list *v1.Event: events.events.k8s.io is forbidden: User "testuser" cannot list resource "events" in API group "events.k8s.io" at the cluster scope
Oct 10 15:37:07 testnode icinga-kubernetes[15247]: E1010 15:37:07.458686   15247 reflector.go:158] "Unhandled Error" err="k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: Failed to watch *v1.Event: failed to list *v1.Event: events.events.k8s.io is forbidden: User \"testuser\" cannot list resource \"events\" in API group \"events.k8s.io\" at the cluster scope" logger="UnhandledError"
Oct 10 15:37:08 testnode icinga-kubernetes[15247]: F1010 15:37:08.251078   15247 main.go:385] can't retry: can't perform "INSERT INTO `annotation` (`uuid`, `name`, `value`) VALUES (:uuid, :name, :value) ON DUPLICATE KEY UPDATE `uuid` = VALUES(`uuid`), `name` = VALUES(`name`), `value` = VALUES(`value`)": Error 1406 (22001): Data too long for column 'name' at row 52
Oct 10 15:37:08 testnode systemd[1]: icinga-kubernetes.service: Main process exited, code=exited, status=255/EXCEPTION
Oct 10 15:37:08 testnode systemd[1]: icinga-kubernetes.service: Failed with result 'exit-code'.

I assume that a permission is missing for the readonly-user (?). But I think it would be sufficient if the daemon logs a warning and shouldn't stop working in this case.

BTW: when you have special-characters in the db-password of your kubernetes-user, make sure that you set the password in double-quotes in your /etc/icinga-kubernetes/config.yml. Otherwise you might be surprised about certain error-messages which are not clearly related to that.

/usr/sbin/icinga-kubernetes --version Icinga Kubernetes version: v0.2.0

Build information: Go version: go1.23.1 (linux, amd64) Git commit: 14ffceef97f30f2ba867588ddfca5616fb1708fe

System information: Platform: SLES Platform version: 15-SP6

[lippserd]

Hi @tectumopticum,

It should be fine that the user is not allowed to read secrets. The real problem is that you have an annotation that is longer than 63 characters. We chose this size because labels can't be longer either. You can try to fix this by issuing the following statement in the Icinga for Kubernetes database:

ALTER TABLE annotation MODIFY COLUMN name varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL;

Best regards, Eric

[tectumopticum]

Hi @lippserd ,

thanks for your helpful advice. Yes the daemon keeps running now. Here's the table-definition:

MariaDB [kubernetes]> show columns from annotation;
+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| uuid  | binary(16)   | NO   | PRI | NULL    |       |
| name  | varchar(255) | NO   |     | NULL    |       |
| value | mediumblob   | NO   |     | NULL    |       |
+-------+--------------+------+-----+---------+-------+

And this is an excerpt of the daemon's current output:

[...]
Oct 16 13:58:11 examplehost icinga-kubernetes[7595]: W1016 13:58:11.181072    7595 reflector.go:561] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list *v1.Secret: secrets is forbidden: User "hugo" cannot list resource "secrets" in API group "" at the cluster scope
Oct 16 13:58:11 examplehost icinga-kubernetes[7595]: E1016 13:58:11.181122    7595 reflector.go:158] "Unhandled Error" err="k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"hugo\" cannot list resource \"secrets\" in API group \"\" at the cluster scope" logger="UnhandledError"
Oct 16 13:58:30 examplehost icinga-kubernetes[7595]: I1016 13:58:30.971172    7595 request.go:700] Waited for 1.199791411s due to client-side throttling, not priority and fairness, request: GET:https://example/k8s/clusters/abcdefg/api/v1/namespaces/kafka-examplehost-helm-development-cluster/pods/kafka-helm-development-zookeeper-1/log?container=zookeeper&sinceSeconds=287
[...]

Best regards, Henning