Error message "kube-state-metrics" is forbidden

[floek]

Hi,

I've found an error in the logs of the kube-state-metrics pod:

deployments.apps "kube-state-metrics" is forbidden: User "system:serviceaccount:kubeprod:kube-state-metrics" cannot get resource "deployments" in API group "apps" in the namespace "kubeprod"

Can anyone help, what's the problem here?

Thanks

floek

[javsalgar]

Hi,

Could you check if the roles and rolebindings in the kubeprod namespace for kube-state-metrics are created?

[floek]

Hi,

yes I've the following role binding created (from kubectl -n kubeprod get rolebindings kube-state-metrics -o yaml):

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    kubecfg.ksonnet.io/garbage-collect-tag: kube_prod_runtime
    name: kube-state-metrics
  name: kube-state-metrics
  namespace: kubeprod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kubeprod

floek

[floek]

And here is the Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kube-state-metrics-resizer
  namespace: kubeprod
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - extensions
  resourceNames:
  - kube-state-metrics
  resources:
  - deployments
  verbs:
  - get
  - update

Seems there is no entry for deployments and apiGroup "apps".

floek

[javsalgar]

Hi,

This is strange, as the apps API group is in the jsonnet manifest

    clusterRole: kube.ClusterRole($.p + "kube-state-metrics") {
      local core = "",  // workaround empty-string-key bug in `jsonnet fmt`
      local listwatch = {
        [core]: ["configmaps", "endpoints", "limitranges", "namespaces", "nodes", "persistentvolumeclaims", "persistentvolumes", "pods", "replicationcontrollers", "resourcequotas", "secrets", "services"],
        "admissionregistration.k8s.io": ["mutatingwebhookconfigurations", "validatingwebhookconfigurations"],
        apps: ["daemonsets", "deployments", "replicasets", "statefulsets"],
        autoscaling: ["horizontalpodautoscalers"],
        "autoscaling.k8s.io": ["verticalpodautoscalers"],
        batch: ["cronjobs", "jobs"],
        "certificates.k8s.io": ["certificatesigningrequests"],
        extensions: ["daemonsets", "deployments", "ingresses", "replicasets"],
        "networking.k8s.io": ["ingresses", "networkpolicies"],
        policy: ["poddisruptionbudgets"],
        "storage.k8s.io": ["storageclasses", "volumeattachments"],
        "storageclasses.k8s.io": ["storageclasses"],
      },
      all_resources:: std.set(std.flattenArrays(kube.objectValues(listwatch))),
      rules: [{
        apiGroups: [k],
        resources: listwatch[k],
        verbs: ["list", "watch"],
      } for k in std.objectFields(listwatch)],
    },

As you can see, there's an apps entry.

[floek]

Hi,

you showed me the code for the "ClusterRole". In the same file a view lines down there ist the code for the "Role":

    role: kube.Role($.p + "kube-state-metrics-resizer") + $.metadata {
      rules: [
        {
          apiGroups: [""],
          resources: ["pods"],
          verbs: ["get"],
        },
        {
          apiGroups: ["extensions"],
          resources: ["deployments"],
          resourceNames: ["kube-state-metrics"],
          verbs: ["get", "update"],
        },
      ],
    },

Here apps is missing, and from my understanding this is referenced by (https://github.com/bitnami/kube-prod-runtime/issues/1095#issuecomment-780018862)

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-state-metrics-resizer

and corresponds to my role shown in the comment https://github.com/bitnami/kube-prod-runtime/issues/1095#issuecomment-780020442

Floek