How can I create botkube without giving "clusterrole"

[daegeun-ha]

Is your feature request related to a problem? Please describe. In our company, kubernetes cluster is big, so not every developers has authorization to make 'clusterrole' resource. And if I use just 'role' than 'clusetrrole', lots of error msgs are appear like below. (pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:test-rbac:botkube-sa" cannot list resource "statefulsets" in API group "apps" at the cluster scope)

Describe the solution you'd like Can I use different k8s library for getting events from k8s api server?

Describe alternatives you've considered Can I use different k8s library for getting events from k8s api server? ( I don't have a exact solution...)

Additional context

[PrasadG193]

@daegeun-ha if we use clusterrole, then BotKube won't be able to monitor cluster-scoped resources like - PV, Namespace, Node, etc

We should have a field in BotKube config (something like isNamespaced) to restrict the scope of BotKube within a Namespace

[daegeun-ha]

We also know about that problem (won't be able to monitor cluster-scoped resources) But if I give just normal 'role' & 'rolebinding' to botkube, errors are occured like below.

Can I resolve this problem?

(E1216 01:55:23.030699 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "statefulsets" in API group "apps" at the cluster scope E1216 01:55:23.032901 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.RoleBinding: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E1216 01:55:23.032944 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.ClusterRole: clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope E1216 01:55:23.033483 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "daemonsets" in API group "apps" at the cluster scope E1216 01:55:23.033600 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "namespaces" in API group "" at the cluster scope E1216 01:55:23.033607 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "replicasets" in API group "apps" at the cluster scope E1216 01:55:23.033717 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "persistentvolumes" in API group "" at the cluster scope E1216 01:55:23.037779 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list v1.Event: events is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "events" in API group "" at the cluster scope E1216 01:55:23.129221 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "services" in API group "" at the cluster scope)

[daegeun-ha]

I hope this problem is solved. If the problem is solved, our team can use botkube to improve work efficiency.

[PrasadG193]

Hey @daegeun-ha, Sorry, I missed to reply to your earlier comment. Could you please let us know what role bindings you are using? We are planning to use dynamic informers which will hopefully solve this issue (https://github.com/infracloudio/botkube/issues/200), but that will take some time. Let's try to find another solution till then.

[daegeun-ha]

@PrasadG193 Thanks for reply! our role, service account, role binding are like below:

[role] apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: 2019-12-16T01:55:14Z name: botkube-role namespace: monitoring resourceVersion: "274459817" selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/monitoring/roles/botkube-role uid: 1a3c793d-1fa7-11ea-bbbc-fa163eaf6fe5 rules:

• apiGroups:
  • rbac.authorization.k8s.io
  • extensions
  • apps
  • batch resources:
  • pods
  • nodes
  • services
  • namespaces
  • secrets
  • configmaps
  • deployments
  • daemonsets
  • replicasets
  • statefulsets
  • ingresses
  • jobs
  • cronjobs
  • roles
  • rolebindings verbs:
  • get
  • list
  • watch
  • create
  • update
  • patch
  • delete
  • deletecollection

[serviceAccount] apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: 2019-12-16T01:55:14Z name: botkube-sa namespace: monitoring resourceVersion: "274459820" selfLink: /api/v1/namespaces/monitoring/serviceaccounts/botkube-sa uid: 1a3b1fa7-1fa7-11ea-bbbc-fa163eaf6fe5 secrets:

• name: ***(hidden)

[roleBinding] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: 2019-12-16T01:55:14Z name: botkube-rb namespace: monitoring resourceVersion: "274459819" selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/monitoring/rolebindings/botkube-rb uid: 1a3dbdf6-1fa7-11ea-bbbc-fa163eaf6fe5 roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: botkube-role subjects:

• kind: ServiceAccount name: botkube-sa

[PrasadG193]

Also, can you send the BotKube resource configuration ?

[daegeun-ha]

And.. I think it's the problem of getting k8s event from api server without clusterrole. Could a dynamic informer solve this problem..?

[daegeun-ha]

Below is botkube resource configuration ! (Currently, for experiment, I removed other resources from configuration)

config: resources:

• name: pod # Name of the resources e.g pod, deployment, ingress, etc. (Resource name must be in singular form) namespaces: include:
  • all ignore: # List of namespaces to be ignored (omitempty), used only with include: all

  • example : include [all], ignore [x,y,z]

    events: # List of lifecycle events you want to receive, e.g create, update, delete, error OR all

    • create
    • delete
    • error
• name: service namespaces: include:
  • all ignore:

  • events:

    • create
    • delete
    • error
• name: deployment namespaces: include:
  • all ignore:

  • events:

    • create
    • update
    • delete
    • error

[PrasadG193]

If you set “default” service account in deployment, you don’t need to create RBAC resources - clusterrrole, clusterrolebindings and serviceaccount. We should support changing service account through helm chart

[PrasadG193]

If you use role and rolebindings or “default” namespace, setting “all” namespace in resource config won’t work. You will be able to watch only namespace in which BotKube is deployed. Can you try replacing “all” namespaces with appropriate namespace?

[daegeun-ha]

@PrasadG193 OK, I'll try it the day after tomorrow. Happy new year! :>

[nadilas]

I have set include namespaces to one particular namespace and using the default service account, and only watching namespace relevant elements, still I get the message that the account doesn't have cluster scope.

  | E0213 04:22:50.567085 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:namespace:default" cannot list nodes at the cluster scope: no RBAC policy matched

I presume the issue lies in these two being initialized:

INFO[2020-02-13T04:22:50Z] Registering the filter NamespaceChecker
--
| INFO[2020-02-13T04:22:50Z] Registering the filter NodeEventsChecker

Although I don't understand this line: E0213 04:22:50.566979 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:namespace:default" cannot list replicasets.apps at the cluster scope: no RBAC policy matched

[PrasadG193]

@nadilas How are you deploying BotKube? I am assuming you are using deploy-all-in-one.yaml. If you are deploying BotKube in other namespace than botkube, then you need to update ClusteRoleBinding accordingly. Please make sure you are setting subject in ClusterRoleBinding correctly: https://github.com/infracloudio/botkube/blob/master/deploy-all-in-one.yaml#L260

[nadilas]

@PrasadG193 hi, I’m deploying namespace in a dedicated xy namespace, which I want to monitor. It should be namespaced and only look at resources within that namespace. I do not have cluster level access, hence cannot create a ClusterRoleBinding. I want to restrict it to just the one.

[PrasadG193]

Understood. By default BotKube register informers for all the resources. There should be a check to skip cluster level resources as proposed in this issue. Can you update your resource settings and set namespace xy for all the resources?

[nadilas]

@PrasadG193 Should be no issue, although I think I had that initially.. let me try it first thing on Monday.

[PrasadG193]

Now it is possible to change default rbac rule while installing the BotKube via helm. Please have a look at default values: https://github.com/infracloudio/botkube/blob/develop/helm/botkube/values.yaml#L311

[nadilas]

Understood. By default BotKube register informers for all the resources. There should be a check to skip cluster level resources as proposed in this issue. Can you update your resource settings and set namespace xy for all the resources?

Sorry @PrasadG193 ... I eventually got around to test this... it doesn't seem to work at all. I have set the included namespace for every single resource, like:

- name: pod
        namespaces:
          include:
            - namespace
        events:
          - all

But on startup it still runs into the same issue:

  | E0524 11:01:29.849217       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:namespace:default" cannot list services at the cluster scope: no RBAC policy matched
  | E0524 11:01:30.049081       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:namespace:default" cannot list nodes at the cluster scope: no RBAC policy matched
  | E0524 11:01:30.249372       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:namespace:default" cannot list pods at the cluster scope: no RBAC policy matched

[PrasadG193]

Fixed with https://github.com/infracloudio/botkube/pull/253

[kelvinwijaya]

Hi @PrasadG193

I did a test and it seem the clusterrole is still needed, and have been using the latest image with tag v0.11.0 as per your merged for fix #253 but no luck

E1002 16:36:21.845948       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108:
Failed to list *unstructured.Unstructured: events is forbidden: User "system:serviceaccount:mynamepsace:botkube-sa" cannot list resource "events" in API group "" at the cluster scope

I have follow the same instructions above as well to list all the resources to include namespace

namespace: 
  include:
    - mynamespace

[PrasadG193]

Thanks @kelvinwijaya for reporting. I will take a look

[akankshakumari393]

hey @kelvinwijaya Can you please share the role, rolebinding, serviceaccount you used to test.

[akankshakumari393]

@PrasadG193 we are using DynamicInformer which by default takes all namespace https://github.com/infracloudio/botkube/blob/develop/pkg/utils/utils.go#L141 Also, we have ResourceInformer Map https://github.com/infracloudio/botkube/blob/develop/pkg/utils/utils.go#L144 which stores Infomer per resource for all namespace by default. I think we can modify the map and store the informer for a particular namespace for that resource. Let me know your take on it.

[LaurentDumont]

Is this still current?

I am trying to deploy botkube in a cluster where we cannot leverage the ClusterRole as low-privileged tenants.

I am able to list ressources within my own namespace and have scoped the config to do just that. But I still see botkube trying to list ressources in that namespace at the Cluster scope.

E0716 16:22:17.633792       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: events is forbidden: User "system:serviceaccount:NAMESPACE_HERE:botkube" cannot list resource "events" in API group "" at the cluster scope
E0716 16:22:20.327095       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: pods is forbidden: User "system:serviceaccount:NAMESPACE_HERE:botkube" cannot list resource "pods" in API group "" at the cluster scope

Our service-account manifest

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubebot
  namespace: NAMESPACE_HERE
rules:
- apiGroups: [""]
  resources: ["daemonsets", "deployments", "pods", "replicationcontrollers"]
  verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubebot
  namespace: NAMESPACE_HERE
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: kubebot
  namespace: NAMESPACE_HERE
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubebot
subjects:
  - kind: ServiceAccount
    name: kubebot
    namespace: NAMESPACE_HERE

As an example, kubewatch is able to restrict itself to a namespace, without requiring Cluster wide permissions.

https://github.com/bitnami-labs/kubewatch

[pkosiec]

Hi, Please check out the latest Botkube which comes with RBAC configuration: https://docs.botkube.io/configuration/rbac If there's anything missing, please create a new issue. Cheers!