Feature Request: config to disable various controllers (kibana, elastic map, etc.)

[leonz]

Proposal

Use case. Why is this important?

Hi,

We are currently using ECK to manage our Elasticsearch clusters, but do not manage any other Elastic resources. In the interest of minimizing risk surface area, we would like to disable any functionality that we do not actively use. For instance, one concern is that even though we don't deploy Kibana, we are required to grant the ECK manager permissions to interact with Kibana resources.

A simple feature flag for each resource type would be greatly appreciated.

[thbkrkr]

For instance, one concern is that even though we don't deploy Kibana, we are required to grant the ECK manager permissions to interact with Kibana resources.

Can you explain why this is concerning?

[leonz]

Hey - our main concerns come from some environments where we deploy where we are not the only tenants in the Kubernetes stack.

This article describes our concerns well - https://loft.sh/blog/kubernetes-crds-huge-pain-in-multi-tenant-clusters/

But to summarize,

• we can’t prevent other tenants from using these CRDs even if we don’t want them to
• the more CRDs we define the more likely we may overlap with others
• from a principle of least privilege, we would like to permission all of our API server clients to have the very minimal set of authorization to perform their needed tasks (in our case, deploy only Elasticsearch)

[thbkrkr]

Ok, thanks for the additional explanation.

A simple feature flag for each resource type would be greatly appreciated.

Something like that?

--disable-agent-controller
--disable-apmserver-controller
--disable-beat-controller
--disable-elasticsearch-controller
--disable-enterprisesearch-controller
--disable-kibana-controller
--disable-elasticmapsserver-controller

Wouldn't a single flag be more practical?

--disabled-controllers="agent,apmserver,beat,kibana,elasticmapsserver"

Maybe also support the shortnames (obtainable through kubectl api-resources | grep elastic)?

--disabled-controllers="agent,apm,beat,kb,ems"

[leonz]

Sure that’s reasonable! I don’t have any strong opinions on the implementation.

[pebrc]

While I understand the intention (and we can definitely look into adding the ability to turn off individual controllers) I am still wondering why Kubernetes RBAC is not sufficient to prevent unauthorised users from trying to create Kibanas?

Could you simply not give them the necessary privileges to create the Kibana custom resource?

[leonz]

I may be wrong, but I’m pretty sure that we can’t use RBAC to prevent other users (we don’t control them) from being authorized to global resources, especially if they are working within their own namespace. And since the Operator listens to all namespaces they will then have the ability to influence our operator.

Granted I think in a multi tenant setup you could in theory get away with all kinds of crazy things, so none of this is fool proof.

[thbkrkr]

I may be wrong, but I’m pretty sure that we can’t use RBAC to prevent other users (we don’t control them) from being authorized to global resources, especially if they are working within their own namespace. And since the Operator listens to all namespaces they will then have the ability to influence our operator.

CRD are cluster-wide but custom resources are namespaces. So you can use RBAC to limit a ServiceAccount to access to a specific custom resource. Here is a small example so that:

• ServiceAccount my-a can manage Elasticsearch and Kibana in namespace a.
• ServiceAccount my-b can only manage Elasticsearch in namespace b.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: es-kb-role
rules:
- apiGroups:
    - ""
  resources:
    - pods
    - events
    - persistentvolumeclaims
    - secrets
    - services
    - configmaps
  verbs: ["get", "list"]
- apiGroups:
    - apps
  resources:
    - deployments
    - statefulsets
  verbs: ["get", "list"]
- apiGroups:
    - elasticsearch.k8s.elastic.co
  resources:
    - elasticsearches
    - elasticsearches/status
    - elasticsearches/finalizers
  verbs: ["get", "watch", "list", "create", "update", "patch", "delete"]
- apiGroups:
    - kibana.k8s.elastic.co
  resources:
    - kibanas
    - kibanas/status
    - kibanas/finalizers
  verbs: ["get", "watch", "list", "create", "update", "patch", "delete"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-a
  namespace: a
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: es-kb-a
  namespace: a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: es-kb-role
subjects: 
- kind: ServiceAccount
  name: my-a
  namespace: a

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: es-role
rules:
- apiGroups:
    - ""
  resources:
    - pods
    - events
    - persistentvolumeclaims
    - secrets
    - services
    - configmaps
  verbs: ["get", "list"]
- apiGroups:
    - apps
  resources:
    - deployments
    - statefulsets
  verbs: ["get", "list"]
- apiGroups:
    - elasticsearch.k8s.elastic.co
  resources:
    - elasticsearches
    - elasticsearches/status
    - elasticsearches/finalizers
  verbs: ["get", "watch", "list", "create", "update", "patch", "delete"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-b
  namespace: b
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: es-b
  namespace: b
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: es-role
subjects: 
- kind: ServiceAccount
  name: my-b
  namespace: b

Then:

# ~/.kube/config
contexts:
- context:
    cluster: ...
    namespace: a
    user: my-a

> k get es,kb,po     
NAME                                              HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/test   green    1       8.0.0     Ready   95s

NAME                                HEALTH   NODES   VERSION   AGE
kibana.kibana.k8s.elastic.co/test   green    1       8.0.0     95s

NAME                           READY   STATUS    RESTARTS   AGE
pod/test-es-master-0           1/1     Running   0          92s
pod/test-kb-545d78485f-hjljd   1/1     Running   0          91s

# ~/.kube/config
contexts:
- context:
    cluster: ...
    namespace: b
    user: my-b

> k get es,kb,po
NAME                                              HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/test   green    1       8.0.0     Ready   69s

NAME                   READY   STATUS    RESTARTS   AGE
pod/test-es-master-0   1/1     Running   0          69s
Error from server (Forbidden): kibanas.kibana.k8s.elastic.co is forbidden: User "system:serviceaccount:b:my-b" cannot list resource "kibanas" in API group "kibana.k8s.elastic.co" in the namespace "b"

[leonz]

Will ECK work though if we limit its permissions to only read in a given namespace? I tested this a while back and ECK must have access to all namespaces because of the way its listers are set up, so any service account that we don't control will be able to create custom resources that will affect our ECK.

[pebrc]

Will ECK work though if we limit its permissions to only read in a given namespace?

That is not what @thbkrkr is suggesting here. This is not about restricting the service account the ECK operator uses. The proposal is to give the service accounts related to human users who should be entitled to create Elastic Stack resources the necessary RBAC permissions to do CRUD operations on the custom resources managed by ECK and limit everyone else.

I tested this a while back and ECK must have access to all namespaces because of the way its listers are set up,

That is not correct. ECK can be configured to only manage resources in certain namespaces with correspondingly restricted RBAC permission for the service account the that ECK operator uses. If you use the Helm charts there is even a preconfigured profile for that https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-install-helm.html#k8s-install-helm-restricted If you don't use helm you can still replicate the same setup by adjusting the default installation manifests accordingly.

so any service account that we don't control will be able to create custom resources that will affect our ECK.

This is not how RBAC works in Kubernetes. Unless an administrator gives them explicit access to the custom resources related to ECK as illustrated above by @thbkrkr they will not have access (unless they are admins themselves of course).

[leonz]

The proposal is to give the service accounts related to human users who should be entitled to create Elastic Stack resources the necessary RBAC permissions to do CRUD operations on the custom resources managed by ECK and limit everyone else.

Right, my point is, I don't know how we can limit everyone else. They do have admin access, at least to their own namespaces.

ECK can be configured to only manage resources in certain namespaces with correspondingly restricted RBAC permission for the service account the that ECK operator uses

I think this is actually the missing piece that I've been looking for and is 50% of what I'm asking for here. Will try it out, thank you!

The other 50% is for the operator to function if some of the CRDs aren't installed, which I think does not work still?

[pebrc]

Right, my point is, I don't know how we can limit everyone else. They do have admin access, at least to their own namespaces.

So unless they have the ability to create new (cluster) role bindings I think you should be OK.

The other 50% is for the operator to function if some of the CRDs aren't installed, which I think does not work still?

Correct that won't work.