Closed krnowak closed 6 years ago
Another idea could be to split the big ClusterRole into one small ClusterRole only for the global resources and a Role for the namespaced resources. Still would be nice to have an operator working entirely for a specific namespace.
The thing with working for a specific namespace could be done through some --namespace
flag we could pass to the operator, so it would know in which namespace it should operate. I think that prometheus operator does something like this.
I've been working on getting permission to share my (company internal) fork of the operator that adds support for exactly this. Here are the biggest things I've had to consider:
corev1.ResourceRequirements
to be provided in the V1beta2
struct to be set on the habitat-service container on the StatefulSetHere's what the deployment manifests w/ limited RBAC permissions looks like:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: habitat-operator
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
labels:
app: habitat-operator
name: habitat-operator
spec:
replicas: 1
template:
metadata:
labels:
operator: habitat
spec:
containers:
- env:
- name: HAB_OPERATOR_NAMESPACE # let k8s populate this so this manifest is more portable
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: <IMAGE>
name: habitat-operator
serviceAccountName: habitat-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: habitat-operator
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- get
- apiGroups:
- habitat.sh
resources:
- habitats
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- apps
resources:
- deployments
- statefulsets
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- deletecollection
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: habitat-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: habitat-operator
subjects:
- kind: ServiceAccount
name: habitat-operator
namespace: <NAMESPACE>
I opted for an environment variable over a CLI flag purely for the convenience of the fieldRef
.
I asked my cluster administrator to install the CRD for me by hand:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: habitats.habitat.sh
spec:
group: habitat.sh
scope: Namespaced
version: v1beta1
names:
kind: Habitat
listKind: HabitatList
plural: habitats
shortNames:
- hab
singular: habitat
Right now we use ClusterRole, because we are granting some permissions to do actions on some cluster-wide resources. Those resources are:
For customresourcedefinitions we probably could provide some CRD.yaml which an admin of a k8s cluster can use to register the CRD themselves. Then we could try dropping that resource from RBAC rules and see if the operator can handle the lack of permissions gracefully (likely it won't, it handles only the case where registering CRD fails because it already was registered).
For namespaces resources we only require listing. I haven't noticed any
CoreV1().Namespaces().List()
or anything like that, so my guess is that the permission is not required (not likely, though) or it is required because of our use ofNamespaceAll
member. Here I would try dropping the permission from the RBAC rules, change ClusterRole to Role restricted to some namespace and try running the operator with these modified permissions to see if everything works (creating, deleting and modifying habitats, statefulsets and pods in the same namespace as the RBAC rules allow).