Figure out if we can limit the operator's RBAC permissions

[krnowak]

Right now we use ClusterRole, because we are granting some permissions to do actions on some cluster-wide resources. Those resources are:

• customresourcedefinitions
• namespaces

For customresourcedefinitions we probably could provide some CRD.yaml which an admin of a k8s cluster can use to register the CRD themselves. Then we could try dropping that resource from RBAC rules and see if the operator can handle the lack of permissions gracefully (likely it won't, it handles only the case where registering CRD fails because it already was registered).

For namespaces resources we only require listing. I haven't noticed any CoreV1().Namespaces().List() or anything like that, so my guess is that the permission is not required (not likely, though) or it is required because of our use of NamespaceAll member. Here I would try dropping the permission from the RBAC rules, change ClusterRole to Role restricted to some namespace and try running the operator with these modified permissions to see if everything works (creating, deleting and modifying habitats, statefulsets and pods in the same namespace as the RBAC rules allow).

[krnowak]

Another idea could be to split the big ClusterRole into one small ClusterRole only for the global resources and a Role for the namespaced resources. Still would be nice to have an operator working entirely for a specific namespace.

The thing with working for a specific namespace could be done through some --namespace flag we could pass to the operator, so it would know in which namespace it should operate. I think that prometheus operator does something like this.

[HT154]

I've been working on getting permission to share my (company internal) fork of the operator that adds support for exactly this. Here are the biggest things I've had to consider:

• CRD permissions
  • I was able to ask a cluster admin to install the CRD
  • I had to modify the operator's CreateCRD function(s) to Get first, and if it exists (and versions match up), don't try to create it
  • To do this, I had to ask for get permissions for CRDs
• Some clusters have resource quotas and default pod resource limits
  • I modified the operator to allow a corev1.ResourceRequirements to be provided in the V1beta2 struct to be set on the habitat-service container on the StatefulSet
  • The way the operator does a create-if(error:exists)-update on StatefulSets and ConfigMaps wreaks havoc on the Kubernetes quota counting mechanism and causes used counts to hit the quota even though the objects aren't actually created. I had to modify the logic to get-if(exists)-update-else-create to solve this.

Here's what the deployment manifests w/ limited RBAC permissions looks like:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: habitat-operator
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  labels:
    app: habitat-operator
  name: habitat-operator
spec:
  replicas: 1
  template:
    metadata:
      labels:
        operator: habitat
    spec:
      containers:
      - env:
        - name: HAB_OPERATOR_NAMESPACE # let k8s populate this so this manifest is more portable
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: <IMAGE>
        name: habitat-operator
      serviceAccountName: habitat-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: habitat-operator
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
- apiGroups:
  - habitat.sh
  resources:
  - habitats
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - apps
  resources:
  - deployments
  - statefulsets
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
  - deletecollection
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: habitat-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: habitat-operator
subjects:
- kind: ServiceAccount
  name: habitat-operator
  namespace: <NAMESPACE>

I opted for an environment variable over a CLI flag purely for the convenience of the fieldRef.

I asked my cluster administrator to install the CRD for me by hand:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: habitats.habitat.sh
spec:
  group: habitat.sh
  scope: Namespaced
  version: v1beta1
  names:
    kind: Habitat
    listKind: HabitatList
    plural: habitats
    shortNames:
    - hab
    singular: habitat