search

fluxcd / kustomize-controller

The GitOps Toolkit Kustomize reconciler
https://fluxcd.io
Apache License 2.0
253 stars 182 forks source link

Dry-run apply Roles & ClusterRoles before RoleBindings & ClusterRoleBindings #748

Closed demikl closed 2 years ago

demikl commented 2 years ago

Short description of the issue

When the kustomize controller wants to reconcile a commit that adds a linked (Cluster)Role and (Cluster)RoleBinding, the dry-run step would fail if the impersonated service account is not cluster-admin.

My hypothesis is that the RBAC controller cannot check whether the (cluster)rolebinding would escalate privileges because it does not know the content of the attached role, and thus will deny the request. The reconciliation works fine when the kustomize-controller is using a cluster-admin service account for the Kustomization, such as the kustomize-controller service account generated by the flux install command.

How to reproduce

Create a git repository with the following structure:

/
├── rolebinding.yaml
└── role.yaml

Create a Kustomization that would reconciliate from this repo, using an impersonated service-account that is allowed to create roles and rolebindings but is not cluster-admin:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: test
  namespace: myns
spec:
  interval: 5m
  path: ./
  prune: true
  sourceRef:
    kind: GitRepository
    name: test
  wait: true
  serviceAccountName: flux-deployer

Here is the content of the two files on the git repo:

role.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: test-role
  namespace: myns
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]

rolebinding.yaml

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: test-rolebinding
  namespace: myns
subjects:
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: test-role
  apiGroup: rbac.authorization.k8s.io

The reconciliation will fail with this message:

RoleBinding/myns/test-rolebinding namespace not specified, error: roles.rbac.authorization.k8s.io "test-role" not found

stefanprodan commented 2 years ago

Can you post here the flux-deployer RBAC please

demikl commented 2 years ago

The service account flux-deployer has only one rolebinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: flux-deployer
  namespace: myns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flux:deployer:aggregated
subjects:
- kind: ServiceAccount
  name: flux-deployer
  namespace: myns

Here is the clusterrole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: flux:deployer:aggregated
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.authorization.k8s.io/aggregate-to-flux-deployer: "true"
rules:
- apiGroups:
  - cert-manager.io
  resources:
  - certificates
  - certificaterequests
  - issuers
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
- apiGroups:
  - cert-manager.io
  resources:
  - certificates/status
  verbs:
  - update
- apiGroups:
  - acme.cert-manager.io
  resources:
  - challenges
  - orders
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
- apiGroups:
  - cert-manager.io
  resources:
  - certificates
  - certificaterequests
  - issuers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - acme.cert-manager.io
  resources:
  - challenges
  - orders
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - update
- apiGroups:
  - elbv2.k8s.aws
  resources:
  - targetgroupbindings
  verbs:
  - create
  - patch
  - update
  - delete
  - get
  - list
  - watch
- apiGroups:
  - vpcresources.k8s.aws
  resources:
  - securitygrouppolicies
  verbs:
  - create
  - patch
  - update
  - delete
  - get
  - list
  - watch
- apiGroups:
  - external-secrets.io
  resources:
  - externalsecrets
  - secretstores
  - clustersecretstores
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
- apiGroups:
  - external-secrets.io
  resources:
  - externalsecrets
  - secretstores
  - clustersecretstores
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - apps
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - namespaces
  - persistentvolumeclaims
  - pods
  - secrets
  - serviceaccounts
  - services
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  - networkpolicies
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - update
  - patch
  - delete
  - list
  - get
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  - roles
  verbs:
  - create
  - delete
  - deletecollection
  - patch
  - update
  - list
  - get
- apiGroups:
  - vpcresources.k8s.aws
  resources:
  - securitygrouppolicies
  verbs:
  - list
  - get
  - create
  - patch
  - update
  - delete
- apiGroups:
  - source.toolkit.fluxcd.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - kustomize.toolkit.fluxcd.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - helm.toolkit.fluxcd.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - notification.toolkit.fluxcd.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - image.toolkit.fluxcd.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - keda.sh
  resources:
  - scaledobjects
  - scaledjobs
  - triggerauthentications
  verbs:
  - create
  - update
  - patch
  - delete
- apiGroups:
  - kyverno.io
  resources:
  - generaterequests
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - kyverno.io
  resources:
  - policies
  - clusterpolicies
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - wgpolicyk8s.io
  resources:
  - policyreport
  - clusterpolicyreport
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - kyverno.io
  resources:
  - reportchangerequests
  - clusterreportchangerequests
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - create
  - update
  - patch
  - delete
- apiGroups:
  - autoscaling.k8s.io
  resources:
  - verticalpodautoscalers
  verbs:
  - get
  - list
  - watch
  - create
  - patch
  - update
  - delete
- apiGroups:
  - container.evtech
  resources:
  - staticwaftunnels
  verbs:
  - create
  - patch
  - update
  - delete
stefanprodan commented 2 years ago

You need this for server-side apply to work:

- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  - rolebindings
  verbs:
  - '*'
demikl commented 2 years ago

Woow, that works! Thanks for the tip !

Which verb is used by the dry-run process ? According to the documentation, create and patch should be enough:

Since Server-Side Apply is a type of PATCH, a role will require the PATCH permission to edit resources, but will also need the CREATE verb permission in order to create resources with Server-Side Apply.

source: https://kubernetes.io/docs/reference/using-api/server-side-apply/#rbac-and-permissions

stefanprodan commented 2 years ago

Which verb is used by the dry-run process ? According to the documentation, create and patch should be enough:

This is also a mystery to me

demikl commented 2 years ago

The missing verb is bind on the roles resources!

Explanations: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#restrictions-on-role-binding-creation-or-update

stefanprodan commented 2 years ago

Nice find, can you please add an answer to this question please? https://github.com/fluxcd/flux2/discussions/3203