Support for impersonation when running GC pass

[nairb774]

After upgrading to v0.5.2 to pick up https://github.com/fluxcd/kustomize-controller/issues/209 I noticed GC still wasn't working - no errors, no messages. I think I've narrowed it down to the lack of impersonation support as noted here: https://github.com/fluxcd/kustomize-controller/blob/37e1c9847335016808d3003a61fe4e5322aac275/controllers/kustomization_controller.go#L995

Mostly filing the ticket to raise awareness of the missing feature, and hopefully get eyes towards having it addressed. Thanks!

[stefanprodan]

I noticed GC still wasn't working - no errors, no messages.

This is not related to impersonation, it should work regardless of the service account. Can you please provide an example?

[stefanprodan]

@nairb774 GC should be fixed in flux2 v0.5.7, to clean up any stale objects you have to make a change to any YAML in Git and when the new revision is reconciled on the cluster, it will trigger a GC sweep. If you have any more issues with GC, you can now enable debug logging with flux install/bootstrap --log-level=debug.

[nairb774]

Running v0.5.3 of the kustomization-controller, with debug logs, I get the following log entries when GC is triggered:

{
  "msg": "gc query failed for DestinationRule: destinationrules.networking.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"destinationrules\" in API group \"networking.istio.io\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.821Z"
}
{
  "msg": "gc query failed for Sidecar: sidecars.networking.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"sidecars\" in API group \"networking.istio.io\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.822Z"
}
{
  "msg": "gc query failed for ServiceEntry: serviceentries.networking.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"serviceentries\" in API group \"networking.istio.io\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.822Z"
}
{
  "msg": "gc query failed for PeerAuthentication: peerauthentications.security.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"peerauthentications\" in API group \"security.istio.io\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.823Z"
}
{
  "msg": "gc query failed for Service: services is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"services\" in API group \"\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.824Z"
}
{
  "msg": "gc query failed for StatefulSet: statefulsets.apps is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"statefulsets\" in API group \"apps\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.824Z"
}
{
  "msg": "gc query failed for SealedSecret: sealedsecrets.bitnami.com is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"sealedsecrets\" in API group \"bitnami.com\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.825Z"
}
{
  "msg": "gc query failed for Deployment: deployments.apps is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.826Z"
}
{
  "msg": "gc query failed for VerticalPodAutoscaler: verticalpodautoscalers.autoscaling.k8s.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"verticalpodautoscalers\" in API group \"autoscaling.k8s.io\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.826Z"
}
{
  "msg": "gc query failed for PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"poddisruptionbudgets\" in API group \"policy\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.827Z"
}
{
  "msg": "gc query failed for ServiceAccount: serviceaccounts is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"serviceaccounts\" in API group \"\" in the namespace \"kafka\"",
  "level": "info",
  "kustomization": "flux-system/deploy-ns-kafka",
  "logger": "controllers.Kustomization",
  "ts": "2020-12-20T19:11:03.828Z"
}

Kustomization object in question:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  labels:
    managed-namespace: kafka
    shell-operator: 006-flux-provision
  name: deploy-ns-kafka
  namespace: flux-system
  ownerReferences:
  - apiVersion: v1
    kind: Namespace
    name: kafka
    uid: fc483992-68fe-4aa5-8c8d-440607a0fcc6
spec:
  dependsOn:
  - name: cluster
  interval: 1m
  path: ./namespace/kafka/ai1
  prune: true
  serviceAccountName: deploy-ns-kafka
  sourceRef:
    kind: GitRepository
    name: k8s
  suspend: false
status:
  conditions:
  - lastTransitionTime: "2020-12-20T19:17:17Z"
    message: 'Applied revision: master/0f9cf461e8cfce4558c2b30c8cd9717bd9cadd8c'
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  lastAppliedRevision: master/0f9cf461e8cfce4558c2b30c8cd9717bd9cadd8c
  lastAttemptedRevision: master/0f9cf461e8cfce4558c2b30c8cd9717bd9cadd8c
  observedGeneration: 3
  snapshot:
    checksum: 83263030bad94b94e74f0acf8ff152c6f3e2e271
    entries:
    - kinds:
        /v1, Kind=Service: Service
        /v1, Kind=ServiceAccount: ServiceAccount
        apps/v1, Kind=Deployment: Deployment
        apps/v1, Kind=StatefulSet: StatefulSet
        autoscaling.k8s.io/v1, Kind=VerticalPodAutoscaler: VerticalPodAutoscaler
        bitnami.com/v1alpha1, Kind=SealedSecret: SealedSecret
        networking.istio.io/v1beta1, Kind=DestinationRule: DestinationRule
        networking.istio.io/v1beta1, Kind=ServiceEntry: ServiceEntry
        networking.istio.io/v1beta1, Kind=Sidecar: Sidecar
        policy/v1beta1, Kind=PodDisruptionBudget: PodDisruptionBudget
        security.istio.io/v1beta1, Kind=PeerAuthentication: PeerAuthentication
      namespace: kafka

The flux-system/flux-deployer ServiceAccount was modified from the default install to have limited permissions - it only has the crd-controller-flux-system role attached. Each Kustomization has a paired service account with the precise permissions needed for that Kustomization object (whitelisted namespace and object types). The errors are indicating that the operator is trying to use the wrong service account for GC operations. The correct service account per the Kustomization object would be the flux-system/deploy-ns-kafka ServiceAccount.

[nairb774]

In double checking a few things, I realized I had replaced the default ServiceAccount with the flux-system/flux-deployer ServiceAccount. This was done as we disallow any role bindings to target the default service accounts. I additionally added get/list/watch permissions on secrets and serviceaccounts in the flux-system namespace (to enable impersonation). Lastly, as I noted before, the cluster-reconciler-flux-system ClusterRoleBinding attaching the cluster-admin role to the default/flux-deployer ServiceAccount was removed.