Closed nairb774 closed 3 years ago
I noticed GC still wasn't working - no errors, no messages.
This is not related to impersonation, it should work regardless of the service account. Can you please provide an example?
@nairb774 GC should be fixed in flux2 v0.5.7, to clean up any stale objects you have to make a change to any YAML in Git and when the new revision is reconciled on the cluster, it will trigger a GC sweep. If you have any more issues with GC, you can now enable debug logging with flux install/bootstrap --log-level=debug
.
Running v0.5.3 of the kustomization-controller, with debug logs, I get the following log entries when GC is triggered:
{
"msg": "gc query failed for DestinationRule: destinationrules.networking.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"destinationrules\" in API group \"networking.istio.io\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.821Z"
}
{
"msg": "gc query failed for Sidecar: sidecars.networking.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"sidecars\" in API group \"networking.istio.io\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.822Z"
}
{
"msg": "gc query failed for ServiceEntry: serviceentries.networking.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"serviceentries\" in API group \"networking.istio.io\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.822Z"
}
{
"msg": "gc query failed for PeerAuthentication: peerauthentications.security.istio.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"peerauthentications\" in API group \"security.istio.io\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.823Z"
}
{
"msg": "gc query failed for Service: services is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"services\" in API group \"\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.824Z"
}
{
"msg": "gc query failed for StatefulSet: statefulsets.apps is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"statefulsets\" in API group \"apps\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.824Z"
}
{
"msg": "gc query failed for SealedSecret: sealedsecrets.bitnami.com is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"sealedsecrets\" in API group \"bitnami.com\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.825Z"
}
{
"msg": "gc query failed for Deployment: deployments.apps is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.826Z"
}
{
"msg": "gc query failed for VerticalPodAutoscaler: verticalpodautoscalers.autoscaling.k8s.io is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"verticalpodautoscalers\" in API group \"autoscaling.k8s.io\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.826Z"
}
{
"msg": "gc query failed for PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"poddisruptionbudgets\" in API group \"policy\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.827Z"
}
{
"msg": "gc query failed for ServiceAccount: serviceaccounts is forbidden: User \"system:serviceaccount:flux-system:flux-deployer\" cannot list resource \"serviceaccounts\" in API group \"\" in the namespace \"kafka\"",
"level": "info",
"kustomization": "flux-system/deploy-ns-kafka",
"logger": "controllers.Kustomization",
"ts": "2020-12-20T19:11:03.828Z"
}
Kustomization object in question:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
labels:
managed-namespace: kafka
shell-operator: 006-flux-provision
name: deploy-ns-kafka
namespace: flux-system
ownerReferences:
- apiVersion: v1
kind: Namespace
name: kafka
uid: fc483992-68fe-4aa5-8c8d-440607a0fcc6
spec:
dependsOn:
- name: cluster
interval: 1m
path: ./namespace/kafka/ai1
prune: true
serviceAccountName: deploy-ns-kafka
sourceRef:
kind: GitRepository
name: k8s
suspend: false
status:
conditions:
- lastTransitionTime: "2020-12-20T19:17:17Z"
message: 'Applied revision: master/0f9cf461e8cfce4558c2b30c8cd9717bd9cadd8c'
reason: ReconciliationSucceeded
status: "True"
type: Ready
lastAppliedRevision: master/0f9cf461e8cfce4558c2b30c8cd9717bd9cadd8c
lastAttemptedRevision: master/0f9cf461e8cfce4558c2b30c8cd9717bd9cadd8c
observedGeneration: 3
snapshot:
checksum: 83263030bad94b94e74f0acf8ff152c6f3e2e271
entries:
- kinds:
/v1, Kind=Service: Service
/v1, Kind=ServiceAccount: ServiceAccount
apps/v1, Kind=Deployment: Deployment
apps/v1, Kind=StatefulSet: StatefulSet
autoscaling.k8s.io/v1, Kind=VerticalPodAutoscaler: VerticalPodAutoscaler
bitnami.com/v1alpha1, Kind=SealedSecret: SealedSecret
networking.istio.io/v1beta1, Kind=DestinationRule: DestinationRule
networking.istio.io/v1beta1, Kind=ServiceEntry: ServiceEntry
networking.istio.io/v1beta1, Kind=Sidecar: Sidecar
policy/v1beta1, Kind=PodDisruptionBudget: PodDisruptionBudget
security.istio.io/v1beta1, Kind=PeerAuthentication: PeerAuthentication
namespace: kafka
The flux-system/flux-deployer
ServiceAccount was modified from the default install to have limited permissions - it only has the crd-controller-flux-system
role attached. Each Kustomization has a paired service account with the precise permissions needed for that Kustomization object (whitelisted namespace and object types). The errors are indicating that the operator is trying to use the wrong service account for GC operations. The correct service account per the Kustomization object would be the flux-system/deploy-ns-kafka
ServiceAccount.
In double checking a few things, I realized I had replaced the default
ServiceAccount with the flux-system/flux-deployer
ServiceAccount. This was done as we disallow any role bindings to target the default service accounts. I additionally added get/list/watch permissions on secrets and serviceaccounts in the flux-system namespace (to enable impersonation). Lastly, as I noted before, the cluster-reconciler-flux-system
ClusterRoleBinding attaching the cluster-admin
role to the default/flux-deployer ServiceAccount was removed.
After upgrading to v0.5.2 to pick up https://github.com/fluxcd/kustomize-controller/issues/209 I noticed GC still wasn't working - no errors, no messages. I think I've narrowed it down to the lack of impersonation support as noted here: https://github.com/fluxcd/kustomize-controller/blob/37e1c9847335016808d3003a61fe4e5322aac275/controllers/kustomization_controller.go#L995
Mostly filing the ticket to raise awareness of the missing feature, and hopefully get eyes towards having it addressed. Thanks!