gavinbunney / terraform-provider-kubectl

Terraform provider to handle raw kubernetes manifest yaml files
https://registry.terraform.io/providers/gavinbunney/kubectl
Mozilla Public License 2.0
612 stars 105 forks source link

override_namespace does not override roleRef/subjects for (Cluster)Role(Binding) resources #235

Closed bogdando closed 1 year ago

bogdando commented 1 year ago

When deploying setup-csi-snapshotter and rbac-snapshot-controller with kube_manifest provider, its override_namespace does not change, nor adds the wanted namespace value for roleRef/subjects. Resources end up created with mismatching ns, and controller fails to locate its RBAC definitions:

$ kubectl get clusterrolebinding -n csi-snapshotter snapshot-controller-role -o yaml  
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      annotations:
          <...>
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: snapshot-controller-runner
          # namespace is missing!
    subjects:
    - kind: ServiceAccount
      name: snapshot-controller
      namespace: kube-system # mismatches the desired override_namespace value

Also

$ kubectl get clusterrolebinding -n csi-snapshotter snapshot-controller-role -o yaml        
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
  < snip >
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: snapshot-controller-runner # missing namespace override
subjects:
- kind: ServiceAccount
  name: snapshot-controller
  namespace: kube-system # wrong value

Then the controller errors out like:

$ kubectl logs pod/snapshot-controller-76fcf499c5-68bqh -n csi-snapshotter
E1122 10:48:29.094055       1 main.go:87] Failed to list v1 volumesnapshots with 
error=volumesnapshots.snapshot.storage.k8s.io is forbidden: 
User "system:serviceaccount:csi-snapshotter:snapshot-controller" 
cannot list resource "volumesnapshots" in API group "snapshot.storage.k8s.io"
at the cluster scope

$ kubectl describe statefulset.apps/csi-snapshotter -n csi-snapshotter
  Warning  FailedCreate  2m30s (x17 over 7m58s)  statefulset-controller  create
Pod csi-snapshotter-0 in StatefulSet csi-snapshotter failed error: pods
"csi-snapshotter-0" is forbidden: error looking up service account 
csi-snapshotter/csi-snapshotter: serviceaccount "csi-snapshotter" not found

Please excuse me not providing a reproducer, but my code link where I hit by this. For my case, I can w/a that as I have introduced there a facility for merging patches (almost like kubectl patch, but at terraform layer, before the merged data gets to the k8s API server). But in general, this is a problem for the provider users.

yongzhang commented 1 year ago

I guess you misunderstand what override_namespace is, override_namespace is to override namespace for the resource itself, in your case, it is to add namespace to ClusterRoleBinding (although it does not have namespace).

So if you want to deploy the 2 yamls into a different namespace, you have to change roleRef to match the namespace.