search

cloudfoundry-incubator / kubecf

Cloud Foundry on Kubernetes
Apache License 2.0
115 stars 62 forks source link

Incorrect cluster role binding with Eirini and multiple deployments #1602

Closed mudler closed 4 years ago

mudler commented 4 years ago

Describe the bug By deploying Eirini twice in the same cluster, the following is observed in the eirini logs:

W1118 12:45:49.023665       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"timestamp":"2020-11-18T12:45:49.026286043Z","level":"info","source":"handler","message":"handler.opi-connected","data":{}}
{"timestamp":"2020-11-18T12:47:27.541348858Z","level":"debug","source":"handler","message":"handler.list-apps.requested","data":{"session":"2"}}
{"timestamp":"2020-11-18T12:47:27.690538039Z","level":"error","source":"desirer","message":"desirer.list.failed-to-list-statefulsets","data":{"error":"statefulsets.apps is forbidden: User \"system:serviceaccount:kubecf:opi\" cannot list resource \"statefulsets\" in API group \"apps\" at the cluster scope","session":"1"}}
{"timestamp":"2020-11-18T12:47:27.690665343Z","level":"error","source":"handler","message":"handler.list-apps.bifrost-failed","data":{"error":"failed to list desired LRPs: failed to list statefulsets: statefulsets.apps is forbidden: User \"system:serviceaccount:kubecf:opi\" cannot list resource \"statefulsets\" in API group \"apps\" at the cluster scope","session":"2"}}

and the cloud-controller-clock container in the scheduler pod crashes because of this.

To Reproduce Deploy two KubeCF with Eirini and 1 operator

Environment

Additional context https://github.com/cloudfoundry-incubator/quarks-operator/issues/1227

jandubois commented 4 years ago

Please verify if #1605 is fixing this issue as well.

Given that the cluster role definition works when Eirini is deployed only once, it feels like the role definition is correct.

But since the role binding is using a static global name, the second install may be breaking the first one by modifying its role binding. #1605 prepends cluster role and binding names with the release namespace name, so there should be no more name collision.

jandubois commented 4 years ago

@divyaaswath #1605 has been merged to master.

I hope it fixes https://github.com/cloudfoundry-incubator/quarks-operator/issues/1227 for you; please re-open this bug if it doesn't!