search

bentoml / yatai-deployment

🚀 Launching Bento in a Kubernetes cluster
17 stars 14 forks source link

RBAC issues after helm installation: Failed to watch *v1.Secret and other resources in yatai-system #129

Closed tmyhu closed 11 months ago

tmyhu commented 1 year ago

After installing yatai-deployment 1.1.16 via helm with mostly default values (only added enableRestrictedSecurityContext: true), the yatai-deployment pod logs show that it failed to list secrets in yatai-system:

yatai-deployment-8586fcd67c-hn65b manager E1107 21:25:58.840002       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:yatai-deployment:yatai-deployment" cannot list resource "secrets" in API group "" in the namespace "yatai-system"

I could see that a Role and RoleBinding for it had been created by helm that should give get/list/watch for the two secrets called yatai-common-env and yatai-deployment-shared-env but it seems that is not enough. I had to manually create a Role and RoleBinding to give access to all secrets in yatai-system namespace to fix this.

Then more errors appeared, failing to list other resources e.g.:

yatai-deployment-8586fcd67c-hn65b manager E1107 21:35:18.742937       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:yatai-deployment:yatai-deployment" cannot list resource "deployments" in API group "apps" in the namespace "yatai-system"
yatai-deployment-8586fcd67c-hn65b manager W1107 21:35:21.361358       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: failed to list *v1alpha1.Bento: bentoes.resources.yatai.ai is forbidden: User "system:serviceaccount:yatai-deployment:yatai-deployment" cannot list resource "bentoes" in API group "resources.yatai.ai" in the namespace "yatai-system"
yatai-deployment-8586fcd67c-hn65b manager E1107 21:35:21.361441       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.Bento: failed to list *v1alpha1.Bento: bentoes.resources.yatai.ai is forbidden: User "system:serviceaccount:yatai-deployment:yatai-deployment" cannot list resource "bentoes" in API group "resources.yatai.ai" in the namespace "yatai-system"
yatai-deployment-8586fcd67c-hn65b manager W1107 21:35:21.383532       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.25.0/tools/cache/reflector.go:169: failed to list *v2alpha1.BentoDeployment: bentodeployments.serving.yatai.ai is forbidden: User "system:serviceaccount:yatai-deployment:yatai-deployment" cannot list resource "bentodeployments" in API group "serving.yatai.ai" in the namespace "yatai-system"
(...)

To fix this, I had to add yatai-system to the values.bentoDeploymentNamespaces which creates the appropriate Role/RoleBinding for all these resources ie

bentoDeploymentNamespaces: ['yatai', 'yatai-system']

However, I suspect that this should not happen if bentos are supposed to be deployed in yatai namespace only, not yatai-system?

yetone commented 11 months ago

The latest version has resolved this issue; please update to yatai-deployment v1.1.20.