Open owwweiha opened 1 year ago
@owwweiha Can you check the service account for your cns-manager deployment?
get-kubeconfig.sh script creates service account cnsmanager-sa
, ClusterRole & ClusterRoleBinding on the remote Kubernetes server(with vSphere CSI driver) that it will be managing. The purpose of it is to create a service account on remote cluster with minimum necessary privileges, and then create a kubeconfig with that service account which can be used for cluster registration.
There's a different service account for cns-manager deployment itself(assuming you're using basicauth deployment) - https://github.com/vmware-samples/cloud-native-storage-self-service-manager/blob/main/deploy/basic-auth/deploy-template.yaml#L1 . This service account is bound to a ClusterRole which has all the necessary permissions.
I'm not sure why you have a service account(cnsmanager-sa
) which is supposed to manage resource access on remote Kubernetes cluster(with vSphere CSI driver) trying to access resources meant for cns-manager
application. You may have CNS manager deployed on the same kubernetes cluster, but then there should be 2 different service accounts each bound to a different ClusterRole.
Hi @gohilankit,
thank you for your response. You're right...
I used the get-kubeconfig script to generate the sv_kubeconfig
, that's obviously wrong. The reason why I did this is that we're using OIDC auth-provider. When using my admins' kubeconfig containing OIDC, I'll get:
2023-04-12T08:47:51.579Z ERROR Main volumemigrationjob/volumemigrationjob_controller.go:79 KubeClient creation failed {"error": "no Auth Provider found for name \"oidc\""} gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/cnsoperator/controller/volumemigrationjob.Add /go/src/pkg/cnsoperator/controller/volumemigrationjob/volumemigrationjob_controller.go:79 gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/cnsoperator/controller.AddToManager /go/src/pkg/cnsoperator/controller/controller.go:32 gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/cnsoperator/manager.InitCnsManagerOperator /go/src/pkg/cnsoperator/manager/init.go:92 main.initCnsManagerOperator.func1 /go/src/main.go:64 2023-04-12T08:47:51.579Z ERROR Main manager/init.go:93 failed to setup controller for Cns manager operator {"error": "no Auth Provider found for name \"oidc\""} gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/cnsoperator/manager.InitCnsManagerOperator /go/src/pkg/cnsoperator/manager/init.go:93 main.initCnsManagerOperator.func1 /go/src/main.go:64 2023-04-12T08:47:51.579Z ERROR Main src/main.go:65 Error initializing Cns manager Operator {"error": "no Auth Provider found for name \"oidc\""} main.initCnsManagerOperator.func1 /go/src/main.go:65
I modified the kubeconfig to use the cns-manager
SA now. So there is no issue for volumemigrationtasks
.
But I think events
are missing on the ClusterRole for the cns-manager
SA:
'events is forbidden: User "system:serviceaccount:cns-manager:cns-manager" cannot create resource "events" in API group "" in the namespace "cns-manager"' (will not retry!)
Nevertheless, using the cns-manager
SA will only work once it's created (which will be done by the deployment script) - so it won't work to pass the sv_kubeconfig
containing cns-manager
SA to the first run of deploy.sh
because the ServiceAccount does not exist yet. Any chance to get this working with OIDC as auth provider?
By the way, we're using TKGI. Maybe something is different here? E.g., the deployment uses the psp:vmware-system-privileged
ClusterRole (https://github.com/vmware-samples/cloud-native-storage-self-service-manager/blob/main/deploy/basic-auth/deploy-template.yaml#L15) which does not exist in TKGI. I created a ClusterRole to use the pks-privileged
PSP instead.
Closing this issue was a mistake, sorry!
Any news on this? Would be great to use a valid kubeconfig file during installation process (deploy.sh + basicauth) without modifying it. Right now, fresh (and totally valid) kubeconfig with my admin user gives me:
2023-07-10T08:19:35.135Z ERROR OrphanVolumeMonitoring ov/monitoring.go:403 Failed to create kube client. {"TraceId": "3784cc71-5a17-4c04-acfb-aa3afb983c3e", "ClusterID": "sv_kubeconfig", "erro r": "no Auth Provider found for name \"oidc\""} gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/ov.getPVsInRegisteredClusters /go/src/pkg/ov/monitoring.go:403 gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/ov.updateOrphanVolumeCache /go/src/pkg/ov/monitoring.go:106 gitlab.eng.vmware.com/calatrava/storage-sre/cns-manager/pkg/ov.InitOrphanVolumeMonitoring.func1 /go/src/pkg/ov/monitoring.go:61 reflect.Value.call /usr/local/go/src/reflect/value.go:556 reflect.Value.Call /usr/local/go/src/reflect/value.go:339 github.com/go-co-op/gocron.callJobFuncWithParams /go/pkg/mod/github.com/go-co-op/gocron@v1.6.2/gocron.go:76 github.com/go-co-op/gocron.(executor).start.func1.1 /go/pkg/mod/github.com/go-co-op/gocron@v1.6.2/executor.go:90 golang.org/x/sync/singleflight.(Group).doCall.func2 /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/singleflight/singleflight.go:193 golang.org/x/sync/singleflight.(Group).doCall /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/singleflight/singleflight.go:195 golang.org/x/sync/singleflight.(Group).Do /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/singleflight/singleflight.go:108 github.com/go-co-op/gocron.(*executor).start.func1 /go/pkg/mod/github.com/go-co-op/gocron@v1.6.2/executor.go:82
Describe the bug
When using the get-kubeconfig.sh script, the created cnsmanager-sa lacks of some permissions:
Adding this to the ClusterRole makes the error messages disappear:
Reproduction steps
Expected behavior
All needed RBAC permissions should be included and it shouldn't be necessary to add some in addition to the scripts that create them.
Additional context
No response