search

kubernetes / cloud-provider-vsphere

Kubernetes Cloud Provider for vSphere https://cloud-provider-vsphere.sigs.k8s.io
Apache License 2.0
242 stars 177 forks source link

Configuration and HA enhancements for running CCM external of an cluster #356

Closed MaxRink closed 3 years ago

MaxRink commented 4 years ago

/kind feature

We are currently using Cluster API with the vSphere provider. During that process we stumbled across the dependency to run the CCM internally of the created cluster. We would like to remove that dependency because of security reasons ( if the workload clusters have no way to even contact the vcenter, they cant influence other VMs running on the shared cluster) The CAPV Issue is tracked here: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/issues/924

In our experminets with running the CCM externally, which is working in general, we came across two Issues:

davidvonthenen commented 4 years ago

Hi @MaxRink

My KubeCon EU session is exactly this use case. You need to have individual vCenter accounts per workload cluster. Using vCenter RBAC, restrict the user access to only that particular vSphere folder or Resource Group. That should limit the scope to exactly those users VMs. This is can be setup in the CPI config as tenant configurations and the functionality exists in the latest CPI release. Creds can also be a per-tenant secret. More info: https://github.com/kubernetes/cloud-provider-vsphere/pull/2413

It would be nice to have an way to pass the password as secret without needing to specify the whole config as secret.

CPI only stores the center creds and not the entire config. This is only true for CSI https://github.com/kubernetes-sigs/vsphere-csi-driver. Please take a look at that repo and possibly file an issue there if it doesn't meet your needs.

I can take a look at the CPI leader election and make sure it's working correctly.

MaxRink commented 4 years ago

hi @dvonthenen , The Link above is broken. I also think its not directly fitting our case, as our securitiy requirement is actually no network connectivity from the workload clusters to the vcenter. Running the CCM on the CAPI mgmt cluster works fine, as that is already the CPOM. Scoping ontop of that is an added bonus, but keeping the ccm external is basically an hard requirement. Its only used for providing providerIDs to nodes and for removing taints, as we dont use the CSI. My Issue is handeling that credentials in the first place, as "secret-name" and "secret-namespace" dont work for pulling in secrets and i havent found a way to pass them via env or in another mount. If i write them directly in the config, deploying it works just fine, but secret handeling becomes a pain as i need to seperate them from other objects for security reasons (currently we use SealedSecrets for that)

davidvonthenen commented 4 years ago

@MaxRink You should be able to use CCM installed on the management cluster only (In your case, you do not want to then install CCM on guest/workload clusters). To do that, create a tenant configuration/entry per tenant vSphere account and use vSphere Role Based Access to limit account permissions to just their resource pool/cluster.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/cloud-provider-vsphere/issues/356#issuecomment-752791577): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.