search

kubernetes-sigs / cluster-api-provider-vsphere

Apache License 2.0
368 stars 292 forks source link

Support for NSX-T #506

Open akutz opened 5 years ago

akutz commented 5 years ago

/kind feature

Describe the solution you'd like CAPV should support NSX-T for:

tkrausjr commented 5 years ago

I have been giving this some thought before this was opened and I think while possible it will probably require a baseline NSX-T environment to be in place and installed on given vSphere Cluster nodes before CAPV can install and configure NCP components and Kubernetes clusters to support NSX-T. NSX-T in its current incarnation requires the following to install and I am not sure how feasible it would be to do the NSX-T Install as part of a CAPV cluster setup. (Full Install Instructions are here. https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/installation/GUID-3E0C4CEC-D593-4395-84C4-150CD6285963.html)

  1. Installation of Controllers / Managers which are Virtual Machines(Usually 3)

  2. Installation of Edge VMs for Ingress / Egress from NSX-T Logical Networks (2-10)

  3. Preparation of the ESXi Hosts in a given vSphere cluster with the NSX-T Kernel Modules and creation of a Tunnel End Point (TEP) Interface to Encap/Decap traffic which usually requires a reboot.

  4. Networking Infrastructure Requirements / Because of the Encapsulation overhead physical networking switches that will carry NSX-T Geneve Frames physical switches carrying VTEP traffic will need to have Jumbo frames enabled or MTU changed to at least 1600.

  5. Like other SDN solutions user will need to account for IPAM or plan addresses for several networks including Management, VTEP subnet, and Uplink interfaces on the Edge VMs for Ingress and Egress. This does not include the K8s Node and POD networks or the SNAT / VIP subnet for the Load Balancers.

yktsubo commented 5 years ago

Hi, I managed to install ncp as cni in workload cluster. In the next version, ncp doesn't require to install ovs and cni manually. But we still have some challenges here.

I can write up how to install ncp once next ncp is ready.

moshloop commented 5 years ago

/assign @timothysc Can you assist with some test infra for this?

timothysc commented 5 years ago

/unassign @timothysc
We will have federated CI signal at some point on PRs, but that would be @akutz.

tkrausjr commented 4 years ago

Hey @yktsubo , Are you saying you were able to setup and configure NCP on K8s as part of Guest Cluster deployment with CAPV or did you just post-install NCP / NSX-T on a CAPV deployed cluster ?

yktsubo commented 4 years ago

Sorry when I posted here, I thought it worked since I could see IP from kubectl get pod -o wide. However the network didn't work properly because no interface was configured on continer namespace.

This deployment was done on workload cluster deployed by CAPV. After workload cluster is deployed, I deployed ncp bootstrap containers on the cluster and tried to make it work. But I couldn't finish installation. I think because ncp doesn't work with containerd yet.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

moshloop commented 4 years ago

/remove-lifecycle stale

jayunit100 commented 4 years ago

There is renewed working going into this effort, not specifically around NCP though, but rather, around supporting

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

andrewrothstein commented 2 years ago

any action here?

lukasmrtvy commented 1 year ago

@yastij @akutz is there any chance to resurrect https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/722 ? Cloud provider for vSphere is supporting NSX-T loadbalancers, see https://github.com/kubernetes/cloud-provider-vsphere/blob/master/pkg/cloudprovider/vsphere/loadbalancer/README.md