search

vmware-tanzu / tanzu-framework

Tanzu Framework provides a set of building blocks to build atop of the Tanzu platform and leverages Carvel packaging and plugins to provide users with a much stronger, more integrated experience than the loose coupling and stand-alone commands of the previous generation of tools.
Apache License 2.0
196 stars 193 forks source link

Antrea routable pod customization #1709

Closed vijaykatam closed 2 years ago

vijaykatam commented 2 years ago

AntreaConfig will be customized and pre-created when using routable pod feature. ClusterBootstrapController should be able to handle if a config already exists and only set ownerReferences.

Done criteria: Env test demonstrating a pre-created AntreaConfig is updated with ownerReferences

shivaani0505 commented 2 years ago

@vijaykatam , My Analysis:

  1. I see that NSXT_POD_ROUTING_ENABLED variable is available as a part of the Tanzu Cluster Configuration Variables excel sheet

  2. It is a configuration variable which can also be set in Tanzu Kubernetes cluster yaml ( Section: NSX-T Pod Routing in https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-tanzu-config-reference.html#common-variables-for-all-infrastructure-providers-0 )

  3. Further, if NSXT_POD_ROUTING_ENABLED is true, ANTREA_PROXY should be set to true (https://github.com/vmware-tanzu/tanzu-framework/blob/e8a083109169b54f6d1ca1c5df16019dfcff1d8f/pkg/v1/providers/ytt/02_addons/cni/antrea/antrea_addon_data.lib.yaml#L58) ( for vsphere only). Also, set the following 2 variables
trafficEncapMode: "noEncap" noSNAT: true

  4. NSXT_POD_ROUTING_ENABLED can be fetched from cluster.Spec.Topology.Variables and should be able to use as “tkg.tanzu.vmware.com/nsxt_pod_routing_enabled”. I saw some variables in excel sheet for cluster class are using this pattern => change to lower case have prefix as “tkg.tanzu.vmware.com/“ I have pinged @lzhan for confirmation.

shivaani0505 commented 2 years ago

@vijaykatam, some updates:

vijaykatam commented 2 years ago

Based on discussion with @tvs and @sidharthsurana cluster variables should be limited to topology and self contained with cluster and provider objects.

vijaykatam commented 2 years ago

Discussion with @lubronzhan and doc updated

  1. GCM controller is going to precreate AntreaConfig and VsphereCPIConfig with [cluster-name]-[package-name]-package
  2. ClusterBootstrap controller should not clone the configs again and should reuse whatever was created.
  3. Need to ensure that name of configs should not exceed 253 characters, currently we are using the full package name that has potential to result in name longer than 253 chars.
vijaykatam commented 2 years ago

I updated description to match latest requirement. This is being taken care in https://github.com/vmware-tanzu/tanzu-framework/pull/1881/files

lubronzhan commented 2 years ago

Good catch! Thanks Does that mean during the cluster/tkr upgrade process, GCM should also be responsible for updating the AntreaConfig to new version? So the GCM should watch the TKR associated with the cluster.

danniel1205 commented 2 years ago

This will be addressed by https://github.com/vmware-tanzu/tanzu-framework/pull/1881