unable to deploy management cluster from linux bootstrap machine to vSphere while using proxied network

[krachchh]

Bug Report

unable to deploy management cluster from linux bootstrap machine to vSphere while using proxied network, but deploying the cluster by using the unrestricted network works fine.

Cluster should have been deployed with proxies as well , since this is provided in the Tanzu UI.

Steps to Reproduce the Bug

1. Used Linux machine as bootstrap machine.
2. Opted for vSphere for the target machine, and followed the steps mentioned in https://tanzucommunityedition.io/docs/latest/vsphere-install-mgmt/
3. In Step 6: Kubernetes Network, opted for provate interface, through which dhcp server will provide the ip (which is getting provided successfully in the new node inside the vSphere)
4. In the same step, enabled the toggle button for proxies, and entered the subnet which is used by private interface as no_proxy, and configured http_proxy and https_proxy as per requirement.
5. Deployment is getting stuck in after the new node is getting created within the vSphere.

Environment Details

• Build version (tanzu version): tanzu version version: v0.2.1 buildDate: 2021-09-29 sha: ceaa474
• Deployment (Managed/Standalone cluster): Management Cluster
• Infrastructure Provider (Docker/AWS/Azure/vSphere): vSphere
• Operating System (bootstrap machine): Linux Ubuntu 18.04.6

• Operating System (kubernetes node) : Verified the same behaviour for Photon OS 3 as well as for Ubuntu 20.04

Tanzu log on UI : 
tanzu management-cluster create --ui

Validating the pre-requisites...
Serving kickstart UI at http://127.0.0.1:8080
Identity Provider not configured. Some authentication features won't work.
Validating configuration...
web socket connection established
sending pending 2 logs to UI
Using infrastructure provider vsphere:v0.7.10
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /home/krachchh/.kube-tkg/tmp/config_Q11REWS0
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Start creating management cluster...

Failure while deploying management cluster, Here are some steps to investigate the cause:

Debug:
    kubectl get po,deploy,cluster,kubeadmcontrolplane,machine,machinedeployment -A --kubeconfig /home/krachchh/.kube-tkg/tmp/config_Q11REWS0
    kubectl logs deployment.apps/<deployment-name> -n <deployment-namespace> manager --kubeconfig /home/krachchh/.kube-tkg/tmp/config_Q11REWS0

To clean up the resources created by the management cluster:
          tanzu management-cluster delete
unable to set up management cluster, : unable to wait for cluster and get the cluster kubeconfig: error waiting for cluster to be provisioned (this may take a few minutes): timed out waiting for cluster creation to complete: cluster control plane is still being initialized

Debug logs :

krachchh@bootstrap-tanzu-ubuntu:~$ kubectl get po,deploy,cluster,kubeadmcontrolplane,machine,machinedeployment -A --kubeconfig /home/krachchh/.kube-tkg/tmp/config_Q11REWS0
NAMESPACE                           NAME                                                                      READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system       pod/capi-kubeadm-bootstrap-controller-manager-6494884869-qnclv            2/2     Running   0          69m
capi-kubeadm-control-plane-system   pod/capi-kubeadm-control-plane-controller-manager-857d687b9d-tfsjb        2/2     Running   0          69m
capi-system                         pod/capi-controller-manager-778bd4dfb9-28cc8                              2/2     Running   0          69m
capi-webhook-system                 pod/capi-controller-manager-9995bdc94-km6nm                               2/2     Running   0          69m
capi-webhook-system                 pod/capi-kubeadm-bootstrap-controller-manager-68845b65f8-4bt5d            2/2     Running   0          69m
capi-webhook-system                 pod/capi-kubeadm-control-plane-controller-manager-9847c6747-zs5hz         2/2     Running   0          69m
capi-webhook-system                 pod/capv-controller-manager-55bf67fbd5-lmvm2                              2/2     Running   0          69m
capv-system                         pod/capv-controller-manager-5c77b7f5f7-7qr4k                              2/2     Running   0          69m
cert-manager                        pod/cert-manager-77f6fb8fd5-8kgjg                                         1/1     Running   0          74m
cert-manager                        pod/cert-manager-cainjector-6bd4cff7bb-clcjp                              1/1     Running   0          74m
cert-manager                        pod/cert-manager-webhook-fbfcb9d6c-jxzrc                                  1/1     Running   0          74m
kube-system                         pod/coredns-8dcb5c56b-dzmwq                                               1/1     Running   0          74m
kube-system                         pod/coredns-8dcb5c56b-hq2m7                                               1/1     Running   0          74m
kube-system                         pod/etcd-tkg-kind-c6b39fspfse13sms6upg-control-plane                      1/1     Running   0          74m
kube-system                         pod/kindnet-rqfj8                                                         1/1     Running   0          74m
kube-system                         pod/kube-apiserver-tkg-kind-c6b39fspfse13sms6upg-control-plane            1/1     Running   0          74m
kube-system                         pod/kube-controller-manager-tkg-kind-c6b39fspfse13sms6upg-control-plane   1/1     Running   0          74m
kube-system                         pod/kube-proxy-hqc2h                                                      1/1     Running   0          74m
kube-system                         pod/kube-scheduler-tkg-kind-c6b39fspfse13sms6upg-control-plane            1/1     Running   0          74m
local-path-storage                  pod/local-path-provisioner-8b46957d4-sxlz4                                1/1     Running   0          74m

NAMESPACE                           NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
capi-kubeadm-bootstrap-system       deployment.apps/capi-kubeadm-bootstrap-controller-manager       1/1     1            1           69m
capi-kubeadm-control-plane-system   deployment.apps/capi-kubeadm-control-plane-controller-manager   1/1     1            1           69m
capi-system                         deployment.apps/capi-controller-manager                         1/1     1            1           69m
capi-webhook-system                 deployment.apps/capi-controller-manager                         1/1     1            1           69m
capi-webhook-system                 deployment.apps/capi-kubeadm-bootstrap-controller-manager       1/1     1            1           69m
capi-webhook-system                 deployment.apps/capi-kubeadm-control-plane-controller-manager   1/1     1            1           69m
capi-webhook-system                 deployment.apps/capv-controller-manager                         1/1     1            1           69m
capv-system                         deployment.apps/capv-controller-manager                         1/1     1            1           69m
cert-manager                        deployment.apps/cert-manager                                    1/1     1            1           74m
cert-manager                        deployment.apps/cert-manager-cainjector                         1/1     1            1           74m
cert-manager                        deployment.apps/cert-manager-webhook                            1/1     1            1           74m
kube-system                         deployment.apps/coredns                                         2/2     2            2           74m
local-path-storage                  deployment.apps/local-path-provisioner                          1/1     1            1           74m

NAMESPACE    NAME                                PHASE
tkg-system   cluster.cluster.x-k8s.io/nrftanzu   Provisioned

NAMESPACE    NAME                                                                       INITIALIZED   API SERVER AVAILABLE   VERSION            REPLICAS   READY   UPDATED   UNAVAILABLE
tkg-system   kubeadmcontrolplane.controlplane.cluster.x-k8s.io/nrftanzu-control-plane                                        v1.21.2+vmware.1   1                  1         1

NAMESPACE    NAME                                                     PROVIDERID                                       PHASE          VERSION
tkg-system   machine.cluster.x-k8s.io/nrftanzu-control-plane-67d2t    vsphere://4235fb1a-53e6-976f-cb9c-e67c984aa371   Provisioning   v1.21.2+vmware.1
tkg-system   machine.cluster.x-k8s.io/nrftanzu-md-0-946b54d78-thppd                                                    Pending        v1.21.2+vmware.1

NAMESPACE    NAME                                               PHASE       REPLICAS   READY   UPDATED   UNAVAILABLE
tkg-system   machinedeployment.cluster.x-k8s.io/nrftanzu-md-0   ScalingUp   1                  1         1

Logs of capv-controller-manager pod of capv-system namespace :

I1118 11:24:26.144198       1 main.go:221] entrypoint "msg"="creating controller manager"  "version"=""
I1118 11:24:27.609456       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"="127.0.0.1:8080"
I1118 11:24:27.614758       1 main.go:232] entrypoint "msg"="starting controller manager"  
I1118 11:24:27.614964       1 leaderelection.go:242] attempting to acquire leader lease  capv-system/capv-controller-manager-runtime...
I1118 11:24:27.615134       1 internal.go:356] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I1118 11:24:27.629818       1 leaderelection.go:252] successfully acquired lease capv-system/capv-controller-manager-runtime
I1118 11:24:27.630077       1 recorder.go:52] controller-runtime/manager/events "msg"="Normal"  "message"="capv-controller-manager-5c77b7f5f7-7qr4k_e04e66c2-2b14-4aba-9cc9-3911c08cee7e became leader" "object"={"kind":"ConfigMap","namespace":"capv-system","name":"capv-controller-manager-runtime","uid":"4b925002-e117-4412-8be9-05140d47a264","apiVersion":"v1","resourceVersion":"3042"} "reason"="LeaderElection"
I1118 11:24:27.630485       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheredeploymentzone" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"placementConstraint":{}},"status":{}}}
I1118 11:24:27.630583       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherevm" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"template":"","network":{"devices":null}},"status":{}}}
I1118 11:24:27.630712       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"template":"","network":{"devices":null}},"status":{"ready":false}}}
I1118 11:24:27.630905       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="haproxyloadbalancer" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"virtualMachineConfiguration":{"template":"","network":{"devices":null}}},"status":{}}}
I1118 11:24:27.631452       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherecluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"cloudProviderConfiguration":{"global":{},"network":{},"disk":{},"workspace":{},"labels":{},"providerConfig":{}},"controlPlaneEndpoint":{"host":"","port":0}},"status":{}}}
I1118 11:24:27.633466       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vsphereclusteridentity" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{}}}
I1118 11:24:27.731235       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheredeploymentzone" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"region":{"name":"","type":"","tagCategory":""},"zone":{"name":"","type":"","tagCategory":""},"topology":{"datacenter":""}}}}
I1118 11:24:27.731470       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"template":"","network":{"devices":null}},"status":{}}}
I1118 11:24:27.731593       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="haproxyloadbalancer" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"template":"","network":{"devices":null}},"status":{}}}
I1118 11:24:27.732048       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherevm" "source"=
I1118 11:24:27.732144       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I1118 11:24:27.732278       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="haproxyloadbalancer" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I1118 11:24:27.732336       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherecluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I1118 11:24:27.732551       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherevm" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I1118 11:24:27.734423       1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="vsphereclusteridentity" 
I1118 11:24:27.831868       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheredeploymentzone" "source"=
I1118 11:24:27.831968       1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="vspheredeploymentzone" 
I1118 11:24:27.832072       1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="vspheredeploymentzone" "worker count"=10
I1118 11:24:27.832528       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="haproxyloadbalancer" "source"=
I1118 11:24:27.832642       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheremachine" "source"=
I1118 11:24:27.832765       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="haproxyloadbalancer" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I1118 11:24:27.832845       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherecluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"template":"","network":{"devices":null}},"status":{"ready":false}}}
I1118 11:24:27.832782       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspheremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I1118 11:24:27.832871       1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="haproxyloadbalancer" 
I1118 11:24:27.832901       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherecluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"virtualMachineConfiguration":{"template":"","network":{"devices":null}}},"status":{}}}
I1118 11:24:27.832950       1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="vspherevm" 
I1118 11:24:27.832931       1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="vspheremachine" 
I1118 11:24:27.832997       1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="haproxyloadbalancer" "worker count"=10
I1118 11:24:27.833039       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherecluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"placementConstraint":{}},"status":{}}}
I1118 11:24:27.833070       1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="vspheremachine" "worker count"=10
I1118 11:24:27.833106       1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="vspherevm" "worker count"=10
I1118 11:24:27.833144       1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="vspherecluster" "source"=
I1118 11:24:27.833233       1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="vspherecluster" 
I1118 11:24:27.833323       1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="vspherecluster" "worker count"=10
I1118 11:24:27.834627       1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="vsphereclusteridentity" "worker count"=10
I1118 11:24:31.562995       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.578279       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.581236       1 vspherecluster_controller.go:162] capv-controller-manager/vspherecluster-controller "msg"="Waiting for Cluster Controller to set OwnerRef on VSphereCluster"  
I1118 11:24:31.581273       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.583396       1 vspherecluster_controller.go:162] capv-controller-manager/vspherecluster-controller "msg"="Waiting for Cluster Controller to set OwnerRef on VSphereCluster"  
I1118 11:24:31.583431       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.599496       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
E1118 11:24:31.729933       1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="Secret \"nrftanzu\" not found" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.730403       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.734027       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.734199       1 vspherecluster_controller.go:651] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="start polling API server for online check"  
I1118 11:24:31.759618       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.759889       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.762713       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.762749       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.769986       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.770298       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.772806       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.772830       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.773620       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.859650       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.863082       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.863114       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.863635       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.876271       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:31.876320       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:31.899847       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:31.899911       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:31.906605       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.910639       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.910834       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.911640       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.913958       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:31.913984       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:31.918990       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.921521       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:31.921557       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:31.924543       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.924595       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.925165       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.934228       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.940986       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.941013       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.941527       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:31.971166       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:31.973963       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:31.973996       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:31.974491       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:32.052289       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:32.056613       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:32.056642       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:32.057085       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:32.066137       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:24:32.102612       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:32.102849       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:24:32.103262       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:32.173507       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:24:32.173999       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:32.180479       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:24:32.181318       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:32.212217       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:24:32.213584       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:24:33.518309       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:33.518343       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.536335       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:33.536374       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.547067       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:33.547139       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.552530       1 vspheremachine_controller.go:162] capv-controller-manager/vspheremachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on VSphereMachine"  
I1118 11:24:33.552557       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.591142       1 vspheremachine_controller.go:333] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="Waiting for bootstrap data to be available"  
I1118 11:24:33.614234       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.614467       1 vspheremachine_controller.go:333] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="Waiting for bootstrap data to be available"  
I1118 11:24:33.614859       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.629527       1 vspheremachine_controller.go:333] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="Waiting for bootstrap data to be available"  
I1118 11:24:33.630148       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.661384       1 vspheremachine_controller.go:333] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="Waiting for bootstrap data to be available"  
I1118 11:24:33.661911       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.688080       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="/, Kind=" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:33.688102       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:33.688830       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.689484       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:33.689504       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:33.689931       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:33.694126       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:33.694158       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:33.695404       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:35.909168       1 session.go:106] capv-controller-manager/vspherevm-controller "msg"="cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:35.931627       1 util.go:84] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="using inventory path to find vm"  "path"="/WPSDatacenter/vm/10.122.122.113/nrftanzu-control-plane-67d2t"
I1118 11:24:35.952299       1 clone.go:52] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/vcenter "msg"="starting clone process"  
I1118 11:24:35.952344       1 clone.go:56] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/vcenter "msg"="applied bootstrap data to VM clone spec"  
I1118 11:24:36.026604       1 clone.go:247] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/vcenter "msg"="cloning machine"  "cloneType"="fullClone" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:36.058282       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:36.058341       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:36.059639       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:36.071417       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:36.071466       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:36.072010       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:36.078230       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:36.078262       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:36.078684       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:36.080602       1 util.go:222] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="enqueuing reconcile request on task completion"  "task-description-id"="VirtualMachine.clone" "task-entity-name"="ubuntu-2004-kube-v1.21.2+vmware.1" "task-name"="CloneVM_Task" "task-ref"={"Type":"Task","Value":"task-54096"}
I1118 11:24:36.080659       1 vspherevm_controller.go:366] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VM state is not reconciled"  "actual-vm-state"="pending" "expected-vm-state"="ready"
I1118 11:24:36.088531       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:36.088600       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:36.105657       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:36.106171       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3230" "remote-resource-version"="3252"
I1118 11:24:36.106208       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:36.106230       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:36.106260       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:36.106699       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:36.116047       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:36.116071       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:36.116494       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:36.126564       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:36.130185       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54096 "msg"="task found"  "description-id"="VirtualMachine.clone" "state"="running"
I1118 11:24:36.130216       1 util.go:133] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54096 "msg"="task is still running"  "description-id"="VirtualMachine.clone"
I1118 11:24:36.130232       1 vspherevm_controller.go:366] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VM state is not reconciled"  "actual-vm-state"="pending" "expected-vm-state"="ready"
I1118 11:24:36.130991       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3252" "remote-resource-version"="3252"
I1118 11:24:36.131029       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:36.659723       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:36.663057       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:36.663094       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:39.670059       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:45.952707       1 util.go:267] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="triggering GenericEvent"  "reason"="task" "task-description-id"="VirtualMachine.clone" "task-entity-name"="ubuntu-2004-kube-v1.21.2+vmware.1" "task-name"="CloneVM_Task" "task-ref"={"Type":"Task","Value":"task-54096"} "task-state"="success"
I1118 11:24:45.955737       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:45.958819       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54096 "msg"="task found"  "description-id"="VirtualMachine.clone" "state"="success"
I1118 11:24:45.958872       1 util.go:136] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54096 "msg"="task is a success"  "description-id"="VirtualMachine.clone"
I1118 11:24:45.961856       1 util.go:95] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by instance uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:24:45.974946       1 service.go:247] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="updating metadata"  
I1118 11:24:45.988468       1 service.go:254] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="wait for VM metadata to be updated"  
I1118 11:24:45.990633       1 util.go:222] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="enqueuing reconcile request on task completion"  "task-description-id"="VirtualMachine.reconfigure" "task-entity-name"="nrftanzu-control-plane-67d2t" "task-name"="ReconfigVM_Task" "task-ref"={"Type":"Task","Value":"task-54097"}
I1118 11:24:45.990725       1 vspherevm_controller.go:366] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VM state is not reconciled"  "actual-vm-state"="pending" "expected-vm-state"="ready"
I1118 11:24:46.003391       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3252" "remote-resource-version"="3306"
I1118 11:24:46.003490       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:46.003683       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.003705       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.004192       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.006494       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:46.008719       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54097 "msg"="task found"  "description-id"="VirtualMachine.reconfigure" "state"="running"
I1118 11:24:46.008748       1 util.go:133] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54097 "msg"="task is still running"  "description-id"="VirtualMachine.reconfigure"
I1118 11:24:46.008765       1 vspherevm_controller.go:366] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VM state is not reconciled"  "actual-vm-state"="pending" "expected-vm-state"="ready"
I1118 11:24:46.009434       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3306" "remote-resource-version"="3306"
I1118 11:24:46.009464       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:46.192079       1 util.go:267] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="triggering GenericEvent"  "reason"="task" "task-description-id"="VirtualMachine.reconfigure" "task-entity-name"="nrftanzu-control-plane-67d2t" "task-name"="ReconfigVM_Task" "task-ref"={"Type":"Task","Value":"task-54097"} "task-state"="success"
I1118 11:24:46.194768       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:46.197150       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54097 "msg"="task found"  "description-id"="VirtualMachine.reconfigure" "state"="success"
I1118 11:24:46.197183       1 util.go:136] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54097 "msg"="task is a success"  "description-id"="VirtualMachine.reconfigure"
I1118 11:24:46.199922       1 util.go:95] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by instance uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:24:46.210656       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:24:46.210760       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:24:46.212954       1 service.go:265] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powering on"  
I1118 11:24:46.233356       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.233416       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.234304       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.240186       1 service.go:284] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="wait for VM to be powered on"  
I1118 11:24:46.242313       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.242368       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.243396       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.245261       1 util.go:222] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="enqueuing reconcile request on task completion"  "task-description-id"="VirtualMachine.powerOn" "task-entity-name"="nrftanzu-control-plane-67d2t" "task-name"="PowerOnVM_Task" "task-ref"={"Type":"Task","Value":"task-54098"}
I1118 11:24:46.245349       1 vspherevm_controller.go:366] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VM state is not reconciled"  "actual-vm-state"="pending" "expected-vm-state"="ready"
I1118 11:24:46.254531       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.254572       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.262262       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3306" "remote-resource-version"="3309"
I1118 11:24:46.262303       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:46.271671       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.272432       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.272458       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.272989       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.282653       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:46.282673       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.282694       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.283171       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.288351       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54098 "msg"="task found"  "description-id"="VirtualMachine.powerOn" "state"="running"
I1118 11:24:46.288406       1 util.go:133] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54098 "msg"="task is still running"  "description-id"="VirtualMachine.powerOn"
I1118 11:24:46.288435       1 vspherevm_controller.go:366] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VM state is not reconciled"  "actual-vm-state"="pending" "expected-vm-state"="ready"
I1118 11:24:46.290483       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3309" "remote-resource-version"="3309"
I1118 11:24:46.290541       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:24:46.673648       1 util.go:267] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="triggering GenericEvent"  "reason"="task" "task-description-id"="VirtualMachine.powerOn" "task-entity-name"="nrftanzu-control-plane-67d2t" "task-name"="PowerOnVM_Task" "task-ref"={"Type":"Task","Value":"task-54098"} "task-state"="success"
I1118 11:24:46.676653       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:46.679208       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54098 "msg"="task found"  "description-id"="VirtualMachine.powerOn" "state"="success"
I1118 11:24:46.679265       1 util.go:136] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54098 "msg"="task is a success"  "description-id"="VirtualMachine.powerOn"
I1118 11:24:46.681952       1 util.go:95] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by instance uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:24:46.694931       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:24:46.694983       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:24:46.696369       1 util.go:516] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="the VM is missing the requested IP address"  "addressType"="dhcp4"
I1118 11:24:46.697267       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:24:46.697324       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:24:46.718259       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.718317       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.719197       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.724526       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3309" "remote-resource-version"="3312"
I1118 11:24:46.726605       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:24:46.726800       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:24:46.727096       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:46.728315       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:24:46.729665       1 util.go:127] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54098 "msg"="task found"  "description-id"="VirtualMachine.powerOn" "state"="success"
I1118 11:24:46.729714       1 util.go:136] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t/task-54098 "msg"="task is a success"  "description-id"="VirtualMachine.powerOn"
I1118 11:24:46.738403       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:24:46.751330       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:24:46.751370       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:24:46.753611       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:24:46.753665       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:24:46.765398       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3312" "remote-resource-version"="3313"
I1118 11:24:46.767746       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:46.771540       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:24:46.782789       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:24:46.782835       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:24:46.785033       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:24:46.785086       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:24:46.786603       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3313" "remote-resource-version"="3313"
I1118 11:24:47.126799       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:24:47.133999       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:24:47.134229       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:24:48.886650       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:24:56.727906       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:24:56.731833       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:24:56.744260       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:24:56.744311       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:24:56.746471       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:24:56.746530       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:24:56.748083       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3313" "remote-resource-version"="3313"
I1118 11:25:06.752435       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:06.755751       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:06.769175       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:06.769226       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:06.771901       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:06.771956       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:06.774124       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3313" "remote-resource-version"="3313"
I1118 11:25:16.777599       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:16.787062       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:16.797622       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:16.797669       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:16.799651       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:16.799709       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:16.801245       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3313" "remote-resource-version"="3313"
I1118 11:25:26.804748       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:26.809131       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:26.820375       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:26.820402       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:26.822994       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:26.823125       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:26.826898       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3313" "remote-resource-version"="3313"
I1118 11:25:27.928392       1 util.go:475] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="discovered IP address"  "addressType"="dhcp4" "addressValue"="10.14.7.82"
I1118 11:25:27.928506       1 util.go:438] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="ignoring IP address"  "reason"="failed to validate ip addr=fe80::250:56ff:feb5:9ff6: link-local-unicast"
I1118 11:25:27.928534       1 util.go:546] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="the VM has all of the requested IP addresses"  
I1118 11:25:27.928668       1 util.go:301] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="triggering GenericEvent"  "ipAddress"="10.14.7.82" "reason"="network"
I1118 11:25:27.931220       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:27.937618       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:27.948239       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:27.948291       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:27.950279       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:27.950335       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:27.950363       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 11:25:27.974376       1 vspheremachine_controller.go:668] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="status.ready not found"  "vmGVK"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM" "vmName"="nrftanzu-control-plane-67d2t" "vmNamespace"="tkg-system"
I1118 11:25:27.974494       1 vspheremachine_controller.go:365] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="waiting for ready state"  
I1118 11:25:27.984106       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3313" "remote-resource-version"="3494"
I1118 11:25:27.984229       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:25:27.988670       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:27.992277       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:28.002730       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:25:28.004760       1 vspheremachine_controller.go:651] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-control-plane-qnq7t "msg"="updated provider ID"  "provider-id"="vsphere://4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:28.006396       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:28.006441       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:28.008958       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:28.008994       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:28.009010       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 11:25:28.022358       1 vspherevm_controller.go:269] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource is patched"  "local-resource-version"="3494" "remote-resource-version"="3495"
I1118 11:25:28.022424       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:25:28.025055       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:28.028417       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:25:28.029567       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:28.030919       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:25:28.045741       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:28.045785       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:28.048206       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:28.048259       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:28.048288       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 11:25:28.050565       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 11:25:28.050620       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:25:31.996813       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:25:32.000242       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:25:32.000329       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:25:34.970170       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:25:34.970695       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:25:34.982600       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:25:34.982675       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:25:36.830629       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:25:36.835080       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:25:36.850291       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:25:36.850317       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:25:36.852634       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:25:36.852664       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:25:36.852680       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 11:25:36.853622       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 11:25:36.853661       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:25:38.038219       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:25:38.114939       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:25:38.122712       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:25:38.122774       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:25:41.114083       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:25:47.124321       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:25:47.131314       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:25:47.131378       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:25:50.326066       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:25:59.573343       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:25:59.585902       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:42:44.869018       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:42:44.869866       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:42:44.870887       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:43:40.544626       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:43:40.545640       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:43:40.548878       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:44:18.618161       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:44:18.634193       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:44:18.634232       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:44:19.798255       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:44:19.798851       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:44:19.808557       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:44:19.808653       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:44:22.871613       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:44:34.935277       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:44:35.579897       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:44:35.583381       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:44:35.598963       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:44:35.599035       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:44:35.601136       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:44:35.601181       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:44:35.601209       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 11:44:35.602979       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 11:44:35.603032       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 11:51:53.436979       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:51:53.438441       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:51:53.441284       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:53:16.995764       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 11:53:16.996830       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 11:53:16.997532       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:54:14.058804       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:54:14.066906       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:54:14.066970       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:54:16.950185       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:54:16.950601       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 11:54:16.962249       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 11:54:16.962312       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 11:54:20.022091       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 11:54:38.578549       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 11:54:39.838504       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 11:54:39.842278       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 11:54:39.856699       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 11:54:39.856728       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 11:54:39.859340       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 11:54:39.859385       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 11:54:39.859413       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 11:54:39.862296       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 11:54:39.863173       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 12:01:02.002070       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:01:02.002592       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:01:02.003986       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:02:53.446738       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:02:53.447571       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:02:53.450687       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:04:09.499626       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 12:04:09.511813       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 12:04:09.511873       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 12:04:11.798026       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 12:04:11.798427       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 12:04:11.805231       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 12:04:11.805291       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 12:04:14.874074       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 12:04:42.222822       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:04:43.170251       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 12:04:43.173618       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 12:04:43.186020       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 12:04:43.186048       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 12:04:43.188527       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 12:04:43.188584       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 12:04:43.188602       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 12:04:43.190015       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 12:04:43.190052       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 12:10:10.569074       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:10:10.569923       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:10:10.570156       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:12:29.897529       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:12:29.898396       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:12:29.899292       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:14:04.940423       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 12:14:04.954115       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 12:14:04.954180       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 12:14:06.038037       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 12:14:07.241423       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 12:14:07.252844       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 12:14:07.252887       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 12:14:09.110161       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 12:14:45.867036       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:14:46.832460       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 12:14:46.835644       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 12:14:46.850140       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 12:14:46.850192       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 12:14:46.853378       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 12:14:46.853445       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 12:14:46.853473       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 12:14:46.855193       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 12:14:46.855254       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 12:19:19.135556       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:19:19.136477       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:19:19.137228       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:22:06.348373       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:22:06.349205       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:22:06.350239       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:24:00.380963       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 12:24:00.386877       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 12:24:00.386978       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 12:24:00.789875       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 12:24:03.160955       1 vspherecluster_controller.go:285] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="Reconciling VSphereCluster"  
I1118 12:24:03.172016       1 vspherecluster_controller.go:445] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping load balancer reconciliation"  "reason"="VSphereCluster.Spec.LoadBalancerRef is nil"
I1118 12:24:03.172081       1 vspherecluster_controller.go:644] capv-controller-manager/vspherecluster-controller/tkg-system/nrftanzu "msg"="skipping reconcile when API server is online"  "reason"="alreadyPolling"
I1118 12:24:04.790570       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherecluster" "name"="nrftanzu" "namespace"="tkg-system"
I1118 12:24:49.510141       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"
I1118 12:24:50.762677       1 session.go:63] capv-controller-manager/vspherevm-controller "msg"="found active cached vSphere client session"  "datacenter"="/WPSDatacenter" "server"="rtp-policy-vc-67.cisco.com"
I1118 12:24:50.766314       1 util.go:68] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm found by bios uuid"  "vmref"={"Type":"VirtualMachine","Value":"vm-5707"}
I1118 12:24:50.781999       1 service.go:296] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="storage policy not defined. skipping reconcile storage policy"  
I1118 12:24:50.782053       1 service.go:495] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="hosts topology in failure domain not defined. skipping reconcile VM group"  
I1118 12:24:50.784822       1 service.go:287] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="powered on"  
I1118 12:24:50.784879       1 vspherevm_controller.go:374] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="vm bios-uuid"  "biosuuid"="4235fb1a-53e6-976f-cb9c-e67c984aa371"
I1118 12:24:50.784906       1 vspherevm_controller.go:394] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="VSphereVM is ready"  
I1118 12:24:50.786568       1 vspherevm_controller.go:280] capv-controller-manager/vspherevm-controller/tkg-system/nrftanzu-control-plane-67d2t "msg"="resource patch was not required"  "local-resource-version"="3495" "remote-resource-version"="3495"
I1118 12:24:50.786617       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspherevm" "name"="nrftanzu-control-plane-67d2t" "namespace"="tkg-system"
I1118 12:28:27.702155       1 vspheremachine_controller.go:329] capv-controller-manager/vspheremachine-controller/tkg-system/nrftanzu-worker-lrlfb "msg"="Waiting for the control plane to be initialized"  
I1118 12:28:27.702965       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-worker-lrlfb" "namespace"="tkg-system"
I1118 12:28:27.704006       1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="vspheremachine" "name"="nrftanzu-control-plane-qnq7t" "namespace"="tkg-system"

[github-actions[bot]]

Hey @krachchh! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Tanzu Community Edition.

[krachchh]

I would also like to mention that Tanzu needs to provide some better errors in case something fails like this so that user can debug.

Also, would like to mention that workaround is to use an unrestricted interface without any proxy, but in my case, that's not an option.

[stmcginnis]

Thanks for the report! This part of the process happens in the code from the tanzu-framework repo. Transferring issue there to get more eyes on it.

[snehalbiche]

do we at least have a workaround for this issue? how long it will take to provide a workaround or resolution to this issue?

[snehalbiche]

attaching the tanzu diagnostic logs. bootstrap.tkg-kind-c6b39fspfse13sms6upg.diagnostics.tar.gz