kubernetes-retired / cluster-api-provider-nested

Cluster API Provider for Nested Clusters
Apache License 2.0
301 stars 67 forks source link

Nested cluster not creating #197

Closed sergeimonakhov closed 2 years ago

sergeimonakhov commented 3 years ago

What steps did you take and what happened: Hi! I followed this instruction: https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/docs but nested cluster not creating

logs capi-kubeadm-bootstrap-controller-manager:

I0723 18:52:26.418852       1 request.go:668] Waited for 1.037262934s due to client-side throttling, not priority and fairness, request: GET:https://10.233.0.1:443/apis/ceph.rook.io/v1?timeout=32s
I0723 18:52:30.579387       1 deleg.go:130] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"="localhost:8080"
I0723 18:52:30.580877       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfig"}
I0723 18:52:30.580956       1 deleg.go:130] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfig"} "path"="/validate-bootstrap-cluster-x-k8s-io-v1alpha4-kubeadmconfig"
I0723 18:52:30.581123       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-bootstrap-cluster-x-k8s-io-v1alpha4-kubeadmconfig"
I0723 18:52:30.581356       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/convert"
I0723 18:52:30.582717       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{},"status":{}}
I0723 18:52:30.582771       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfigList"}
I0723 18:52:30.582812       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a validating webhook, admission.Validator interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfigList"}
I0723 18:52:30.582954       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{},"items":null}
I0723 18:52:30.583011       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfigTemplate"}
I0723 18:52:30.583042       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a validating webhook, admission.Validator interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfigTemplate"}
I0723 18:52:30.583200       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{"template":{"spec":{}}}}
I0723 18:52:30.583263       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfigTemplateList"}
I0723 18:52:30.583313       1 deleg.go:130] controller-runtime/builder "msg"="skip registering a validating webhook, admission.Validator interface is not implemented"  "GVK"={"Group":"bootstrap.cluster.x-k8s.io","Version":"v1alpha4","Kind":"KubeadmConfigTemplateList"}
I0723 18:52:30.583457       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{},"items":null}
I0723 18:52:30.583607       1 deleg.go:130] setup "msg"="starting manager"  "version"=""
I0723 18:52:30.583719       1 leaderelection.go:243] attempting to acquire leader lease capi-kubeadm-bootstrap-system/kubeadm-bootstrap-manager-leader-election-capi...
I0723 18:52:30.583861       1 server.go:208] controller-runtime/webhook/webhooks "msg"="starting webhook server"
I0723 18:52:30.583938       1 internal.go:384] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0723 18:52:30.584619       1 deleg.go:130] controller-runtime/certwatcher "msg"="Updated current TLS certificate"
I0723 18:52:30.584864       1 deleg.go:130] controller-runtime/webhook "msg"="serving webhook server"  "host"="" "port"=9443
I0723 18:52:30.584939       1 deleg.go:130] controller-runtime/certwatcher "msg"="Starting certificate watcher"
I0723 18:52:30.600241       1 leaderelection.go:253] successfully acquired lease capi-kubeadm-bootstrap-system/kubeadm-bootstrap-manager-leader-election-capi
I0723 18:52:30.600797       1 controller.go:165] controller-runtime/manager/controller/kubeadmconfig "msg"="Starting EventSource" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{}}}
I0723 18:52:30.601151       1 controller.go:165] controller-runtime/manager/controller/kubeadmconfig "msg"="Starting EventSource" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0723 18:52:30.601476       1 controller.go:165] controller-runtime/manager/controller/kubeadmconfig "msg"="Starting EventSource" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:30.601537       1 controller.go:173] controller-runtime/manager/controller/kubeadmconfig "msg"="Starting Controller" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig"
I0723 18:52:30.902890       1 controller.go:207] controller-runtime/manager/controller/kubeadmconfig "msg"="Starting workers" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig" "worker count"=10

capi-controller-manager:

I0723 18:52:18.070621       1 request.go:668] Waited for 1.0307624s due to client-side throttling, not priority and fairness, request: GET:https://10.233.0.1:443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s
I0723 18:52:22.132149       1 deleg.go:130] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"="localhost:8080"
I0723 18:52:22.167333       1 deleg.go:130] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"Cluster"} "path"="/mutate-cluster-x-k8s-io-v1alpha4-cluster"
I0723 18:52:22.167488       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/mutate-cluster-x-k8s-io-v1alpha4-cluster"
I0723 18:52:22.167618       1 deleg.go:130] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"Cluster"} "path"="/validate-cluster-x-k8s-io-v1alpha4-cluster"
I0723 18:52:22.167739       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-cluster-x-k8s-io-v1alpha4-cluster"
I0723 18:52:22.168004       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/convert"
I0723 18:52:22.168687       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}
I0723 18:52:22.168743       1 deleg.go:130] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"Machine"} "path"="/mutate-cluster-x-k8s-io-v1alpha4-machine"
I0723 18:52:22.168843       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/mutate-cluster-x-k8s-io-v1alpha4-machine"
I0723 18:52:22.168931       1 deleg.go:130] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"Machine"} "path"="/validate-cluster-x-k8s-io-v1alpha4-machine"
I0723 18:52:22.169028       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-cluster-x-k8s-io-v1alpha4-machine"
I0723 18:52:22.169360       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}
I0723 18:52:22.169417       1 deleg.go:130] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"MachineSet"} "path"="/mutate-cluster-x-k8s-io-v1alpha4-machineset"
I0723 18:52:22.169517       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/mutate-cluster-x-k8s-io-v1alpha4-machineset"
I0723 18:52:22.169606       1 deleg.go:130] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"MachineSet"} "path"="/validate-cluster-x-k8s-io-v1alpha4-machineset"
I0723 18:52:22.169703       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-cluster-x-k8s-io-v1alpha4-machineset"
I0723 18:52:22.170095       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}
I0723 18:52:22.170162       1 deleg.go:130] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"MachineDeployment"} "path"="/mutate-cluster-x-k8s-io-v1alpha4-machinedeployment"
I0723 18:52:22.170282       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/mutate-cluster-x-k8s-io-v1alpha4-machinedeployment"
I0723 18:52:22.170377       1 deleg.go:130] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"MachineDeployment"} "path"="/validate-cluster-x-k8s-io-v1alpha4-machinedeployment"
I0723 18:52:22.170480       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-cluster-x-k8s-io-v1alpha4-machinedeployment"
I0723 18:52:22.170782       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}
I0723 18:52:22.170840       1 deleg.go:130] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"MachineHealthCheck"} "path"="/mutate-cluster-x-k8s-io-v1alpha4-machinehealthcheck"
I0723 18:52:22.170936       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/mutate-cluster-x-k8s-io-v1alpha4-machinehealthcheck"
I0723 18:52:22.171022       1 deleg.go:130] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha4","Kind":"MachineHealthCheck"} "path"="/validate-cluster-x-k8s-io-v1alpha4-machinehealthcheck"
I0723 18:52:22.171117       1 server.go:140] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-cluster-x-k8s-io-v1alpha4-machinehealthcheck"
I0723 18:52:22.171402       1 deleg.go:130] controller-runtime/builder "msg"="conversion webhook enabled"  "object"={"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"unhealthyConditions":null},"status":{}}
I0723 18:52:22.171443       1 deleg.go:130] setup "msg"="starting manager"  "version"=""
I0723 18:52:22.171518       1 leaderelection.go:243] attempting to acquire leader lease capi-system/controller-leader-election-capi...
I0723 18:52:22.171657       1 server.go:208] controller-runtime/webhook/webhooks "msg"="starting webhook server"
I0723 18:52:22.172207       1 deleg.go:130] controller-runtime/certwatcher "msg"="Updated current TLS certificate"
I0723 18:52:22.172393       1 deleg.go:130] controller-runtime/webhook "msg"="serving webhook server"  "host"="" "port"=9443
I0723 18:52:22.172531       1 deleg.go:130] controller-runtime/certwatcher "msg"="Starting certificate watcher"
I0723 18:52:22.186135       1 leaderelection.go:253] successfully acquired lease capi-system/controller-leader-election-capi
I0723 18:52:22.372735       1 internal.go:384] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0723 18:52:22.373050       1 controller.go:165] controller-runtime/manager/controller/cluster "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:22.373056       1 controller.go:165] controller-runtime/manager/controller/machinehealthcheck "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineHealthCheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"unhealthyConditions":null},"status":{}}}
I0723 18:52:22.373113       1 controller.go:173] controller-runtime/manager/controller/cluster "msg"="Starting Controller" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"
I0723 18:52:22.373295       1 controller.go:165] controller-runtime/manager/controller/cluster "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:22.373169       1 controller.go:165] controller-runtime/manager/controller/machinehealthcheck "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineHealthCheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0723 18:52:22.373225       1 controller.go:165] controller-runtime/manager/controller/machine "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Machine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0723 18:52:22.373431       1 controller.go:165] controller-runtime/manager/controller/cluster "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0723 18:52:22.373453       1 controller.go:165] controller-runtime/manager/controller/machinehealthcheck "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineHealthCheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:22.373481       1 controller.go:165] controller-runtime/manager/controller/machine "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Machine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:22.373499       1 controller.go:165] controller-runtime/manager/controller/machineset "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0723 18:52:22.373520       1 controller.go:173] controller-runtime/manager/controller/machinehealthcheck "msg"="Starting Controller" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineHealthCheck"
I0723 18:52:22.373544       1 controller.go:173] controller-runtime/manager/controller/machine "msg"="Starting Controller" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Machine"
I0723 18:52:22.373489       1 controller.go:173] controller-runtime/manager/controller/cluster "msg"="Starting Controller" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"
I0723 18:52:22.373608       1 controller.go:165] controller-runtime/manager/controller/machineset "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0723 18:52:22.373628       1 controller.go:165] controller-runtime/manager/controller/machinedeployment "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0723 18:52:22.373674       1 controller.go:165] controller-runtime/manager/controller/machineset "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0723 18:52:22.373732       1 controller.go:165] controller-runtime/manager/controller/machinedeployment "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0723 18:52:22.373743       1 controller.go:165] controller-runtime/manager/controller/machineset "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:22.373798       1 controller.go:173] controller-runtime/manager/controller/machineset "msg"="Starting Controller" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet"
I0723 18:52:22.373811       1 controller.go:165] controller-runtime/manager/controller/machinedeployment "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0723 18:52:22.373887       1 controller.go:165] controller-runtime/manager/controller/machinedeployment "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false}}}
I0723 18:52:22.373994       1 controller.go:173] controller-runtime/manager/controller/machinedeployment "msg"="Starting Controller" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment"
I0723 18:52:22.675132       1 controller.go:207] controller-runtime/manager/controller/cluster "msg"="Starting workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "worker count"=10
I0723 18:52:22.675155       1 controller.go:207] controller-runtime/manager/controller/cluster "msg"="Starting workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "worker count"=10
I0723 18:52:22.675136       1 controller.go:207] controller-runtime/manager/controller/machineset "msg"="Starting workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet" "worker count"=10
I0723 18:52:22.675194       1 controller.go:207] controller-runtime/manager/controller/machinehealthcheck "msg"="Starting workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineHealthCheck" "worker count"=10
I0723 18:52:22.675211       1 controller.go:207] controller-runtime/manager/controller/machine "msg"="Starting workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Machine" "worker count"=10
I0723 18:52:22.675212       1 controller.go:207] controller-runtime/manager/controller/machinedeployment "msg"="Starting workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment" "worker count"=10
I0723 18:54:28.384140       1 request.go:668] Waited for 1.047364486s due to client-side throttling, not priority and fairness, request: GET:https://10.233.0.1:443/apis/rook.io/v1alpha2?timeout=32s
I0723 18:54:29.201892       1 tracker.go:55] controller-runtime/manager/controller/cluster "msg"="Adding watcher on external object" "name"="cluster-sample" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "GroupVersionKind"="infrastructure.cluster.x-k8s.io/v1alpha4, Kind=NestedCluster"
I0723 18:54:29.202008       1 controller.go:130] controller-runtime/manager/controller/cluster "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "source"={"Type":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha4","kind":"NestedCluster"}}
I0723 18:54:29.221881       1 tracker.go:55] controller-runtime/manager/controller/cluster "msg"="Adding watcher on external object" "name"="cluster-sample" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "GroupVersionKind"="controlplane.cluster.x-k8s.io/v1alpha4, Kind=NestedControlPlane"
I0723 18:54:29.222003       1 controller.go:130] controller-runtime/manager/controller/cluster "msg"="Starting EventSource" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "source"={"Type":{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha4","kind":"NestedControlPlane"}}
I0723 19:01:30.530794       1 cluster_controller.go:278] controller-runtime/manager/controller/cluster "msg"="Cluster still has descendants - need to requeue" "name"="cluster-sample" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "controlPlaneRef"="cluster-sample-control-plane"
I0723 19:01:30.570381       1 cluster_controller.go:278] controller-runtime/manager/controller/cluster "msg"="Cluster still has descendants - need to requeue" "name"="cluster-sample" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "controlPlaneRef"="cluster-sample-control-plane"
I0723 19:01:30.596626       1 cluster_controller.go:278] controller-runtime/manager/controller/cluster "msg"="Cluster still has descendants - need to requeue" "name"="cluster-sample" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster" "controlPlaneRef"="cluster-sample-control-plane"
E0723 19:02:15.794509       1 controller.go:302] controller-runtime/manager/controller/cluster "msg"="Reconciler error" "error"="clusters.cluster.x-k8s.io \"cluster-sample\" not found" "name"="cluster-sample" "namespace"="default" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"

capn-nested-control-plane-controller-manager:

I0723 18:52:25.273812       1 main.go:186] Valid token audiences:
I0723 18:52:25.273945       1 main.go:232] Generating self signed cert as no cert is provided
I0723 18:52:26.926252       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
I0723 18:52:26.927076       1 main.go:288] Listening securely on 0.0.0.0:8443
I0727 09:53:28.698970       1 request.go:645] Throttling request took 1.011972761s, request: GET:https://10.233.0.1:443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s
I0727 09:53:29.153602       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"="127.0.0.1:8080"
I0727 09:53:29.154477       1 main.go:194] setup "msg"="Starting manager"  "version"=""
I0727 09:53:29.154555       1 leaderelection.go:243] attempting to acquire leader lease  capn-nested-control-plane-system/controller-leader-election-capn...
I0727 09:53:29.154687       1 internal.go:408] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0727 09:53:46.270397       1 leaderelection.go:253] successfully acquired lease capn-nested-control-plane-system/controller-leader-election-capn
I0727 09:53:46.271124       1 controller.go:158] controller/nestedetcd "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedEtcd" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{"healthy":false}}}
I0727 09:53:46.271242       1 controller.go:158] controller/nestedcontrollermanager "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControllerManager" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{"healthy":false}}}
I0727 09:53:46.271286       1 controller.go:158] controller/nestedapiserver "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedAPIServer" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{"healthy":false}}}
I0727 09:53:46.271438       1 controller.go:158] controller/nestedcontrolplane "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControlPlane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"initialized":false,"ready":false}}}
I0727 09:53:46.886615       1 controller.go:158] controller/nestedcontrolplane "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControlPlane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{"healthy":false}}}
I0727 09:53:46.886744       1 controller.go:158] controller/nestedcontrolplane "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControlPlane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{"healthy":false}}}
I0727 09:53:46.888723       1 controller.go:158] controller/nestedetcd "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedEtcd" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"selector":null,"template":{"metadata":{"creationTimestamp":null},"spec":{"containers":null}},"serviceName":"","updateStrategy":{}},"status":{"replicas":0}}}
I0727 09:53:46.888832       1 controller.go:165] controller/nestedetcd "msg"="Starting Controller" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedEtcd"
I0727 09:53:46.888883       1 controller.go:192] controller/nestedetcd "msg"="Starting workers" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedEtcd" "worker count"=1
I0727 09:53:46.972495       1 controller.go:158] controller/nestedapiserver "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedAPIServer" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"selector":null,"template":{"metadata":{"creationTimestamp":null},"spec":{"containers":null}},"serviceName":"","updateStrategy":{}},"status":{"replicas":0}}}
I0727 09:53:46.972647       1 controller.go:165] controller/nestedapiserver "msg"="Starting Controller" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedAPIServer"
I0727 09:53:46.972746       1 controller.go:192] controller/nestedapiserver "msg"="Starting workers" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedAPIServer" "worker count"=1
I0727 09:53:46.985737       1 controller.go:158] controller/nestedcontrollermanager "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControllerManager" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"selector":null,"template":{"metadata":{"creationTimestamp":null},"spec":{"containers":null}},"serviceName":"","updateStrategy":{}},"status":{"replicas":0}}}
I0727 09:53:46.985862       1 controller.go:165] controller/nestedcontrollermanager "msg"="Starting Controller" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControllerManager"
I0727 09:53:46.985925       1 controller.go:192] controller/nestedcontrollermanager "msg"="Starting workers" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControllerManager" "worker count"=1
I0727 09:53:46.987922       1 controller.go:158] controller/nestedcontrolplane "msg"="Starting EventSource" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControlPlane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"resources":{}},"status":{"healthy":false}}}
I0727 09:53:46.987991       1 controller.go:165] controller/nestedcontrolplane "msg"="Starting Controller" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControlPlane"
I0727 09:53:46.988039       1 controller.go:192] controller/nestedcontrolplane "msg"="Starting workers" "reconcilerGroup"="controlplane.cluster.x-k8s.io" "reconcilerKind"="NestedControlPlane" "worker count"=1
I0727 09:53:46.988172       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:53:47.187013       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.201091       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
E0727 09:54:40.201172       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.206572       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
E0727 09:54:40.206675       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.217607       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
E0727 09:54:40.217715       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.224011       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.224140       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.224183       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.229631       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.229722       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.229803       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.238919       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
E0727 09:54:40.239021       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.240054       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.240146       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.240218       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.248461       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.248554       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.248605       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.253637       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.253915       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.254051       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.254121       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.261197       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.261290       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.261364       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.264387       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.264483       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.264551       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.285589       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.285686       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.285727       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.302212       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.302339       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.302379       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.306649       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.306938       1 nestedcontrolplane_controller.go:186] controllers/controlplane/NestedControlPlane "msg"="Reconcile NestedControlPlane" "cluster"="cluster-sample" "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.385582       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.386501       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.386551       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.386655       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.386665       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.386769       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.388286       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:40.388449       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.388565       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.485668       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.485827       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.485892       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.585758       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:40.585772       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.585870       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.585874       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.585920       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.585939       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.686291       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:40.686355       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:40.686381       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.686422       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.686434       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.686473       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:40.785654       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:40.785746       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.785784       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.826283       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:40.826381       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.826432       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.985707       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:40.985812       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.985710       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:40.985853       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:40.985898       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:40.985994       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:41.007041       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:41.007158       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:41.007213       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:41.147040       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:41.147177       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:41.147296       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:41.467562       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:41.467656       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:41.467695       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:41.626195       1 nestedetcd_controller.go:56] controllers/controlplane/NestedEtcd "msg"="Reconciling NestedEtcd..." "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"}
I0727 09:54:41.626290       1 nestedetcd_controller.go:61] controllers/controlplane/NestedEtcd "msg"="creating NestedEtcd" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:41.626328       1 nestedetcd_controller.go:70] controllers/controlplane/NestedEtcd "msg"="the owner has not been set yet, will retry later" "nestedetcd"={"Namespace":"default","Name":"cluster-sample-nestedetcd"} "name"="cluster-sample-nestedetcd" "namespace"="default"
I0727 09:54:41.685639       1 nestedapiserver_controller.go:54] controllers/controlplane/NestedAPIServer "msg"="Reconciling NestedAPIServer..." "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"}
I0727 09:54:41.685723       1 nestedapiserver_controller.go:59] controllers/controlplane/NestedAPIServer "msg"="creating NestedAPIServer" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:41.685760       1 nestedapiserver_controller.go:69] controllers/controlplane/NestedAPIServer "msg"="the owner has not been set yet, will retry later" "nestedapiserver"={"Namespace":"default","Name":"cluster-sample-nestedapiserver"} "name"="cluster-sample-nestedapiserver" "namespace"="default"
I0727 09:54:42.186107       1 nestedcontrollermanager_controller.go:49] controllers/controlplane/NestedControllerManager "msg"="Reconciling NestedControllerManager..." "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"}
I0727 09:54:42.186240       1 nestedcontrollermanager_controller.go:54] controllers/controlplane/NestedControllerManager "msg"="creating NestedControllerManager" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"
I0727 09:54:42.186284       1 nestedcontrollermanager_controller.go:64] controllers/controlplane/NestedControllerManager "msg"="the owner has not been set yet, will retry later" "nestedcontrollermanager"={"Namespace":"default","Name":"cluster-sample-nestedcontrollermanager"} "name"="cluster-sample-nestedcontrollermanager" "namespace"="default"

capn-controller-manager:

I0727 09:37:29.273700       1 main.go:186] Valid token audiences:
I0727 09:37:29.274111       1 main.go:232] Generating self signed cert as no cert is provided
I0727 09:37:35.772923       1 request.go:645] Throttling request took 1.007788349s, request: GET:https://10.233.0.1:443/apis/policy/v1beta1?timeout=32s
I0727 09:37:36.366724       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"="127.0.0.1:8080"
I0727 09:37:36.377804       1 main.go:160] setup "msg"="Starting manager"  "version"=""
I0727 09:37:36.377939       1 leaderelection.go:243] attempting to acquire leader lease  capn-system/controller-leader-election-capn...
I0727 09:37:30.157758       1 main.go:281] Starting TCP socket on 0.0.0.0:8443
I0727 09:37:30.158394       1 main.go:288] Listening securely on 0.0.0.0:8443
I0727 09:37:36.378030       1 internal.go:408] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0727 09:37:51.542727       1 leaderelection.go:253] successfully acquired lease capn-system/controller-leader-election-capn
I0727 09:37:51.543327       1 controller.go:158] controller/nestedcluster "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="NestedCluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{}}}
I0727 09:37:51.644740       1 controller.go:158] controller/nestedcluster "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="NestedCluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"initialized":false,"ready":false}}}
I0727 09:37:51.845324       1 controller.go:165] controller/nestedcluster "msg"="Starting Controller" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="NestedCluster"
I0727 09:37:51.845399       1 controller.go:192] controller/nestedcluster "msg"="Starting workers" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="NestedCluster" "worker count"=1
I0727 09:37:51.845557       1 nestedcluster_controller.go:68] controllers/infrastructure/NestedCluster "msg"="Reconciling NestedCluster..." "nestedcluster"={"Namespace":"default","Name":"cluster-sample"}

cluster-sample:

kubectl get cluster cluster-sample -o yaml                                
apiVersion: cluster.x-k8s.io/v1alpha4
kind: Cluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cluster.x-k8s.io/v1alpha4","kind":"Cluster","metadata":{"annotations":{},"name":"cluster-sample","namespace":"default"},"spec":{"controlPlaneEndpoint":{"host":"cluster-sample-apiserver","port":6443},"controlPlaneRef":{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha4","kind":"NestedControlPlane","name":"cluster-sample-control-plane"},"infrastructureRef":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha4","kind":"NestedCluster","name":"cluster-sample"}}}
  creationTimestamp: "2021-07-27T09:54:40Z"
  finalizers:
  - cluster.cluster.x-k8s.io
  generation: 1
  name: cluster-sample
  namespace: default
  resourceVersion: "10549825"
  selfLink: /apis/cluster.x-k8s.io/v1alpha4/namespaces/default/clusters/cluster-sample
  uid: a8946ecd-cd01-4e02-a78e-c65a99c26d92
spec:
  controlPlaneEndpoint:
    host: cluster-sample-apiserver
    port: 6443
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha4
    kind: NestedControlPlane
    name: cluster-sample-control-plane
    namespace: default
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
    kind: NestedCluster
    name: cluster-sample
    namespace: default
status:
  conditions:
  - lastTransitionTime: "2021-07-27T09:54:40Z"
    reason: WaitingForControlPlane
    severity: Info
    status: "False"
    type: Ready
  - lastTransitionTime: "2021-07-27T09:54:40Z"
    message: Waiting for control plane provider to indicate the control plane has
      been initialized
    reason: WaitingForControlPlaneProviderInitialized
    severity: Info
    status: "False"
    type: ControlPlaneInitialized
  - lastTransitionTime: "2021-07-27T09:54:40Z"
    reason: WaitingForControlPlane
    severity: Info
    status: "False"
    type: ControlPlaneReady
  - lastTransitionTime: "2021-07-27T09:54:40Z"
    reason: WaitingForInfrastructure
    severity: Info
    status: "False"
    type: InfrastructureReady
  observedGeneration: 1
  phase: Provisioning

What did you expect to happen:

Anything else you would like to add: The problem here, as I understand it: https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/ea5f01bb8367b336f56ffa0239bffa9e09f560bb/controlplane/nested/controllers/nestedapiserver_controller.go#L83

https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/d2f95df415d0af011094633f27258c4ab812a170/controlplane/nested/controllers/nestedetcd_controller.go#L84

https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/d2f95df415d0af011094633f27258c4ab812a170/controlplane/nested/controllers/nestedcontrollermanager_controller.go#L78

Environment:

/kind bug

christopherhein commented 3 years ago

@D1abloRUS can you run kubectl get pods in the namespace where the Cluster CR is created, I can't seem to see anything in the logs that would show what's going on. This looks like the pods for the control plane aren't going healthy. Once you list pods you will probably see a failing pod, if you could log that and paste those that would be helpful.

sergeimonakhov commented 3 years ago

@christopherhein

E0727 09:54:40.201172       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.206572       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
E0727 09:54:40.206675       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
I0727 09:54:40.217607       1 nestedcontrolplane_controller.go:74] controllers/controlplane/NestedControlPlane "msg"="Reconciling NestedControlPlane..." "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
E0727 09:54:40.217715       1 nestedcontrolplane_controller.go:89] controllers/controlplane/NestedControlPlane "msg"="Failed to retrieve owner Cluster from the API Server" "error"=null "nestedcontrolplane"={"Namespace":"default","Name":"cluster-sample-control-plane"}
kubectl get pod 
No resources found in default namespace.
kubectl get cluster
NAME             PHASE
cluster-sample   Provisioning

No resource is created for the nested control plane

christopherhein commented 3 years ago

Can you check statefulsets? also events?

sergeimonakhov commented 3 years ago
kubectl get statefulsets.apps
No resources found in default namespace.

events empty

gyliu513 commented 3 years ago

Seems the controller failed to create the statefulset for tenant cluster, not sure when this case will happen, like in which case we will get the owner has not been set yet, will retry later error? @christopherhein

         // 1. check if the ownerreference has been set by the
    // NestedControlPlane controller.
    owner := getOwner(nkas.ObjectMeta)
    if owner == (metav1.OwnerReference{}) {
        // requeue the request if the owner NestedControlPlane has
        // not been set yet.
        log.Info("the owner has not been set yet, will retry later",
            "namespace", nkas.GetNamespace(),
            "name", nkas.GetName())
        return ctrl.Result{Requeue: true}, nil
    }
ydp commented 3 years ago

@D1abloRUS have you figured out the causes?

sergeimonakhov commented 3 years ago

@D1abloRUS have you figured out the causes?

I don't have time for this yet

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/197#issuecomment-1073702950): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.