Can't have 3 master nodes when following 'Quick Start' in book

Insullone commented 3 years ago

I followed the instruction in https://cluster-api.sigs.k8s.io/user/quick-start.html with docker infrastructure provider, at the end, I saw 1 master node and 3 worker nodes ready, but I should see 3 master nodes instead.

When I run 'kubectl describe kubeadmcontrolplane capi-quickstart-control-plane', I saw a warning in the end:

Events: Type Reason Age From Message

Warning ControlPlaneUnhealthy 4m6s (x1310 over 10h) kubeadm-control-plane-controller Waiting for control plane to pass preflight checks to continue reconciliation: [machine capi-quickstart-control-plane-5skgf does not have APIServerPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have ControllerManagerPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have SchedulerPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have EtcdPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have EtcdMemberHealthy condition]

Is the warning the reason that only 1 master node is up?

What did you expect to happen: I expect to see 3 master nodes ready (mandatory), and the warning is removed (optional).

Environment:

Cluster-api version: clusterctl version: &version.Info{Major:"0", Minor:"3", GitVersion:"v0.3.11", GitCommit:"e9cf6846b6d93dedadfcf44c00357d15f5ccba64", GitTreeState:"clean", BuildDate:"2020-11-19T18:49:17Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Minikube/KIND version: kind v0.7.0 go1.13.6 linux/amd64
Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:17:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)"

/kind bug /area clusterctl

fabriziopandini commented 3 years ago

is it possible to have a copy of your KubeadmControlPlane object, and possibly a link to take a look at some logs (KCP, CAPI, CAPD)?

Insullone commented 3 years ago

I put some logs below:

kubectl get KubeadmControlPlane NAME INITIALIZED API SERVER AVAILABLE VERSION REPLICAS READY UPDATED UNAVAILABLE capi-quickstart-control-plane true true v1.18.2 1 1 1

kubectl logs capi-kubeadm-control-plane-controller-manager-7556f79cb-4dzr4 -n capi-kubeadm-control-plane-system -c manager I1218 13:49:19.868819 1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen" "addr"="127.0.0.1:8080" I1218 13:49:19.869235 1 main.go:147] setup "msg"="starting manager" "version"="" I1218 13:49:19.869461 1 leaderelection.go:242] attempting to acquire leader lease capi-kubeadm-control-plane-system/kubeadm-control-plane-manager-leader-election-capi... I1218 13:49:19.869596 1 internal.go:356] controller-runtime/manager "msg"="starting metrics server" "path"="/metrics" I1218 13:49:19.876779 1 leaderelection.go:252] successfully acquired lease capi-kubeadm-control-plane-system/kubeadm-control-plane-manager-leader-election-capi I1218 13:49:19.877807 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"version":"","infrastructureTemplate":{},"kubeadmConfigSpec":{}},"status":{"initialized":false,"ready":false}}} I1218 13:49:19.978572 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} I1218 13:49:20.079382 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} I1218 13:49:20.179967 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="kubeadmcontrolplane" I1218 13:49:20.180022 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="kubeadmcontrolplane" "worker count"=10 I1218 13:50:50.689091 1 controller.go:129] controllers/KubeadmControlPlane "msg"="Cluster Controller has not yet set OwnerRef" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" I1218 13:50:51.775440 1 controller.go:244] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="capi-quickstart" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" I1218 13:50:52.991396 1 controller.go:349] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="capi-quickstart" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" "Desired"=3 "Existing"=0 I1218 13:50:53.142187 1 controller.go:182] controllers/KubeadmControlPlane "msg"="Could not connect to workload cluster to fetch status" "cluster"="capi-quickstart" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" "err"="failed to create remote cluster client: default/capi-quickstart: Get https://172.17.0.5:6443/api?timeout=30s: EOF" I1218 13:50:53.158884 1 controller.go:244] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="capi-quickstart" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" I1218 13:50:53.197351 1 controller.go:355] controllers/KubeadmControlPlane "msg"="Scaling up control plane" "cluster"="capi-quickstart" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" "Desired"=3 "Existing"=1 I1218 13:50:53.197466 1 scale.go:206] controllers/KubeadmControlPlane "msg"="Waiting for control plane to pass preflight checks" "cluster"="capi-quickstart" "kubeadmControlPlane"="capi-quickstart-control-plane" "namespace"="default" "failures"="[machine capi-quickstart-control-plane-jv2r6 does not have APIServerPodHealthy condition, machine capi-quickstart-control-plane-jv2r6 does not have ControllerManagerPodHealthy condition, machine capi-quickstart-control-plane-jv2r6 does not have SchedulerPodHealthy condition, machine capi-quickstart-control-plane-jv2r6 does not have EtcdPodHealthy condition, machine capi-quickstart-control-plane-jv2r6 does not have EtcdMemberHealthy condition]" ...

kubectl logs I1218 13:49:12.413880 I1218 13:49:12.415007 I1218 13:49:12.415242 I1218 13:49:12.415611 I1218 13:49:12.422649 I1218 13:49:12.516243 I1218 13:49:12.516675 I1218 13:49:12.516961 I1218 13:49:12.517228 I1218 13:49:12.517351 I1218 13:49:12.518043 I1218 13:49:12.518209 I1218 13:49:12.616703 I1218 13:49:12.616764 I1218 13:49:12.616789 I1218 13:49:12.616813 I1218 13:49:12.617264 I1218 13:49:12.617391 I1218 13:49:12.617489 I1218 13:49:12.617491 I1218 13:49:12.617624 I1218 13:49:12.617760 I1218 13:49:12.617786 I1218 13:49:12.617821 I1218 13:49:12.617870 I1218 13:49:12.617894 I1218 13:49:12.620312 I1218 13:49:12.620363 I1218 13:49:12.620387 I1218 13:49:12.620871 I1218 13:49:12.620910 I1218 13:49:12.620944 I1218 13:49:12.620958 I1218 13:50:50.668041 I1218 13:50:50.668090 I1218 13:50:50.762077 I1218 13:50:50.762107 I1218 13:50:50.798287 I1218 13:50:50.798331 I1218 13:50:50.806268 I1218 13:50:50.807605 I1218 13:50:50.838140 I1218 13:50:50.838192 I1218 13:50:50.853270 I1218 13:50:50.853411 I1218 13:50:50.875488 I1218 13:50:50.880990 I1218 13:50:50.915207 I1218 13:50:50.931208 I1218 13:50:50.955076 I1218 13:50:50.955960 I1218 13:50:50.958889 I1218 13:50:50.959041 I1218 13:50:50.984674 I1218 13:50:50.985623 capi-controller-manager-7c59f8d784-8ghzg -n capi-system -c manager 1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen" "addr"="127.0.0.1:8080" 1 main.go:187] setup "msg"="starting manager" "version"="" 1 leaderelection.go:242] attempting to acquire leader lease capi-system/controller-leader-election-capi... 1 internal.go:356] controller-runtime/manager "msg"="starting metrics server" "path"="/metrics" 1 leaderelection.go:252] successfully acquired lease capi-system/controller-leader-election-capi 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"unhealthyConditions":null},"status":{}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="machinehealthcheck" 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machinehealthcheck" "worker count"=10 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}} 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="machineset" 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machineset" "worker count"=10 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}} 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="machinedeployment" 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machinedeployment" "worker count"=10 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="cluster" 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="cluster" "worker count"=10 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="machine" 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machine" "worker count"=10 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="cluster" 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="cluster" "worker count"=10 1 tracker.go:55] controllers/Cluster "msg"="Adding watcher on external object" "cluster"="capi-quickstart" "namespace"="default" "GroupVersionKind"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=DockerCluster" 1 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="cluster" "source"={"Type":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"DockerCluster"}} 1 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" "creating"=3 "need"=3 1 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 3, ( spec.replicas(3) \u003e currentMachineCount(0) )" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" 1 tracker.go:55] controllers/Cluster "msg"="Adding watcher on external object" "cluster"="capi-quickstart" "namespace"="default" "GroupVersionKind"="controlplane.cluster.x-k8s.io/v1alpha3, Kind=KubeadmControlPlane" 1 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="cluster" "source"={"Type":{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"KubeadmControlPlane"}} 1 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 3 with name \"capi-quickstart-md-0-5477855897-j7jhh\"" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" 1 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 2 of 3, ( spec.replicas(3) \u003e currentMachineCount(0) )" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" 1 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 2 of 3 with name \"capi-quickstart-md-0-5477855897-vbtwr\"" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" 1 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 3 of 3, ( spec.replicas(3) \u003e currentMachineCount(0) )" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" 1 tracker.go:55] controllers/Machine "msg"="Adding watcher on external object" "machine"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "GroupVersionKind"="bootstrap.cluster.x-k8s.io/v1alpha3, Kind=KubeadmConfig" 1 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={"Type":{"apiVersion":"bootstrap.cluster.x-k8s.io/v1alpha3","kind":"KubeadmConfig"}} 1 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 3 of 3 with name \"capi-quickstart-md-0-5477855897-gg8fn\"" "machineset"="capi-quickstart-md-0-5477855897" "namespace"="default" 1 machine_controller_phases.go:219] controllers/Machine "msg"="Bootstrap provider is not ready, requeuing" "machine"="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" 1 tracker.go:55] controllers/Machine "msg"="Adding watcher on external object" "machine"="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "GroupVersionKind"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=DockerMachine" 1 machine_controller_phases.go:219] controllers/Machine "msg"="Bootstrap provider is not ready, requeuing" "machine"="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" 1 machine_controller_phases.go:219] controllers/Machine "msg"="Bootstrap provider is not ready, requeuing" "machine"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" 1 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={"Type":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"DockerMachine"}} 1 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" 1 machine_controller_noderef.go:42] controllers/Machine "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" 1 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" 1 machine_controller_noderef.go:42] controllers/Machine "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default"

kubectl logs capd-controller-manager-69999d8d44-b7wzb -n capd-system -c manager I1218 13:49:28.391246 1 request.go:565] Throttling request took 78.673996ms, request: GET:https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s I1218 13:49:28.441119 1 request.go:565] Throttling request took 128.580989ms, request: GET:https://10.96.0.1:443/apis/cert-manager.io/v1alpha3?timeout=32s I1218 13:49:28.491156 1 request.go:565] Throttling request took 178.615241ms, request: GET:https://10.96.0.1:443/apis/coordination.k8s.io/v1?timeout=32s I1218 13:49:28.541092 1 request.go:565] Throttling request took 228.52425ms, request: GET:https://10.96.0.1:443/apis/controlplane.cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:28.591125 1 request.go:565] Throttling request took 278.513419ms, request: GET:https://10.96.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s I1218 13:49:28.641080 1 request.go:565] Throttling request took 328.486564ms, request: GET:https://10.96.0.1:443/apis/exp.cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:28.691100 1 request.go:565] Throttling request took 378.509515ms, request: GET:https://10.96.0.1:443/apis/node.k8s.io/v1beta1?timeout=32s I1218 13:49:28.741277 1 request.go:565] Throttling request took 428.517542ms, request: GET:https://10.96.0.1:443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:28.791209 1 request.go:565] Throttling request took 478.607319ms, request: GET:https://10.96.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s I1218 13:49:28.841091 1 request.go:565] Throttling request took 528.49687ms, request: GET:https://10.96.0.1:443/apis/acme.cert-manager.io/v1beta1?timeout=32s I1218 13:49:28.891347 1 request.go:565] Throttling request took 578.711029ms, request: GET:https://10.96.0.1:443/apis/acme.cert-manager.io/v1alpha3?timeout=32s I1218 13:49:28.941125 1 request.go:565] Throttling request took 628.491366ms, request: GET:https://10.96.0.1:443/apis/acme.cert-manager.io/v1alpha2?timeout=32s I1218 13:49:28.991086 1 request.go:565] Throttling request took 678.453115ms, request: GET:https://10.96.0.1:443/apis/bootstrap.cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:29.041093 1 request.go:565] Throttling request took 728.445394ms, request: GET:https://10.96.0.1:443/apis/clusterctl.cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:29.091103 1 request.go:565] Throttling request took 778.472752ms, request: GET:https://10.96.0.1:443/apis/bootstrap.cluster.x-k8s.io/v1alpha2?timeout=32s I1218 13:49:29.141085 1 request.go:565] Throttling request took 828.446678ms, request: GET:https://10.96.0.1:443/apis/cert-manager.io/v1beta1?timeout=32s I1218 13:49:29.191083 1 request.go:565] Throttling request took 878.448665ms, request: GET:https://10.96.0.1:443/apis/cert-manager.io/v1alpha2?timeout=32s I1218 13:49:29.241122 1 request.go:565] Throttling request took 928.435339ms, request: GET:https://10.96.0.1:443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:29.291173 1 request.go:565] Throttling request took 978.52849ms, request: GET:https://10.96.0.1:443/apis/cluster.x-k8s.io/v1alpha2?timeout=32s I1218 13:49:29.341158 1 request.go:565] Throttling request took 1.028437907s, request: GET:https://10.96.0.1:443/apis/addons.cluster.x-k8s.io/v1alpha3?timeout=32s I1218 13:49:29.346359 1 webhook.go:92] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented" "GVK"={"Group":"infrastructure.cluster.x-k8s.io","Version":"v1alpha3","Kind":"DockerMachineTemplate"} I1218 13:49:29.346428 1 webhook.go:123] controller-runtime/builder "msg"="Registering a validating webhook" "GVK"={"Group":"infrastructure.cluster.x-k8s.io","Version":"v1alpha3","Kind":"DockerMachineTemplate"} "path"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" I1218 13:49:29.346468 1 server.go:121] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" I1218 13:49:29.346559 1 main.go:96] setup "msg"="starting manager"
I1218 13:49:29.346758 1 shared_informer.go:227] caches populated I1218 13:49:29.346846 1 server.go:141] controller-runtime/webhook/webhooks "msg"="starting webhook server"
I1218 13:49:29.347452 1 certwatcher.go:127] controller-runtime/certwatcher "msg"="Updated current TLS certificate"
I1218 13:49:29.347512 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="dockercluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"ready":false}}} I1218 13:49:29.347718 1 server.go:199] controller-runtime/webhook "msg"="serving webhook server" "host"="" "port"=9443 I1218 13:49:29.347814 1 reflector.go:150] Starting reflector v1alpha3.DockerCluster (9m39.01827735s) from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.347835 1 reflector.go:185] Listing and watching v1alpha3.DockerCluster from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.347960 1 certwatcher.go:83] controller-runtime/certwatcher "msg"="Starting certificate watcher"
I1218 13:49:29.348378 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="dockermachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"ready":false}}} I1218 13:49:29.348738 1 reflector.go:150] Starting reflector v1alpha3.DockerMachine (10m14.103492521s) from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.348750 1 reflector.go:185] Listing and watching v1alpha3.DockerMachine from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.447867 1 shared_informer.go:227] caches populated I1218 13:49:29.448378 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="dockercluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} I1218 13:49:29.448745 1 reflector.go:150] Starting reflector v1alpha3.Cluster (9m39.397996912s) from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.448764 1 reflector.go:185] Listing and watching v1alpha3.Cluster from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.448812 1 shared_informer.go:227] caches populated I1218 13:49:29.449049 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="dockermachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}} I1218 13:49:29.452327 1 reflector.go:150] Starting reflector v1alpha3.Machine (9m42.107660362s) from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.452447 1 reflector.go:185] Listing and watching v1alpha3.Machine from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:49:29.548891 1 shared_informer.go:227] caches populated I1218 13:49:29.549117 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="dockercluster" I1218 13:49:29.549220 1 shared_informer.go:227] caches populated I1218 13:49:29.549245 1 shared_informer.go:227] caches populated I1218 13:49:29.549332 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="dockercluster" "worker count"=1 I1218 13:49:29.550135 1 shared_informer.go:227] caches populated I1218 13:49:29.550257 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="dockermachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"ready":false}}} I1218 13:49:29.550359 1 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="dockermachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}} I1218 13:49:29.550446 1 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="dockermachine" I1218 13:49:29.550483 1 shared_informer.go:227] caches populated I1218 13:49:29.550532 1 shared_informer.go:227] caches populated I1218 13:49:29.550584 1 shared_informer.go:227] caches populated I1218 13:49:29.550625 1 shared_informer.go:227] caches populated I1218 13:49:29.550682 1 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="dockermachine" "worker count"=10 I1218 13:50:50.632492 1 cluster_predicates.go:72] controllers/DockerMachine "msg"="Cluster is not paused, allowing further processing" "cluster"="capi-quickstart" "eventType"="create" "namespace"="default" "predicate"="ClusterCreateNotPaused" I1218 13:50:50.632673 1 cluster_predicates.go:47] controllers/DockerMachine "msg"="Cluster infrastructure is not ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="create" "namespace"="default" "predicate"="ClusterCreateInfraReady" I1218 13:50:50.632748 1 generic_predicates.go:49] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:50.632490 1 cluster_predicates.go:72] controllers/DockerCluster "msg"="Cluster is not paused, allowing further processing" "cluster"="capi-quickstart" "eventType"="create" "namespace"="default" "predicate"="ClusterCreateNotPaused" I1218 13:50:50.632832 1 generic_predicates.go:103] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.633040 1 generic_predicates.go:103] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.633411 1 generic_predicates.go:99] controllers/DockerCluster "msg"="One of the provided predicates returned true, allowing further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:50.635267 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockercluster" "name"="capi-quickstart" "namespace"="default" I1218 13:50:50.637173 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.637181 1 generic_predicates.go:168] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.637417 1 dockercluster_controller.go:74] DockerCluster-controller "msg"="Waiting for Cluster Controller to set OwnerRef on DockerCluster" "docker-cluster"={"Namespace":"default","Name":"capi-quickstart"} I1218 13:50:50.637441 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockercluster" "name"="capi-quickstart" "namespace"="default" I1218 13:50:50.644838 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:50.644878 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:50.644902 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:50.644926 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.644999 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.645057 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:50.645074 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:50.651789 1 http.go:84] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="a62f33ca-b785-4b6c-a1d2-21c1ffa713c7" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","kind":"DockerMachineTemplate"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","resource":"dockermachinetemplates"} I1218 13:50:50.653813 1 http.go:102] controller-runtime/webhook/webhooks "msg"="wrote response" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="a62f33ca-b785-4b6c-a1d2-21c1ffa713c7" "allowed"=true "result"={"metadata":{},"code":200} I1218 13:50:50.654741 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:50.654771 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:50.654789 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:50.654928 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.654944 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.654862 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:50.655045 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:50.665391 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.665392 1 generic_predicates.go:168] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.686295 1 http.go:84] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="ba92890f-d01b-4aab-a44f-9865c6c19dc3" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","kind":"DockerMachineTemplate"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","resource":"dockermachinetemplates"} I1218 13:50:50.686387 1 http.go:102] controller-runtime/webhook/webhooks "msg"="wrote response" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="ba92890f-d01b-4aab-a44f-9865c6c19dc3" "allowed"=true "result"={"metadata":{},"code":200} I1218 13:50:50.723829 1 http.go:84] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="d3c5a443-88ea-470a-842d-0ed362ebfcd3" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","kind":"DockerMachineTemplate"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","resource":"dockermachinetemplates"} I1218 13:50:50.724084 1 http.go:102] controller-runtime/webhook/webhooks "msg"="wrote response" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="d3c5a443-88ea-470a-842d-0ed362ebfcd3" "allowed"=true "result"={"metadata":{},"code":200} I1218 13:50:50.732535 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.733026 1 generic_predicates.go:168] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.733108 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockercluster" "name"="capi-quickstart" "namespace"="default" I1218 13:50:50.793436 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-pgt8j" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.794253 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} I1218 13:50:50.794291 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:50.810687 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.811330 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} I1218 13:50:50.811571 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:50.823007 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.823675 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} I1218 13:50:50.823707 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:50.829642 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.829979 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} I1218 13:50:50.830263 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:50.841742 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.841752 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-tnzx8" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.841869 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} I1218 13:50:50.841914 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.842044 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} I1218 13:50:50.842064 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.842357 1 loadbalancer.go:78] DockerCluster-controller "msg"="Creating load balancer container" "cluster"="capi-quickstart" "docker-cluster"={"Namespace":"default","Name":"capi-quickstart"} I1218 13:50:50.847570 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.847670 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} I1218 13:50:50.848034 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.856470 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.856593 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} I1218 13:50:50.856633 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.867771 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-rnvv9" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.867862 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} I1218 13:50:50.867995 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:50.873360 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="createEvent" I1218 13:50:50.874759 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} I1218 13:50:50.874893 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:50.899021 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.899122 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} I1218 13:50:50.899413 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:50.905165 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.905505 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} I1218 13:50:50.905532 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:50.910184 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:50.910391 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:50.910414 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:50.910529 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:50.910612 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:50.910629 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.910648 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:50.913001 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-tnzx8" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.924834 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.926310 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-tnzx8" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.926926 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:50.942458 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-tnzx8" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.961196 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.961978 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-rnvv9" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.964762 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:50.965677 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:50.971801 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:50.972482 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.975817 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-rnvv9" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.984159 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-pgt8j" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.985073 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:50.985767 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} "machine"="capi-quickstart-md-0-5477855897-gg8fn" I1218 13:50:50.991731 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-pgt8j" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.994627 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:50.995474 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} "machine"="capi-quickstart-md-0-5477855897-j7jhh" I1218 13:50:50.994955 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="updateEvent" I1218 13:50:50.997803 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-rnvv9" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.007354 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:51.007556 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} "machine"="capi-quickstart-md-0-5477855897-gg8fn" I1218 13:50:51.007759 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:51.008310 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-pgt8j" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.013940 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:51.014130 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} "machine"="capi-quickstart-md-0-5477855897-j7jhh" I1218 13:50:51.014328 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:51.024262 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.024427 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} "machine"="capi-quickstart-md-0-5477855897-gg8fn" I1218 13:50:51.024709 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:51.033041 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.033293 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} "machine"="capi-quickstart-md-0-5477855897-j7jhh" I1218 13:50:51.034007 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:51.074819 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.074992 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:51.075237 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:51.115750 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.115971 1 dockermachine_controller.go:135] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for DockerCluster Controller to create cluster infrastructure" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:51.124164 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:51.600370 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.600415 1 generic_predicates.go:168] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.602725 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" I1218 13:50:51.602899 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" I1218 13:50:51.603015 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" I1218 13:50:51.610836 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.611464 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" I1218 13:50:51.611596 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" I1218 13:50:51.611647 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" I1218 13:50:51.611782 1 generic_predicates.go:168] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" E1218 13:50:51.620620 1 dockercluster_controller.go:94] DockerCluster-controller "msg"="failed to patch DockerCluster" "error"="DockerCluster.infrastructure.cluster.x-k8s.io \"capi-quickstart\" is invalid: status.ready: Required value" "cluster"="capi-quickstart" "docker-cluster"={"Namespace":"default","Name":"capi-quickstart"} E1218 13:50:51.620924 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="DockerCluster.infrastructure.cluster.x-k8s.io \"capi-quickstart\" is invalid: status.ready: Required value" "controller"="dockercluster" "name"="capi-quickstart" "namespace"="default" I1218 13:50:51.633905 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:51.633967 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.633986 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:51.634016 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.634032 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.634053 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.634068 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:51.648418 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.648449 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:51.648693 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:51.648780 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.648853 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:51.648905 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.648941 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.654860 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:51.656167 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.656406 1 cluster_predicates.go:103] controllers/DockerMachine "msg"="Cluster infrastructure became ready, allowing further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:51.656510 1 generic_predicates.go:88] controllers/DockerMachine "msg"="One of the provided predicates returned true, allowing further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.656701 1 generic_predicates.go:88] controllers/DockerMachine "msg"="One of the provided predicates returned true, allowing further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.657458 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.657541 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:51.775072 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:51.775194 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.775242 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:51.775299 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.775343 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:51.775354 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:51.775405 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:51.789686 1 http.go:84] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="0ee33962-d36e-4113-95cc-e7168c4cb523" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","kind":"DockerMachineTemplate"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","resource":"dockermachinetemplates"} I1218 13:50:51.789863 1 http.go:102] controller-runtime/webhook/webhooks "msg"="wrote response" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="0ee33962-d36e-4113-95cc-e7168c4cb523" "allowed"=true "result"={"metadata":{},"code":200} I1218 13:50:51.796535 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.802993 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.822135 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.905766 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} "machine"="capi-quickstart-md-0-5477855897-gg8fn" I1218 13:50:51.911752 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-rnvv9" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.913497 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} "machine"="capi-quickstart-md-0-5477855897-j7jhh" I1218 13:50:51.916542 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:51.922181 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-pgt8j" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.928173 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:51.941481 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.950257 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.969961 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:51.976117 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-tnzx8" "namespace"="default" "predicate"="updateEvent" I1218 13:50:51.985133 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:52.010124 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" "predicate"="updateEvent" I1218 13:50:52.047779 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} "machine"="capi-quickstart-md-0-5477855897-gg8fn" I1218 13:50:52.048143 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:52.092070 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} "machine"="capi-quickstart-md-0-5477855897-j7jhh" I1218 13:50:52.092431 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:52.127008 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:52.127795 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:52.142754 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-rnvv9"} "machine"="capi-quickstart-md-0-5477855897-gg8fn" I1218 13:50:52.143012 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-rnvv9" "namespace"="default" I1218 13:50:52.186907 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-pgt8j"} "machine"="capi-quickstart-md-0-5477855897-j7jhh" I1218 13:50:52.187277 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-pgt8j" "namespace"="default" I1218 13:50:52.208962 1 dockermachine_controller.go:200] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the control plane to be initialized" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-md-0-tnzx8"} "machine"="capi-quickstart-md-0-5477855897-vbtwr" I1218 13:50:52.209255 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-tnzx8" "namespace"="default" I1218 13:50:52.688954 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:52.689400 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-vbtwr" "namespace"="default" I1218 13:50:52.689565 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-j7jhh" "namespace"="default" I1218 13:50:52.689694 1 generic_predicates.go:168] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart" "namespace"="default" "predicate"="updateEvent" I1218 13:50:52.689834 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-md-0-5477855897-gg8fn" "namespace"="default" I1218 13:50:52.693491 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockercluster" "name"="capi-quickstart" "namespace"="default" I1218 13:50:52.755870 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockercluster" "name"="capi-quickstart" "namespace"="default" I1218 13:50:53.000758 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="createEvent" I1218 13:50:53.002081 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} I1218 13:50:53.002128 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-control-plane-hvn46" "namespace"="default" I1218 13:50:53.019729 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="createEvent" I1218 13:50:53.022869 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} I1218 13:50:53.022901 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-control-plane-hvn46" "namespace"="default" I1218 13:50:53.033964 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.034240 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} I1218 13:50:53.034266 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-control-plane-hvn46" "namespace"="default" I1218 13:50:53.039930 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.040011 1 dockermachine_controller.go:80] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} I1218 13:50:53.040114 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-control-plane-hvn46" "namespace"="default" I1218 13:50:53.065513 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.071059 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.073971 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.076049 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-control-plane-hvn46" "namespace"="default" I1218 13:50:53.133196 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.169059 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.174209 1 dockermachine_controller.go:205] controllers/DockerMachine/DockerMachine-controller "msg"="Waiting for the Bootstrap provider controller to set bootstrap data" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" I1218 13:50:53.176782 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.183517 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.193364 1 controller.go:281] controller-runtime/controller "msg"="Successfully Reconciled" "controller"="dockermachine" "name"="capi-quickstart-control-plane-hvn46" "namespace"="default" I1218 13:50:53.196912 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:53.196938 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:50:53.197088 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:50:53.197365 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:50:53.197393 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:50:53.197552 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:53.197571 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:50:53.205942 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.216925 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:53.311620 1 machine.go:195] controllers/DockerMachine/DockerMachine-controller "msg"="Creating control plane machine container" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" I1218 13:50:57.497162 1 loadbalancer.go:127] controllers/DockerMachine/DockerMachine-controller "msg"="Updating load balancer configuration" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" I1218 13:50:57.827519 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:50:57.837613 1 reflector.go:150] Starting reflector v1.Secret (10m42.450755375s) from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:50:57.837634 1 reflector.go:185] Listing and watching v1.Secret from pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105 I1218 13:50:57.855990 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:50:57.941713 1 shared_informer.go:227] caches populated I1218 13:51:23.220766 1 machine.go:333] controllers/DockerMachine/DockerMachine-controller "msg"="Setting Kubernetes node providerID" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" I1218 13:51:23.516154 1 machine.go:345] controllers/DockerMachine/DockerMachine-controller "msg"="Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" E1218 13:51:23.516444 1 dockermachine_controller.go:324] controllers/DockerMachine "msg"="failed to patch the Kubernetes node with the machine providerID" "error"="failed update providerID: command \"docker exec --privileged capi-quickstart-control-plane-tvd7x kubectl --kubeconfig /etc/kubernetes/admin.conf patch node capi-quickstart-control-plane-jv2r6 --patch '{\"spec\": {\"providerID\": \"docker:////capi-quickstart-control-plane-jv2r6\"}}'\" failed with error: exit status 1"
I1218 13:51:23.522907 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:51:23.527744 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:51:23.555336 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-hvn46" "namespace"="default" "predicate"="updateEvent" I1218 13:51:23.573695 1 generic_predicates.go:168] controllers/DockerMachine "msg"="Resource is not paused, will attempt to map resource" ""="capi-quickstart-control-plane-jv2r6" "namespace"="default" "predicate"="updateEvent" I1218 13:51:23.719102 1 generic_predicates.go:38] controllers/DockerMachine "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" I1218 13:51:23.719233 1 cluster_predicates.go:138] controllers/DockerMachine "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:51:23.719312 1 cluster_predicates.go:107] controllers/DockerMachine "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" I1218 13:51:23.719485 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:51:23.719560 1 generic_predicates.go:92] controllers/DockerMachine "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" I1218 13:51:23.719137 1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="capi-quickstart" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" I1218 13:51:23.719620 1 generic_predicates.go:92] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" I1218 13:51:23.731340 1 machine.go:333] controllers/DockerMachine/DockerMachine-controller "msg"="Setting Kubernetes node providerID" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" I1218 13:51:23.989437 1 machine.go:345] controllers/DockerMachine/DockerMachine-controller "msg"="Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" E1218 13:51:23.989893 1 dockermachine_controller.go:324] controllers/DockerMachine "msg"="failed to patch the Kubernetes node with the machine providerID" "error"="failed update providerID: command \"docker exec --privileged capi-quickstart-control-plane-tvd7x kubectl --kubeconfig /etc/kubernetes/admin.conf patch node capi-quickstart-control-plane-jv2r6 --patch '{\"spec\": {\"providerID\": \"docker:////capi-quickstart-control-plane-jv2r6\"}}'\" failed with error: exit status 1"
I1218 13:51:24.164344 1 machine.go:333] controllers/DockerMachine/DockerMachine-controller "msg"="Setting Kubernetes node providerID" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" I1218 13:51:24.387374 1 machine.go:345] controllers/DockerMachine/DockerMachine-controller "msg"="Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-hvn46"} "machine"="capi-quickstart-control-plane-jv2r6" E1218 13:51:24.387930 1 dockermachine_controller.go:324] controllers/DockerMachine "msg"="failed to patch the Kubernetes node with the machine providerID" "error"="failed update providerID: command \"docker exec --privileged capi-quickstart-control-plane-tvd7x kubectl --kubeconfig /etc/kubernetes/admin.conf patch node capi-quickstart-control-plane-jv2r6 --patch '{\"spec\": {\"providerID\": \"docker:////capi-quickstart-control-plane-jv2r6\"}}'\" failed with error: exit status 1" ...

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes NAME STATUS ROLES AGE VERSION capi-quickstart-control-plane-jv2r6 Ready master 36m v1.18.2 capi-quickstart-md-0-5477855897-gg8fn Ready 13m v1.18.2 capi-quickstart-md-0-5477855897-j7jhh Ready 32m v1.18.2 capi-quickstart-md-0-5477855897-vbtwr Ready 35m v1.18.2

Insullone commented 3 years ago

@fabriziopandini I pasted some logs above

fabriziopandini commented 3 years ago

kubectl get KubeadmControlPlane NAME INITIALIZED API SERVER AVAILABLE VERSION REPLICAS READY UPDATED UNAVAILABLE capi-quickstart-control-plane true true v1.18.2 1 1

Seems that you are requesting one replica...

brightzheng100 commented 3 years ago

Same issue here: I could see only two docker containers newly created for the workload cluster which means that there was only one master node:

$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                                  NAMES
b962d1e63fe2   kindest/node:v1.18.2           "/usr/local/bin/entr…"   8 minutes ago    Up 8 minutes    44905/tcp, 127.0.0.1:44905->6443/tcp   capi-quickstart-control-plane-fwmrb
2f29fcdb7fd0   kindest/haproxy:2.1.1-alpine   "/docker-entrypoint.…"   9 minutes ago    Up 9 minutes    42317/tcp, 0.0.0.0:42317->6443/tcp     capi-quickstart-lb
...

The weird thing was, after untainting the only master node, the workload cluster was functioning perfectly well. For example, I could access it and deploy workloads there.

$ kubectl get nodes --kubeconfig=./capi-quickstart.kubeconfig
NAME                                  STATUS   ROLES    AGE   VERSION
capi-quickstart-control-plane-h8s2j   Ready    master   19m   v1.18.2

$ kubectl get pods --kubeconfig=./capi-quickstart.kubeconfig
NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-s4kbd   1/1     Running   0          6m11s

But when checking the workload cluster, it's not yet ready and I could see many errors like what @Insullone reported.

$ kubectl get Machine -A
NAMESPACE   NAME                                    PROVIDERID   PHASE          VERSION
default     capi-quickstart-control-plane-h8s2j                  Provisioning   v1.18.2
default     capi-quickstart-md-0-5477855897-lvblt                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-xhc6x                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-z7kn8                Pending        v1.18.2

$ kubectl get MachineDeployment -A
NAMESPACE   NAME                   PHASE       REPLICAS   READY   UPDATED   UNAVAILABLE
default     capi-quickstart-md-0   ScalingUp   3                  3         3

$ kubectl get MachineDeployment capi-quickstart-md-0 -o yaml

apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineDeployment
metadata:
...
spec:
  clusterName: capi-quickstart
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: capi-quickstart
      cluster.x-k8s.io/deployment-name: capi-quickstart-md-0
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: capi-quickstart
        cluster.x-k8s.io/deployment-name: capi-quickstart-md-0
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: KubeadmConfigTemplate
          name: capi-quickstart-md-0
          namespace: default
      clusterName: capi-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: DockerMachineTemplate
        name: capi-quickstart-md-0
        namespace: default
      version: v1.18.2
status:
  observedGeneration: 1
  phase: ScalingUp
  replicas: 3
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart,cluster.x-k8s.io/deployment-name=capi-quickstart-md-0
  unavailableReplicas: 3
  updatedReplicas: 3

fabriziopandini commented 3 years ago

/milestone v0.4.0

After some thinking, I'm inclined to assume this is a networking problem between the kind network and the network used by the pods generated by CAPD. In order for the quick start to complete they should match.

The first depends on the kind version you are using and it can be eventually changed with an env variable as documented in the quick start, the latter is up to us, and in v0.3.x was "bridge", but we are trying to converge everything to use "kind" instead of "bridge" so everything is consistent (v1alpha4 is already migrated for v1alpha3 there is https://github.com/kubernetes-sigs/cluster-api/issues/4055).

@Insullone @brightzheng100 could you kindly check my assumption?

vincepri commented 3 years ago

/help

k8s-ci-robot commented 3 years ago

@vincepri: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/4027): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

vincepri commented 3 years ago

@fabriziopandini @brightzheng100 Any updates on this issue?

brightzheng100 commented 3 years ago

@fabriziopandini @vincepri I was using the latest kind and cluster api so the kind network must have been the dedicated kind network, instead of the default bridge.

I believe it's still quite easy to reproduce as I just did it again while trying to follow up with you folks -- bwt, today I even couldn't bring up the workload cluster as it failed somewhere: the docker container was not spun up properly.

Create kind cluster:

cat > kind-cluster-with-extramounts.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock
EOF
kind create cluster --config kind-cluster-with-extramounts.yaml

Initialize Cluster API:

clusterctl init --infrastructure docker

Create workload cluster:

export SERVICE_CIDR=["10.96.0.0/12"]
export POD_CIDR=["192.168.0.0/16"]
export SERVICE_DOMAIN="k8s.test"

clusterctl config cluster capi-quickstart \
  --flavor development \
  --kubernetes-version v1.18.2 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

kubectl apply -f capi-quickstart.yaml

Check it out:

$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
15b866e086c5   kindest/node:v1.19.1   "/usr/local/bin/entr…"   13 minutes ago   Up 13 minutes   127.0.0.1:65252->6443/tcp   kind-control-plane

$ kubectl get cluster --all-namespaces
NAMESPACE   NAME              PHASE
default     capi-quickstart   Provisioned

$ kubectl get kubeadmcontrolplane --all-namespaces
NAMESPACE   NAME                            INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
default     capi-quickstart-control-plane                                        v1.18.2   1                  1         1

$ kgp -A
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-8594dfd47d-2zcv8                         2/2     Running   0          13m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-ddcc95784-kjmgq        2/2     Running   0          13m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-d85dfd44c-2pxgn    2/2     Running   0          13m
capi-system                         capi-controller-manager-5cd969bf55-p77gw                         2/2     Running   0          13m
capi-webhook-system                 capi-controller-manager-68c9ff8646-br8sh                         2/2     Running   0          13m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-7cf85756c5-9x77b       2/2     Running   0          13m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-7854964ff4-6nhsw   2/2     Running   0          13m
cert-manager                        cert-manager-cainjector-fc6c787db-k4wjr                          1/1     Running   0          13m
cert-manager                        cert-manager-d994d94d7-8mrzd                                     1/1     Running   0          13m
cert-manager                        cert-manager-webhook-845d9df8bf-sm8w8                            1/1     Running   0          13m
kube-system                         coredns-f9fd979d6-c9znk                                          1/1     Running   0          14m
kube-system                         coredns-f9fd979d6-pgtvg                                          1/1     Running   0          14m
kube-system                         etcd-kind-control-plane                                          1/1     Running   0          14m
kube-system                         kindnet-7rnsk                                                    1/1     Running   0          14m
kube-system                         kube-apiserver-kind-control-plane                                1/1     Running   0          14m
kube-system                         kube-controller-manager-kind-control-plane                       1/1     Running   0          14m
kube-system                         kube-proxy-shbms                                                 1/1     Running   0          14m
kube-system                         kube-scheduler-kind-control-plane                                1/1     Running   0          14m
local-path-storage                  local-path-provisioner-78776bfc44-mkjzj                          1/1     Running   0          14m

$ kubectl get Machine -A
NAMESPACE   NAME                                    PROVIDERID   PHASE          VERSION
default     capi-quickstart-control-plane-9w29p                  Provisioning   v1.18.2
default     capi-quickstart-md-0-5477855897-7wbl2                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-ljp2d                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-sxglz                Pending        v1.18.2

brightzheng100 commented 3 years ago

Well, after almost an hour, some docker containers were up:

$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED             STATUS             PORTS                                  NAMES
be73c9696b94   kindest/node:v1.18.2           "/usr/local/bin/entr…"   6 minutes ago       Up 6 minutes       35053/tcp, 127.0.0.1:35053->6443/tcp   capi-quickstart-control-plane-9w29p
4793804e320b   kindest/haproxy:2.1.1-alpine   "/docker-entrypoint.…"   6 minutes ago       Up 6 minutes       45081/tcp, 0.0.0.0:45081->6443/tcp     capi-quickstart-lb
15b866e086c5   kindest/node:v1.19.1           "/usr/local/bin/entr…"   About an hour ago   Up About an hour   127.0.0.1:65252->6443/tcp              kind-control-plane

And I could see the workload containers:

$ kgp -A --kubeconfig capi-quickstart.kubeconfig
NAMESPACE     NAME                                                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-66bff467f8-b4qj9                                      0/1     Pending   0          5m15s
kube-system   coredns-66bff467f8-gvnn6                                      0/1     Pending   0          5m15s
kube-system   etcd-capi-quickstart-control-plane-9w29p                      1/1     Running   0          5m22s
kube-system   kube-apiserver-capi-quickstart-control-plane-9w29p            1/1     Running   0          5m22s
kube-system   kube-controller-manager-capi-quickstart-control-plane-9w29p   1/1     Running   0          5m22s
kube-system   kube-proxy-lgnkf                                              1/1     Running   0          5m14s
kube-system   kube-scheduler-capi-quickstart-control-plane-9w29p            1/1     Running   0          5m22s

But while checking Cluster API itself:

$ kubectl get Machine -A
NAMESPACE   NAME                                    PROVIDERID                                       PHASE          VERSION
default     capi-quickstart-control-plane-9w29p     docker:////capi-quickstart-control-plane-9w29p   Provisioning   v1.18.2
default     capi-quickstart-md-0-5477855897-7wbl2                                                    Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-ljp2d                                                    Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-sxglz                                                    Pending        v1.18.2

The tools that I'm with:

$ clusterctl version
clusterctl version: &version.Info{Major:"0", Minor:"3", GitVersion:"v0.3.12", GitCommit:"9e1dd7e8e428e05bee406602952ae269d55bdbba", GitTreeState:"clean", BuildDate:"2020-12-15T16:42:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}

New clusterctl version available: v0.3.12 -> v0.3.14
https://github.com/kubernetes-sigs/cluster-api/releases/tag/v0.3.14

$ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.7
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:12:42 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ kind version
kind v0.9.0 go1.15.2 darwin/amd64

fabriziopandini commented 3 years ago

The problem is that Kind < 0.8.0 use network bridge, kind >= 0.8.0 use network kind (which is now the majority of users) CAPD v1aplha3 still use bridge, so it is required to set export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge in order to allow communications between the management cluster and CAPD nodes

https://github.com/kubernetes-sigs/cluster-api/issues/4055 will simplify this by deprecating support for Kind < 0.8.0 and moving CAPD to network kind

brightzheng100 commented 3 years ago

Just upgraded to kind v0.10 and export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge before creating the workload cluster, the issue remains the same but sounds the Docker containers come back quicker.

$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                                  NAMES
1828313a5bfd   kindest/node:v1.18.2           "/usr/local/bin/entr…"   14 minutes ago   Up 14 minutes   45533/tcp, 127.0.0.1:45533->6443/tcp   capi-quickstart-control-plane-9vjnl
2ad3ceaf0849   kindest/haproxy:2.1.1-alpine   "/docker-entrypoint.…"   14 minutes ago   Up 14 minutes   45031/tcp, 0.0.0.0:45031->6443/tcp     capi-quickstart-lb
0b5980263004   kindest/node:v1.20.2           "/usr/local/bin/entr…"   18 minutes ago   Up 18 minutes   127.0.0.1:53423->6443/tcp              kind-control-plane

$ kubectl get Machine -A
NAMESPACE   NAME                                    PROVIDERID   PHASE          VERSION
default     capi-quickstart-control-plane-9vjnl                  Provisioning   v1.18.2
default     capi-quickstart-md-0-5477855897-4xdd9                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-vh4lr                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-z5r6s                Pending        v1.18.2

$ kgp -A
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-8594dfd47d-xqs8x                         2/2     Running   0          17m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-ddcc95784-7rkg5        2/2     Running   0          17m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-d85dfd44c-gb299    2/2     Running   0          17m
capi-system                         capi-controller-manager-5cd969bf55-m6qdj                         2/2     Running   0          17m
capi-webhook-system                 capi-controller-manager-68c9ff8646-rdxqv                         2/2     Running   0          17m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-7cf85756c5-w6hr7       2/2     Running   0          17m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-7854964ff4-jzhr7   2/2     Running   0          17m
cert-manager                        cert-manager-cainjector-fc6c787db-28qcl                          1/1     Running   0          18m
cert-manager                        cert-manager-d994d94d7-rr6xv                                     1/1     Running   0          18m
cert-manager                        cert-manager-webhook-845d9df8bf-nqlbt                            1/1     Running   0          18m
kube-system                         coredns-74ff55c5b-rkjd9                                          1/1     Running   0          18m
kube-system                         coredns-74ff55c5b-w4hht                                          1/1     Running   0          18m
kube-system                         etcd-kind-control-plane                                          1/1     Running   0          18m
kube-system                         kindnet-jl5mn                                                    1/1     Running   0          18m
kube-system                         kube-apiserver-kind-control-plane                                1/1     Running   0          18m
kube-system                         kube-controller-manager-kind-control-plane                       1/1     Running   0          18m
kube-system                         kube-proxy-p5bv9                                                 1/1     Running   0          18m
kube-system                         kube-scheduler-kind-control-plane                                1/1     Running   0          18m
local-path-storage                  local-path-provisioner-78776bfc44-bm97s                          1/1     Running   0          18m

$ klogs -n capd-system capd-controller-manager-8594dfd47d-xqs8x --all-containers -f
...
I0225 02:47:01.144618       1 machine.go:333] controllers/DockerMachine/DockerMachine-controller "msg"="Setting Kubernetes node providerID" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-r586r"} "machine"="capi-quickstart-control-plane-9vjnl"
I0225 02:47:01.175710       1 machine.go:345] controllers/DockerMachine/DockerMachine-controller "msg"="Error response from daemon: Container be73c9696b944369f17314e206481cb8002f0b28253feb6fd5dd4567ce131445 is not running" "cluster"="capi-quickstart" "docker-cluster"="capi-quickstart" "docker-machine"={"Namespace":"default","Name":"capi-quickstart-control-plane-r586r"} "machine"="capi-quickstart-control-plane-9vjnl"
E0225 02:47:01.175811       1 dockermachine_controller.go:324] controllers/DockerMachine "msg"="failed to patch the Kubernetes node with the machine providerID" "error"="failed update providerID: command \"docker exec --privileged capi-quickstart-control-plane-9w29p kubectl --kubeconfig /etc/kubernetes/admin.conf patch node capi-quickstart-control-plane-9vjnl --patch '{\"spec\": {\"providerID\": \"docker:////capi-quickstart-control-plane-9vjnl\"}}'\" failed with error: exit status 1"

I'm quite confused about how to make it work now: 1) kind IS using kind network; 2) CAPD v1aplha3 still use bridge

So how?

brightzheng100 commented 3 years ago

Okay, after looking into it carefully, it works now with some points taken:

For kind version >=0.8, by default it uses a dedicated Docker network named kind:

$ docker network ls
NETWORK ID     NAME                                                               DRIVER    SCOPE
a509eff5df52   bridge                                                             bridge    local
9cd2bd14da33   kind                                                               bridge    local
...

But CAPD v1aplha3 still use the default network named bridge;
So to make it work with kind CLI >=0.8, we must instruct kind to use bridge network, by: export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge, before creating the kind cluster.

Please take note though, if we're with export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge, kind will give us a big warning but actually it still works as of v0.10. But I guess we should make the CAPD smart enough to detect which network kind is with and make it work seamlessly.

$ kind create cluster --config kind-cluster-with-extramounts.yaml
Creating cluster "kind" ...
WARNING: Overriding docker network due to KIND_EXPERIMENTAL_DOCKER_NETWORK
WARNING: Here be dragons! This is not supported currently.
...

fabriziopandini commented 3 years ago

/close given that the problem seems fixed and https://github.com/kubernetes-sigs/cluster-api/pull/4176 is going to move CAPD/v1alpha3 to the kind network

k8s-ci-robot commented 3 years ago

@fabriziopandini: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/4027#issuecomment-795598078): >/close >given that the problem seems fixed and https://github.com/kubernetes-sigs/cluster-api/pull/4176 is going to move CAPD/v1alpha3 to the kind network Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

dscaravaggi commented 1 year ago

21th June 2023 starting from KIND version 0.20.0 and export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge clusterctl version clusterctl version: v1.4.3

    Message:               Scaling up control plane to 3 replicas (actual 1)
    Reason:                ScalingUp
    Severity:              Warning
   ....
  Type    Reason               Age   From                Message
  ----    ------               ----  ----                -------
  Normal  Provisioning         21m   cluster-controller  Cluster test-k8s-2cf is Provisioning
  Normal  InfrastructureReady  21m   cluster-controller  Cluster test-k8s-2cf InfrastructureReady is now true
  Normal  Provisioned          21m   cluster-controller  Cluster test-k8s-2cf is Provisioned
  Normal  ControlPlaneReady    15m   cluster-controller  Cluster test-k8s-2cf ControlPlaneReady is now true

and

NAME                                       STATUS   ROLES           AGE   VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-k8s-2cf-md-0-568df9ddb8x8gwcx-d4jz8   Ready    <none>          18m   v1.25.10   10.224.14.118   <none>        Ubuntu 22.04.2 LTS   5.15.0-75-generic   containerd://1.6.20
test-k8s-2cf-md-0-568df9ddb8x8gwcx-fffk6   Ready    <none>          17m   v1.25.10   10.224.14.119   <none>        Ubuntu 22.04.2 LTS   5.15.0-75-generic   containerd://1.6.20
test-k8s-2cf-md-0-568df9ddb8x8gwcx-fhk4j   Ready    <none>          17m   v1.25.10   10.224.14.120   <none>        Ubuntu 22.04.2 LTS   5.15.0-75-generic   containerd://1.6.20
test-k8s-2cf-mmkx2                         Ready    control-plane   22m   v1.25.10   10.224.14.101   <none>        Ubuntu 22.04.2 LTS   5.15.0-75-generic   containerd://1.6.20

dmseg

Complete use this command to add control plane

  kubeadm join 10.224.14.31:6443 --token qgj7wa.*****redact*** \
     --discovery-token-ca-cert-hash sha256:bb0b63*****redact***ac955fdaf18be035e990 \
     --control-plane

everything seems good but cluster api is not scaling to 3 control-planes

kubernetes-sigs / cluster-api

Can't have 3 master nodes when following 'Quick Start' in book #4027

I put some logs below: