search

airshipit / airshipctl

A CLI for managing declarative infrastructure.
Apache License 2.0
43 stars 49 forks source link

Phase run control plane gets stuck with docker provider #413

Closed rkjain closed 3 years ago

rkjain commented 4 years ago

Description

airshipctl phase run control-plane with docker provider is not working with the latest code. It gets stuck while deploying control plane. However, this is working fine, if I checkout a commit from Nov 10 (9c9369088c26e2fc7cea200e78a2966a3c173d68) - "Change image tags"

Steps To Reproduce

airshipctl phase run controlplane-ephemeral --debug --kubeconfig ~/.airship/kubeconfig

[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:129: Getting kubeconfig context name from cluster map
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:134: Getting kubeconfig file information from kubeconfig provider
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:139: Filtering out documents that shouldn't be applied to kubernetes from document bundle
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:147: Using kubeconfig at '/home/rishabh/.airship/kubeconfig' and context 'ephemeral-cluster'
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:118: WaitTimeout: 33m20s
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:76: Getting infos for bundle, inventory id is controlplane-ephemeral
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:106: Inventory Object config Map not found, auto generating Inventory object
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:113: Injecting Inventory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"controlplane-ephemeral"},"name":"airshipit-controlplane-ephemeral","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
[airshipctl] 2020/11/13 10:37:36 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:119: Making sure that inventory object namespace airshipit exists
cluster.cluster.x-k8s.io/target-cluster created
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane created
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 created
5 resource(s) applied. 5 created, 0 unchanged, 0 configured
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is NotFound: Resource not found
cluster.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
cluster.cluster.x-k8s.io/target-cluster is InProgress: 
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is Current: Resource is current
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is Current: Resource is current

Gets stuck here .. After some time throws this

[airshipctl] 2020/11/13 16:54:06 opendev.org/airship/airshipctl/pkg/events/processor.go:77: Received error when applying errors to kubernetes timeout after 2000 seconds waiting for 5 resources to reach condition AllCurrent
Error events received on channel, errors are:
[timeout after 2000 seconds waiting for 5 resources to reach condition AllCurrent]

Expected behavior

airshipctl phase run controlplane-ephemeral --debug --kubeconfig ~/.airship/kubeconfig

❯ airshipctl phase run controlplane-ephemeral --debug --kubeconfig ~/.airship/kubeconfig
[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/executor.go:126: Getting kubeconfig file information from kubeconfig provider[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/executor.go:131: Filtering out documents that shouldn't be applied to kubernetes from document bundle
[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/executor.go:115: WaitTimeout: 33m20s
[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/applier.go:76: Getting infos for bundle, inventory id is controlplane-ephemeral
[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/applier.go:106: Inventory Object config Map not found, auto generating Inventory object
[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/applier.go:113: Injecting Inventory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"controlplane-ephemeral"},"name":"airshipit-controlplane-ephemeral","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
[airshipctl] 2020/11/05 16:31:53 opendev.org/airship/airshipctl/pkg/k8s/applier/applier.go:119: Making sure that inventory object namespace airshipit exists
cluster.cluster.x-k8s.io/target-cluster created
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane created
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 created
5 resource(s) applied. 5 created, 0 unchanged, 0 configured
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is NotFound: Resource not found
cluster.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
cluster.cluster.x-k8s.io/target-cluster is InProgress: 
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is Current: Resource is Ready
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is Current: Resource is current
cluster.cluster.x-k8s.io/target-cluster is InProgress: Scaling up to 1 replicas (actual 0)
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is InProgress: Scaling up to 1 replicas (actual 0)
cluster.cluster.x-k8s.io/target-cluster is Current: Resource is Ready
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is Ready
all resources has reached the Current status

❯ kubectl get machines --kubeconfig ~/.airship/kubeconfig
NAME                                 PROVIDERID                                      PHASE
target-cluster-control-plane-d9pgc   docker:////target-cluster-control-plane-d9pgc   Running

Also check: https://hackmd.io/poGGfCNGTVeR8Yr28MYXNA for using docker provider.

teoyaomiqui commented 4 years ago

@rkjain This means that some resources are not reaching desired state, can you please paste output describing all objects in kubernetes that are delivered in this phase?

rkjain commented 4 years ago

Yeah. Here you go ❯ airshipctl phase render controlplane-ephemeral

---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
metadata:
  name: target-cluster
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 172.17.0.0/16
    serviceDomain: cluster.local
    services:
      cidrBlocks:
      - 10.0.0.0/24
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: KubeadmControlPlane
    name: target-cluster-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: DockerCluster
    name: target-cluster
...
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineHealthCheck
metadata:
  name: target-cluster-mhc-0
spec:
  clusterName: target-cluster
  maxUnhealthy: 100%
  selector:
    matchLabels:
      nodepool: pool1
  unhealthyConditions:
  - status: "True"
    timeout: 30s
    type: E2ENodeUnhealthy
...
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: target-cluster-control-plane
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: DockerMachineTemplate
    name: target-cluster-control-plane
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        certSANs:
        - localhost
        - 127.0.0.1
      controllerManager:
        extraArgs:
          enable-hostpath-provisioner: "true"
    files:
    - content: |
        #!/bin/sh -x
        su - root -c "sleep 10; kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f https://docs.projectcalico.org/v3.12/manifests/calico.yaml"
      owner: root:root
      path: /calico.sh
      permissions: "0755"
    initConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
    joinConfiguration:
      nodeRegistration:
        criSocket: /var/run/containerd/containerd.sock
        kubeletExtraArgs:
          eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
    postKubeadmCommands:
    - sh /calico.sh
  replicas: 1
  version: v1.18.6
...
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: DockerCluster
metadata:
  name: target-cluster
...
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: DockerMachineTemplate
metadata:
  name: target-cluster-control-plane
spec:
  template:
    spec:
      extraMounts:
      - containerPath: /var/run/docker.sock
        hostPath: /var/run/docker.sock
…
teoyaomiqui commented 4 years ago

@rkjain i mean kubectl describe and kubectl get on those objects

rkjain commented 4 years ago 
❯ kubectl get pods -A --kubeconfig ~/.airship/kubeconfig
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-5498cfdfb5-qmt2t                         2/2     Running   0          39m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-556678c94-vndvn        2/2     Running   0          40m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-556d47dffd-vtc6q   2/2     Running   0          39m
capi-system                         capi-controller-manager-67859f6b78-k2dwf                         2/2     Running   0          40m
capi-webhook-system                 capi-controller-manager-5c785c685c-9zw9m                         2/2     Running   0          40m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-77658d7745-8bw2g       2/2     Running   0          40m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-74dcf8b9c-792jg    2/2     Running   0          40m
cert-manager                        cert-manager-578cd6d964-x4zdx                                    1/1     Running   0          40m
cert-manager                        cert-manager-cainjector-5ffff9dd7c-tn949                         1/1     Running   0          40m
cert-manager                        cert-manager-webhook-556b9d7dfd-c9p6j                            1/1     Running   0          40m
kube-system                         coredns-66bff467f8-fdl46                                         1/1     Running   0          41m
kube-system                         coredns-66bff467f8-tfhxq                                         1/1     Running   0          41m
kube-system                         etcd-ephemeral-cluster-control-plane                             1/1     Running   0          41m
kube-system                         kindnet-g7pqw                                                    1/1     Running   0          41m
kube-system                         kube-apiserver-ephemeral-cluster-control-plane                   1/1     Running   0          41m
kube-system                         kube-controller-manager-ephemeral-cluster-control-plane          1/1     Running   0          41m
kube-system                         kube-proxy-cqq24                                                 1/1     Running   0          41m
kube-system                         kube-scheduler-ephemeral-cluster-control-plane                   1/1     Running   0          41m
local-path-storage                  local-path-provisioner-bd4bb6b75-l7v6q                           1/1     Running   0          41m

❯ kubectl describe pod capd-controller-manager-5498cfdfb5-qmt2t -n capd-system --kubeconfig ~/.airship/kubeconfig
Name:         capd-controller-manager-5498cfdfb5-qmt2t
Namespace:    capd-system
Priority:     0
Node:         ephemeral-cluster-control-plane/172.17.0.2
Start Time:   Fri, 13 Nov 2020 15:41:18 -0800
Labels:       cluster.x-k8s.io/provider=infrastructure-docker
              control-plane=controller-manager
              pod-template-hash=5498cfdfb5
Annotations:  <none>
Status:       Running
IP:           10.244.0.14
IPs:
  IP:           10.244.0.14
Controlled By:  ReplicaSet/capd-controller-manager-5498cfdfb5
Containers:
  manager:
    Container ID:  containerd://2a6589929b970653b1fe16413ddf9f4589353298e55a0a9bf6cbd5c9d6f35e74
    Image:         gcr.io/k8s-staging-cluster-api/capd-manager:v20201019-v0.3.10-86-gc1647481f
    Image ID:      gcr.io/k8s-staging-cluster-api/capd-manager@sha256:863b380e403f2c89ef5af1e4cb5e1cf8bb4f3e997623236fa08ff1785a0bf486
    Ports:         9440/TCP, 9443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --feature-gates=MachinePool=false
      --metrics-addr=0
      -v=4
    State:          Running
      Started:      Fri, 13 Nov 2020 15:41:46 -0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:healthz/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:healthz/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/docker.sock from dockersock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jcfnv (ro)
  kube-rbac-proxy:
    Container ID:  containerd://14c2b60af6d8208293d71386d7d82bab3402abb322f73e40b516a3cd44c8b17e
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0
    Image ID:      gcr.io/kubebuilder/kube-rbac-proxy@sha256:297896d96b827bbcb1abd696da1b2d81cab88359ac34cce0e8281f266b4e08de
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Running
      Started:      Fri, 13 Nov 2020 15:41:50 -0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jcfnv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  dockersock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  capd-webhook-service-cert
    Optional:    false
  default-token-jcfnv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jcfnv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                                      Message
  ----     ------       ----               ----                                      -------
  Normal   Scheduled    40m                default-scheduler                         Successfully assigned capd-system/capd-controller-manager-5498cfdfb5-qmt2t to ephemeral-cluster-control-plane
  Warning  FailedMount  40m (x3 over 40m)  kubelet, ephemeral-cluster-control-plane  MountVolume.SetUp failed for volume "cert" : secret "capd-webhook-service-cert" not found
  Normal   Pulling      40m                kubelet, ephemeral-cluster-control-plane  Pulling image "gcr.io/k8s-staging-cluster-api/capd-manager:v20201019-v0.3.10-86-gc1647481f"
  Normal   Pulled       39m                kubelet, ephemeral-cluster-control-plane  Successfully pulled image "gcr.io/k8s-staging-cluster-api/capd-manager:v20201019-v0.3.10-86-gc1647481f"  Normal   Created      39m                kubelet, ephemeral-cluster-control-plane  Created container manager
  Normal   Started      39m                kubelet, ephemeral-cluster-control-plane  Started container manager
  Normal   Pulling      39m                kubelet, ephemeral-cluster-control-plane  Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0"
  Normal   Pulled       39m                kubelet, ephemeral-cluster-control-plane  Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0"
  Normal   Created      39m                kubelet, ephemeral-cluster-control-plane  Created container kube-rbac-proxy
  Normal   Started      39m                kubelet, ephemeral-cluster-control-plane  Started container kube-rbac-proxy

❯ kubectl describe pod kube-apiserver-ephemeral-cluster-control-plane -n kube-system --kubeconfig ~/.airship/kubeconfig
Name:                 kube-apiserver-ephemeral-cluster-control-plane
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ephemeral-cluster-control-plane/172.17.0.2
Start Time:           Fri, 13 Nov 2020 15:39:23 -0800
Labels:               component=kube-apiserver
                      tier=control-plane
Annotations:          kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.17.0.2:6443
                      kubernetes.io/config.hash: 3e826fb52b98b0d6da81c1be034f87c1
                      kubernetes.io/config.mirror: 3e826fb52b98b0d6da81c1be034f87c1
                      kubernetes.io/config.seen: 2020-11-13T23:39:17.32551873Z
                      kubernetes.io/config.source: file
Status:               Running
IP:                   172.17.0.2
IPs:
  IP:           172.17.0.2
Controlled By:  Node/ephemeral-cluster-control-plane
Containers:
  kube-apiserver:
    Container ID:  containerd://6048b7f849b9873ac143cc6deaa76616bbe2aa8f141e7feba483bffe640f1fc7
    Image:         k8s.gcr.io/kube-apiserver:v1.18.2
    Image ID:      sha256:7df05884b1e25c456468f037faa511ba004b2dfab13be4d204fc47524b917cef
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --advertise-address=172.17.0.2
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Running
      Started:      Fri, 13 Nov 2020 15:39:06 -0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        250m
    Liveness:     http-get https://172.17.0.2:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ca-certificates from etc-ca-certificates (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  etc-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  usr-local-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:            <none>

❯ kubectl logs capd-controller-manager-5498cfdfb5-qmt2t -n capd-system -f --kubeconfig ~/.airship/kubeconfig -c manager
I1113 23:41:47.609708       1 request.go:645] Throttling request took 1.018977954s, request: GET:https://10.96.0.1:443/apis/cert-manager.io/v1alpha2?timeout=32s
I1113 23:41:47.614725       1 webhook.go:93] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"infrastructure.cluster.x-k8s.io","Version":"v1alpha3","Kind":"DockerMachineTemplate"}
I1113 23:41:47.614799       1 webhook.go:124] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"infrastructure.cluster.x-k8s.io","Version":"v1alpha3","Kind":"DockerMachineTemplate"} "path"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate"
I1113 23:41:47.614998       1 server.go:130] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" 
I1113 23:41:47.615464       1 main.go:100] setup "msg"="starting manager"  
I1113 23:41:47.615542       1 server.go:179] controller-runtime/webhook/webhooks "msg"="starting webhook server"  
I1113 23:41:47.615983       1 certwatcher.go:128] controller-runtime/certwatcher "msg"="Updated current TLS certificate"  
I1113 23:41:47.616262       1 server.go:222] controller-runtime/webhook "msg"="serving webhook server"  "host"="" "port"=9443
I1113 23:41:47.616430       1 controller.go:158] controller/dockermachine "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerMachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"ready":false}}}
I1113 23:41:47.616428       1 controller.go:158] controller/dockercluster "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerCluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"ready":false}}}
I1113 23:41:47.616465       1 certwatcher.go:84] controller-runtime/certwatcher "msg"="Starting certificate watcher"  
I1113 23:41:47.731177       1 controller.go:158] controller/dockermachine "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerMachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I1113 23:41:47.731202       1 controller.go:158] controller/dockercluster "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerCluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I1113 23:41:47.832548       1 controller.go:165] controller/dockercluster "msg"="Starting Controller" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerCluster" 
I1113 23:41:47.832581       1 controller.go:158] controller/dockermachine "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerMachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"ready":false}}}
I1113 23:41:47.832652       1 controller.go:158] controller/dockermachine "msg"="Starting EventSource" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerMachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I1113 23:41:47.832679       1 controller.go:165] controller/dockermachine "msg"="Starting Controller" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerMachine" 
I1113 23:41:47.832711       1 controller.go:192] controller/dockercluster "msg"="Starting workers" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerCluster" "worker count"=1
I1113 23:41:47.832775       1 controller.go:192] controller/dockermachine "msg"="Starting workers" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerMachine" "worker count"=10
I1114 00:20:46.393143       1 cluster_predicates.go:72]  "msg"="Cluster is not paused, allowing further processing" "cluster"="target-cluster" "eventType"="create" "namespace"="default" "predicate"="ClusterCreateNotPaused" 
I1114 00:20:46.393185       1 cluster_predicates.go:47]  "msg"="Cluster infrastructure is not ready, blocking further processing" "cluster"="target-cluster" "eventType"="create" "namespace"="default" "predicate"="ClusterCreateInfraReady" 
I1114 00:20:46.393201       1 generic_predicates.go:48]  "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" 
I1114 00:20:46.393216       1 generic_predicates.go:102]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:46.393348       1 generic_predicates.go:102]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:46.393505       1 cluster_predicates.go:72] controllers/DockerCluster "msg"="Cluster is not paused, allowing further processing" "cluster"="target-cluster" "eventType"="create" "namespace"="default" "predicate"="ClusterCreateNotPaused" 
I1114 00:20:46.393526       1 generic_predicates.go:98] controllers/DockerCluster "msg"="One of the provided predicates returned true, allowing further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" 
I1114 00:20:46.400344       1 generic_predicates.go:167]  "msg"="Resource is not paused, will attempt to map resource" ""="target-cluster" "namespace"="default" "predicate"="createEvent" 
I1114 00:20:46.400925       1 generic_predicates.go:167] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="target-cluster" "namespace"="default" "predicate"="createEvent" 
I1114 00:20:46.401750       1 dockercluster_controller.go:68] controller/dockercluster "msg"="Waiting for Cluster Controller to set OwnerRef on DockerCluster" "name"="target-cluster" "namespace"="default" "reconcilerGroup"="infrastructure.cluster.x-k8s.io" "reconcilerKind"="DockerCluster" 
I1114 00:20:46.424735       1 http.go:84] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="066a6f54-0879-4825-b982-70ece5b21d32" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","kind":"DockerMachineTemplate"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1alpha3","resource":"dockermachinetemplates"}
I1114 00:20:46.425622       1 http.go:106] controller-runtime/webhook/webhooks "msg"="wrote response" "code"=200 "reason"="" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1alpha3-dockermachinetemplate" "UID"="066a6f54-0879-4825-b982-70ece5b21d32" "allowed"=true
I1114 00:20:46.533004       1 generic_predicates.go:37]  "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" 
I1114 00:20:46.533048       1 cluster_predicates.go:138]  "msg"="Cluster was not unpaused, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" 
I1114 00:20:46.533249       1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" 
I1114 00:20:46.533268       1 generic_predicates.go:91] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" 
I1114 00:20:46.533363       1 cluster_predicates.go:107]  "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" 
I1114 00:20:46.533383       1 generic_predicates.go:91]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:46.533466       1 generic_predicates.go:91]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:46.556799       1 generic_predicates.go:37]  "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" 
I1114 00:20:46.556821       1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" 
I1114 00:20:46.567187       1 generic_predicates.go:91] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" 
I1114 00:20:46.567201       1 cluster_predicates.go:138]  "msg"="Cluster was not unpaused, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" 
I1114 00:20:46.567234       1 cluster_predicates.go:107]  "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" 
I1114 00:20:46.567256       1 generic_predicates.go:91]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:46.567273       1 generic_predicates.go:91]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:47.915337       1 generic_predicates.go:167]  "msg"="Resource is not paused, will attempt to map resource" ""="target-cluster" "namespace"="default" "predicate"="updateEvent" 
I1114 00:20:47.915439       1 generic_predicates.go:167] controllers/DockerCluster "msg"="Resource is not paused, will attempt to map resource" ""="target-cluster" "namespace"="default" "predicate"="updateEvent" 
I1114 00:20:48.088122       1 generic_predicates.go:37]  "msg"="One of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="All" 
I1114 00:20:48.088159       1 cluster_predicates.go:138]  "msg"="Cluster was not unpaused, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" 
I1114 00:20:48.088179       1 cluster_predicates.go:107]  "msg"="Cluster infrastructure did not become ready, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateInfraReady" 
I1114 00:20:48.088192       1 generic_predicates.go:91]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:48.088271       1 generic_predicates.go:91]  "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpausedAndInfrastructureReady" "predicateAggregation"="Any" 
I1114 00:20:48.088308       1 cluster_predicates.go:138] controllers/DockerCluster "msg"="Cluster was not unpaused, blocking further processing" "cluster"="target-cluster" "eventType"="update" "namespace"="default" "predicate"="ClusterUpdateUnpaused" 
I1114 00:20:48.088329       1 generic_predicates.go:91] controllers/DockerCluster "msg"="All of the provided predicates returned false, blocking further processing" "predicate"="ClusterUnpaused" "predicateAggregation"="Any" 

❯ kubectl logs cert-manager-578cd6d964-x4zdx -n cert-manager
I1113 23:41:02.171520       1 start.go:74] cert-manager "msg"="starting controller"  "git-commit"="a7f8065abe6b74cbed35e79e4d646a38eb7ad6ae" "version"="v0.16.1"
W1113 23:41:02.171594       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1113 23:41:02.172948       1 controller.go:162] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.96.0.10:53"] 
I1113 23:41:02.173904       1 controller.go:125] cert-manager/controller "msg"="starting leader election"  
I1113 23:41:02.174260       1 metrics.go:160] cert-manager/controller/build-context/metrics "msg"="listening for connections on" "address"={"IP":"::","Port":9402,"Zone":""} 
I1113 23:41:02.174845       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-controller...
I1113 23:41:02.189086       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-controller
I1113 23:41:02.189334       1 controller.go:167] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="venafi"
I1113 23:41:02.189618       1 controller.go:167] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="ca"
I1113 23:41:02.190198       1 controller.go:167] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="acme"
I1113 23:41:02.190979       1 controller.go:167] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="selfsigned"
I1113 23:41:02.191620       1 controller.go:167] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="vault"
I1113 23:41:02.194996       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="CertificateKeyManager" 
I1113 23:41:02.195021       1 controller.go:100] cert-manager/controller/CertificateKeyManager "msg"="starting control loop"  
I1113 23:41:02.195560       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I1113 23:41:02.195601       1 controller.go:100] cert-manager/controller/certificaterequests-issuer-acme "msg"="starting control loop"  
I1113 23:41:02.195904       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I1113 23:41:02.195929       1 controller.go:100] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="starting control loop"  
I1113 23:41:02.196073       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I1113 23:41:02.196090       1 controller.go:100] cert-manager/controller/orders "msg"="starting control loop"  
I1113 23:41:02.196226       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I1113 23:41:02.196250       1 controller.go:100] cert-manager/controller/certificaterequests-issuer-vault "msg"="starting control loop"  
I1113 23:41:02.196756       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="CertificateIssuing" 
I1113 23:41:02.196780       1 controller.go:100] cert-manager/controller/CertificateIssuing "msg"="starting control loop"  
I1113 23:41:02.197185       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="CertificateReadiness" 
I1113 23:41:02.197208       1 controller.go:100] cert-manager/controller/CertificateReadiness "msg"="starting control loop"  
I1113 23:41:02.198801       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I1113 23:41:02.198873       1 controller.go:100] cert-manager/controller/clusterissuers "msg"="starting control loop"  
I1113 23:41:02.198911       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="CertificateRequestManager" 
I1113 23:41:02.198921       1 controller.go:100] cert-manager/controller/CertificateRequestManager "msg"="starting control loop"  
I1113 23:41:02.198936       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I1113 23:41:02.198944       1 controller.go:100] cert-manager/controller/ingress-shim "msg"="starting control loop"  
I1113 23:41:02.198959       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I1113 23:41:02.198968       1 controller.go:100] cert-manager/controller/certificaterequests-issuer-venafi "msg"="starting control loop"  
I1113 23:41:02.198986       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I1113 23:41:02.199044       1 controller.go:100] cert-manager/controller/certificaterequests-issuer-ca "msg"="starting control loop"  
I1113 23:41:02.199070       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="CertificateTrigger" 
I1113 23:41:02.199081       1 controller.go:100] cert-manager/controller/CertificateTrigger "msg"="starting control loop"  
I1113 23:41:02.199141       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="CertificateMetrics" 
I1113 23:41:02.199166       1 controller.go:100] cert-manager/controller/CertificateMetrics "msg"="starting control loop"  
I1113 23:41:02.199169       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I1113 23:41:02.199189       1 controller.go:100] cert-manager/controller/issuers "msg"="starting control loop"  
I1113 23:41:02.199239       1 reflector.go:175] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.299832       1 reflector.go:175] Starting reflector *v1.Service (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.300015       1 reflector.go:175] Starting reflector *v1alpha2.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.300066       1 reflector.go:175] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.300333       1 reflector.go:175] Starting reflector *v1beta1.Ingress (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.300556       1 controller.go:99] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I1113 23:41:02.300573       1 controller.go:100] cert-manager/controller/challenges "msg"="starting control loop"  
I1113 23:41:02.300654       1 reflector.go:175] Starting reflector *v1alpha2.Issuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.300869       1 reflector.go:175] Starting reflector *v1alpha2.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.300997       1 reflector.go:175] Starting reflector *v1alpha2.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.301123       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.301373       1 reflector.go:175] Starting reflector *v1alpha2.Challenge (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:02.301590       1 reflector.go:175] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I1113 23:41:04.667473       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="cert-manager-test/test-selfsigned" 
I1113 23:41:04.667797       1 conditions.go:92] Setting lastTransitionTime for Issuer "test-selfsigned" condition "Ready" to 2020-11-13 23:41:04.667781104 +0000 UTC m=+2.525206228
I1113 23:41:04.725683       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="cert-manager-test/test-selfsigned" 
I1113 23:41:04.726418       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="cert-manager-test/test-selfsigned" 
I1113 23:41:04.726740       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="cert-manager-test/test-selfsigned" 
I1113 23:41:04.764699       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.764747       1 conditions.go:173] Setting lastTransitionTime for Certificate "selfsigned-cert" condition "Ready" to 2020-11-13 23:41:04.764742552 +0000 UTC m=+2.622167692
I1113 23:41:04.765425       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765460       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765484       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765510       1 conditions.go:173] Setting lastTransitionTime for Certificate "selfsigned-cert" condition "Issuing" to 2020-11-13 23:41:04.765501618 +0000 UTC m=+2.622926744
I1113 23:41:04.765777       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765829       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765892       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765940       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765958       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.765980       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.884839       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.885034       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.885133       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.893175       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.893351       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.894026       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.894115       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.956375       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.956425       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.956467       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:04.960883       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"selfsigned-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.960927       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:04.961305       1 readiness_controller.go:130] cert-manager/controller/CertificateReadiness "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.961344       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.961522       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.964248       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:04.996789       1 requestmanager_controller.go:127] cert-manager/controller/CertificateRequestManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:04.996731       1 readiness_controller.go:130] cert-manager/controller/CertificateReadiness "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.997084       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.964305       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="cert-manager-test/test-selfsigned" 
I1113 23:41:04.996838       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.964369       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.997774       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.964416       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:04.997927       1 controller.go:133] cert-manager/controller/issuers "msg"="issuer in work queue no longer exists" "error"="issuer.cert-manager.io \"test-selfsigned\" not found"  
I1113 23:41:04.997951       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="cert-manager-test/test-selfsigned" 
E1113 23:41:04.998177       1 trigger_controller.go:142] cert-manager/controller/CertificateTrigger "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.964452       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.998491       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:04.986299       1 event.go:260] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"selfsigned-cert.164735b9f3cbf0c2", GenerateName:"", Namespace:"cert-manager-test", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Certificate", Namespace:"cert-manager-test", Name:"selfsigned-cert", UID:"e0c3bc08-e239-4165-acb8-1713ac56e357", APIVersion:"cert-manager.io/v1alpha2", ResourceVersion:"895", FieldPath:""}, Reason:"Issuing", Message:"Issuing certificate as Secret does not exist", Source:v1.EventSource{Component:"cert-manager", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfe3e444390050c2, ext:2813747128, loc:(*time.Location)(0x305ee00)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfe3e444390050c2, ext:2813747128, loc:(*time.Location)(0x305ee00)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "selfsigned-cert.164735b9f3cbf0c2" is forbidden: unable to create new content in namespace cert-manager-test because it is being terminated' (will not retry!)
E1113 23:41:04.998746       1 issuing_controller.go:152] cert-manager/controller/CertificateIssuing "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:04.998778       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:05.961125       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:05.961402       1 readiness_controller.go:130] cert-manager/controller/CertificateReadiness "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:05.961613       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:06.017853       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="secrets \"selfsigned-cert-\" is forbidden: unable to create new content in namespace cert-manager-test because it is being terminated" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:06.017992       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:06.018208       1 keymanager_controller.go:137] cert-manager/controller/CertificateKeyManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:06.018490       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:06.115589       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.116043       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-serving-cert" condition "Ready" to 2020-11-13 23:41:06.116032678 +0000 UTC m=+3.973457813
I1113 23:41:06.117940       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.118131       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.118178       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.118221       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-serving-cert" condition "Issuing" to 2020-11-13 23:41:06.118214361 +0000 UTC m=+3.975639526
I1113 23:41:06.118357       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.118399       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.118423       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.118462       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.119186       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.119711       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.225195       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.225302       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.226044       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.226090       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.226108       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.226137       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.226170       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.226190       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.239414       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.239564       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
E1113 23:41:06.248251       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.248358       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.248419       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-serving-cert" condition "Ready" to 2020-11-13 23:41:06.248411439 +0000 UTC m=+4.105836569
I1113 23:41:06.279053       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.279287       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.279542       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.279747       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.279920       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.280171       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.280366       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.284523       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.285500       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capi-webhook-system/capi-selfsigned-issuer" 
I1113 23:41:06.309141       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.309229       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.336250       1 conditions.go:92] Setting lastTransitionTime for Issuer "capi-selfsigned-issuer" condition "Ready" to 2020-11-13 23:41:06.336236259 +0000 UTC m=+4.193661385
I1113 23:41:06.385020       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.420620       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capi-webhook-system/capi-selfsigned-issuer" 
I1113 23:41:06.420682       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capi-webhook-system/capi-selfsigned-issuer" 
I1113 23:41:06.421228       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capi-webhook-system/capi-selfsigned-issuer" 
I1113 23:41:06.872008       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.873128       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
E1113 23:41:06.908165       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:06.908229       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.006100       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.006766       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.007394       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.014669       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.014718       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.014744       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.014777       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.014804       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.025936       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.048798       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.049166       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.050927       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="cert-manager-test/selfsigned-cert" 
E1113 23:41:07.051226       1 keymanager_controller.go:137] cert-manager/controller/CertificateKeyManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"selfsigned-cert\" not found" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:07.051363       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="cert-manager-test/selfsigned-cert" 
I1113 23:41:07.148429       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.148771       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.149014       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.149213       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.149351       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.149513       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.149669       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.149853       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.149951       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.150271       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.151169       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.151590       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.151718       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.160082       1 selfsigned.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned/sign "msg"="self signed certificate issued" "resource_kind"="CertificateRequest" "resource_name"="capi-serving-cert-2cz75" "resource_namespace"="capi-webhook-system" 
I1113 23:41:07.166727       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "capi-serving-cert-2cz75" condition "Ready" to 2020-11-13 23:41:07.166715725 +0000 UTC m=+5.024140860
I1113 23:41:07.248228       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.248281       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.256215       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.256313       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.285739       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.361438       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.363368       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.363548       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.367929       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.367673       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.368019       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.367776       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.368647       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.367801       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.368736       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.368772       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.369022       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.367825       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.370542       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.369050       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.370573       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert-2cz75" 
I1113 23:41:07.426308       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.426357       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.427096       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.427120       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.427151       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.427286       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.541919       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.542968       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.549243       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.560723       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.571896       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.573084       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.573460       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.573565       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.573612       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
E1113 23:41:07.678276       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.678335       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.682036       1 conditions.go:162] Found status change for Certificate "capi-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:07.682023072 +0000 UTC m=+5.539448206
I1113 23:41:07.782833       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.782948       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
E1113 23:41:07.929645       1 controller.go:158] cert-manager/controller/CertificateIssuing "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.929711       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.929764       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.954634       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.954675       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.962178       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.962671       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.962963       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.963259       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.963498       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.963641       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.963875       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:07.981386       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
E1113 23:41:08.038793       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.038917       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.039997       1 conditions.go:162] Found status change for Certificate "capi-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:08.03998477 +0000 UTC m=+5.897409908
E1113 23:41:08.140989       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.141223       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.141282       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.358886       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.359793       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.361759       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.361825       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capi-webhook-system/capi-serving-cert" "duration_until_renewal"="1439h59m58.638200467s"
I1113 23:41:08.362323       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362376       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362397       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362432       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362479       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362501       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362531       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362556       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.362593       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.776220       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.776260       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.848594       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.974979       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:08.975176       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:09.143153       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:09.143509       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-serving-cert" 
I1113 23:41:09.409017       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.409758       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-kubeadm-bootstrap-serving-cert" condition "Ready" to 2020-11-13 23:41:09.40914567 +0000 UTC m=+7.266570809
I1113 23:41:09.410549       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.411221       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.412541       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.413153       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-kubeadm-bootstrap-serving-cert" condition "Issuing" to 2020-11-13 23:41:09.412625483 +0000 UTC m=+7.270050611
I1113 23:41:09.414029       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.414205       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.414937       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.415061       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.415118       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.415181       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484132       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484179       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484219       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484253       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484276       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484303       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484386       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.484423       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.485626       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.485715       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.512774       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-selfsigned-issuer" 
I1113 23:41:09.512974       1 conditions.go:92] Setting lastTransitionTime for Issuer "capi-kubeadm-bootstrap-selfsigned-issuer" condition "Ready" to 2020-11-13 23:41:09.512968074 +0000 UTC m=+7.370393204
E1113 23:41:09.618696       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-bootstrap-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.618749       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.618795       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-kubeadm-bootstrap-serving-cert" condition "Issuing" to 2020-11-13 23:41:09.618789162 +0000 UTC m=+7.476214305
I1113 23:41:09.644659       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.649239       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-selfsigned-issuer" 
I1113 23:41:09.649488       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-selfsigned-issuer" 
I1113 23:41:09.649986       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-selfsigned-issuer" 
I1113 23:41:09.672226       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.675166       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.679784       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.681358       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.681899       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.682218       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.693610       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.693646       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.693679       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.696091       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.696922       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:09.796874       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:10.792999       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:10.793329       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.170077       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.170181       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.261017       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.262495       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.262527       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.262834       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.263041       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.263071       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.343696       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.348223       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.263129       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.348306       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.263098       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.413531       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.413960       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.549716       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.768517       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.788433       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.788525       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.788555       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.791706       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.791784       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.853132       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.871718       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.873283       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.887663       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.888145       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.888796       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.903668       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.905500       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.908417       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.912193       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:11.917052       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.918118       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:11.947061       1 selfsigned.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned/sign "msg"="self signed certificate issued" "resource_kind"="CertificateRequest" "resource_name"="capi-kubeadm-bootstrap-serving-cert-v7g7z" "resource_namespace"="capi-webhook-system" 
I1113 23:41:11.947259       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "capi-kubeadm-bootstrap-serving-cert-v7g7z" condition "Ready" to 2020-11-13 23:41:11.947244337 +0000 UTC m=+9.804669489
I1113 23:41:12.137024       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.137877       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.137919       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.137967       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.138505       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.138654       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.138710       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.138763       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.138806       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.138849       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.138949       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.138999       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.139024       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.139091       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.140043       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.140084       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.140109       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert-v7g7z" 
I1113 23:41:12.238473       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.238630       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.238681       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.238921       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.288582       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.331225       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.331755       1 conditions.go:162] Found status change for Certificate "capi-kubeadm-bootstrap-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:12.331748321 +0000 UTC m=+10.189173452
I1113 23:41:12.596240       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.596257       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.596298       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.596327       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.596765       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.596822       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.596842       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.685082       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.685182       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.685237       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
E1113 23:41:12.923998       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-bootstrap-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.924111       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:12.942665       1 conditions.go:162] Found status change for Certificate "capi-kubeadm-bootstrap-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:12.942654383 +0000 UTC m=+10.800079520
I1113 23:41:13.205906       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.206060       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.214961       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.215077       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.215273       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.215376       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" "duration_until_renewal"="1439h59m57.784699165s"
I1113 23:41:13.216082       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216195       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216310       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216503       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216650       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216757       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216791       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216909       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.216952       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.423974       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.484411       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.497369       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-kubeadm-control-plane-serving-cert" condition "Ready" to 2020-11-13 23:41:13.497357838 +0000 UTC m=+11.354782998
I1113 23:41:13.490117       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.492882       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.492917       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.492959       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.492985       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.511742       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.511842       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.511848       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.511910       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.511946       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-kubeadm-control-plane-serving-cert" condition "Issuing" to 2020-11-13 23:41:13.511940815 +0000 UTC m=+11.369365941
E1113 23:41:13.529364       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-bootstrap-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.529448       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.653977       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.654536       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.655591       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.655650       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.665458       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.665543       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.665601       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.665642       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" "duration_until_renewal"="1439h59m57.334372167s"
I1113 23:41:13.666073       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.666107       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.666132       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.666154       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.666179       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.681241       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.915910       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916290       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916159       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916466       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916192       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916504       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916221       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.916248       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.928112       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.932151       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.938852       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:13.968395       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
E1113 23:41:13.990951       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-control-plane-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.991052       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:13.991105       1 conditions.go:173] Setting lastTransitionTime for Certificate "capi-kubeadm-control-plane-serving-cert" condition "Ready" to 2020-11-13 23:41:13.991100608 +0000 UTC m=+11.848525733
I1113 23:41:14.022124       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-selfsigned-issuer" 
I1113 23:41:14.023279       1 conditions.go:92] Setting lastTransitionTime for Issuer "capi-kubeadm-control-plane-selfsigned-issuer" condition "Ready" to 2020-11-13 23:41:14.023268275 +0000 UTC m=+11.880693415
I1113 23:41:14.066237       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.067802       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.124365       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-selfsigned-issuer" 
I1113 23:41:14.124519       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-selfsigned-issuer" 
I1113 23:41:14.124926       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-selfsigned-issuer" 
I1113 23:41:14.125015       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125050       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125085       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125110       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125133       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125172       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125294       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.125343       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.167551       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:14.530518       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:14.530585       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-bootstrap-serving-cert" 
I1113 23:41:15.017328       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.186837       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.708253       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.708373       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
E1113 23:41:15.758952       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-control-plane-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.760210       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.872064       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.872441       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.878966       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.879022       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.879049       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.879119       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.879232       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.879396       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.968232       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.968282       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:15.968517       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.038912       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.127628       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.127815       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.127889       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.127922       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.127976       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.128013       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.128073       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.128097       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.135058       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.135095       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.147705       1 selfsigned.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned/sign "msg"="self signed certificate issued" "resource_kind"="CertificateRequest" "resource_name"="capi-kubeadm-control-plane-serving-cert-5p4kt" "resource_namespace"="capi-webhook-system" 
I1113 23:41:16.148099       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "capi-kubeadm-control-plane-serving-cert-5p4kt" condition "Ready" to 2020-11-13 23:41:16.148089084 +0000 UTC m=+14.005514228
I1113 23:41:16.158844       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.159512       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.159562       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.159957       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.160178       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.160993       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.164295       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.228115       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.280983       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.282839       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.283413       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.291833       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.291885       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.291955       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.342491       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.313185       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.342572       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.313219       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.342608       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.313286       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.325625       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.343869       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.325691       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.343922       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.325726       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert-5p4kt" 
I1113 23:41:16.446251       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.446345       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.449384       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.458709       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.592063       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.592118       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.592660       1 conditions.go:162] Found status change for Certificate "capi-kubeadm-control-plane-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:16.592649405 +0000 UTC m=+14.450074540
I1113 23:41:16.674936       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.675058       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.675217       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.747908       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.748062       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.748176       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.752365       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.752998       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.753029       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.753172       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
E1113 23:41:16.794784       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-control-plane-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.794917       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.795280       1 conditions.go:162] Found status change for Certificate "capi-kubeadm-control-plane-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:16.795273937 +0000 UTC m=+14.652699065
I1113 23:41:16.853492       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.854369       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.863955       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.863999       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.864034       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.864057       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" "duration_until_renewal"="1439h59m59.135957205s"
I1113 23:41:16.864507       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.864540       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.864589       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.864630       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:16.864701       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.025309       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.027555       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.027671       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.027759       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.027872       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
E1113 23:41:17.206360       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capi-kubeadm-control-plane-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.207565       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.316331       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.324738       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.324789       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.324820       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.324872       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.324903       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" "duration_until_renewal"="1439h59m58.67511308s"
I1113 23:41:17.325383       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.325419       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.325455       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.325480       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.325560       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.325627       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.325682       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.390205       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:17.839583       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:18.009602       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:18.230343       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:18.230424       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capi-webhook-system/capi-kubeadm-control-plane-serving-cert" 
I1113 23:41:18.966323       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.966908       1 conditions.go:173] Setting lastTransitionTime for Certificate "capd-serving-cert" condition "Ready" to 2020-11-13 23:41:18.966897103 +0000 UTC m=+16.824322253I1113 23:41:18.967774       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.968093       1 conditions.go:173] Setting lastTransitionTime for Certificate "capd-serving-cert" condition "Issuing" to 2020-11-13 23:41:18.968085631 +0000 UTC m=+16.825510782
I1113 23:41:18.968836       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.969002       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.969566       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.969856       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.970344       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.970517       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.970611       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:18.970679       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.033213       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.033467       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.033615       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.036310       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.036250       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.036487       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.036717       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.036808       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.036008       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.042542       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
E1113 23:41:19.053728       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capd-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.054201       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.054345       1 conditions.go:173] Setting lastTransitionTime for Certificate "capd-serving-cert" condition "Issuing" to 2020-11-13 23:41:19.054336743 +0000 UTC m=+16.911761882
I1113 23:41:19.079163       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capd-system/capd-selfsigned-issuer" 
I1113 23:41:19.082033       1 conditions.go:92] Setting lastTransitionTime for Issuer "capd-selfsigned-issuer" condition "Ready" to 2020-11-13 23:41:19.082019319 +0000 UTC m=+16.939444457
E1113 23:41:19.305457       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capd-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.306214       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capd-system/capd-selfsigned-issuer" 
I1113 23:41:19.306655       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.310900       1 controller.go:152] cert-manager/controller/issuers "msg"="syncing item" "key"="capd-system/capd-selfsigned-issuer" 
I1113 23:41:19.313592       1 controller.go:162] cert-manager/controller/issuers "msg"="finished processing work item" "key"="capd-system/capd-selfsigned-issuer" 
I1113 23:41:19.314752       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.316212       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.316441       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.345626       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.345754       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.345874       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.345938       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.346321       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.346411       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.346439       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:19.377814       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:20.062526       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:20.062603       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:20.666404       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:20.689692       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.167399       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.176222       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.191006       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.193833       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.193864       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.196432       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197108       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197270       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197403       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197431       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197551       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197912       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.197949       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.198153       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.198273       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.235837       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.240646       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.247966       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.249271       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.249460       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.257437       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.257631       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.257663       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.258470       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.258651       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.258683       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.260364       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.261310       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.261343       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.261770       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.268698       1 selfsigned.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned/sign "msg"="self signed certificate issued" "resource_kind"="CertificateRequest" "resource_name"="capd-serving-cert-nfmjf" "resource_namespace"="capd-system" 
I1113 23:41:21.272071       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.272480       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "capd-serving-cert-nfmjf" condition "Ready" to 2020-11-13 23:41:21.272472291 +0000 UTC m=+19.129897415
I1113 23:41:21.312723       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.314130       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.314530       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.314583       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.314691       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.314723       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.314855       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.314888       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.314975       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.315011       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.315040       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.315065       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.315148       1 controller.go:152] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.315179       1 controller.go:162] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="capd-system/capd-serving-cert-nfmjf" 
I1113 23:41:21.315208       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.331200       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.331239       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.333047       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.335406       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.335539       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.335747       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.335984       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.352133       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.352323       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.352936       1 conditions.go:162] Found status change for Certificate "capd-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:21.352927773 +0000 UTC m=+19.210352913
I1113 23:41:21.360527       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.360564       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.384416       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.384625       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.384812       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.384961       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.385573       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.386259       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.387265       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
E1113 23:41:21.421776       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capd-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.421830       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.422516       1 conditions.go:162] Found status change for Certificate "capd-serving-cert" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2020-11-13 23:41:21.422506134 +0000 UTC m=+19.279931279
I1113 23:41:21.454815       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.457331       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.459762       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.459797       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.468583       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.468671       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capd-system/capd-serving-cert" "duration_until_renewal"="1439h59m59.531386488s"
I1113 23:41:21.469218       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.469258       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.469952       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.471185       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.471215       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
E1113 23:41:21.571462       1 controller.go:158] cert-manager/controller/CertificateIssuing "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capd-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.571566       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.571653       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.582617       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
E1113 23:41:21.592486       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"capd-serving-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.592530       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.787142       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.789047       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.790722       1 controller.go:152] cert-manager/controller/CertificateRequestManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.790863       1 controller.go:162] cert-manager/controller/CertificateRequestManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.790911       1 controller.go:152] cert-manager/controller/CertificateTrigger "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.790927       1 trigger_controller.go:212] cert-manager/controller/CertificateTrigger "msg"="scheduling renewal" "key"="capd-system/capd-serving-cert" "duration_until_renewal"="1439h59m59.209081088s"
I1113 23:41:21.791651       1 controller.go:162] cert-manager/controller/CertificateTrigger "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.791707       1 controller.go:152] cert-manager/controller/CertificateMetrics "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.791746       1 controller.go:162] cert-manager/controller/CertificateMetrics "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.791858       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.791896       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.791930       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.791958       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:21.976262       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:22.422520       1 controller.go:152] cert-manager/controller/CertificateReadiness "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:22.435531       1 controller.go:162] cert-manager/controller/CertificateReadiness "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:22.630949       1 controller.go:152] cert-manager/controller/CertificateKeyManager "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:22.631039       1 controller.go:162] cert-manager/controller/CertificateKeyManager "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:22.631148       1 controller.go:152] cert-manager/controller/CertificateIssuing "msg"="syncing item" "key"="capd-system/capd-serving-cert" 
I1113 23:41:22.631268       1 controller.go:162] cert-manager/controller/CertificateIssuing "msg"="finished processing work item" "key"="capd-system/capd-serving-cert" 

❯ kubectl describe pod cert-manager-578cd6d964-x4zdx -n cert-manager
Name:         cert-manager-578cd6d964-x4zdx
Namespace:    cert-manager
Priority:     0
Node:         ephemeral-cluster-control-plane/172.17.0.2
Start Time:   Fri, 13 Nov 2020 15:40:42 -0800
Labels:       app=cert-manager
              app.kubernetes.io/component=controller
              app.kubernetes.io/instance=cert-manager
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=cert-manager
              helm.sh/chart=cert-manager-v0.16.1
              pod-template-hash=578cd6d964
Annotations:  prometheus.io/path: /metrics
              prometheus.io/port: 9402
              prometheus.io/scrape: true
Status:       Running
IP:           10.244.0.6
IPs:
  IP:           10.244.0.6
Controlled By:  ReplicaSet/cert-manager-578cd6d964
Containers:
  cert-manager:
    Container ID:  containerd://aba800eb351e93414e591d5aa1fa5edae649dfe8c62a06c16c47e8c25892db11
    Image:         quay.io/jetstack/cert-manager-controller:v0.16.1
    Image ID:      quay.io/jetstack/cert-manager-controller@sha256:ec46154598a4841cc4ca3c8915cf75079743f73fd1686be0c7b154f580fb9ca1
    Port:          9402/TCP
    Host Port:     0/TCP
    Args:
      --v=2
      --cluster-resource-namespace=$(POD_NAMESPACE)
      --leader-election-namespace=kube-system
    State:          Running
      Started:      Fri, 13 Nov 2020 15:41:02 -0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-nb5bk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  cert-manager-token-nb5bk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-token-nb5bk
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                      Message
  ----    ------     ----  ----                                      -------
  Normal  Scheduled  48m   default-scheduler                         Successfully assigned cert-manager/cert-manager-578cd6d964-x4zdx to ephemeral-cluster-control-plane
  Normal  Pulling    48m   kubelet, ephemeral-cluster-control-plane  Pulling image "quay.io/jetstack/cert-manager-controller:v0.16.1"
  Normal  Pulled     48m   kubelet, ephemeral-cluster-control-plane  Successfully pulled image "quay.io/jetstack/cert-manager-controller:v0.16.1"
  Normal  Created    48m   kubelet, ephemeral-cluster-control-plane  Created container cert-manager
  Normal  Started    48m   kubelet, ephemeral-cluster-control-plane  Started container cert-manager

❯ kubectl logs kube-controller-manager-ephemeral-cluster-control-plane -n kube-system
I1113 23:39:07.863894       1 serving.go:313] Generated self-signed cert in-memory
I1113 23:39:08.616292       1 controllermanager.go:161] Version: v1.18.2
I1113 23:39:08.617543       1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.crt
I1113 23:39:08.617588       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
I1113 23:39:08.618512       1 secure_serving.go:178] Serving securely on 127.0.0.1:10257
I1113 23:39:08.618726       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1113 23:39:08.619778       1 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
I1113 23:39:08.619910       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/kube-controller-manager...
E1113 23:39:14.070123       1 leaderelection.go:320] error retrieving resource lock kube-system/kube-controller-manager: endpoints "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "endpoints" in API group "" in the namespace "kube-system"
E1113 23:39:14.092474       1 leaderelection.go:320] error retrieving resource lock kube-system/kube-controller-manager: endpoints "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "endpoints" in API group "" in the namespace "kube-system"
I1113 23:39:18.196198       1 leaderelection.go:252] successfully acquired lease kube-system/kube-controller-manager
I1113 23:39:18.196427       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"92e50d18-1433-475f-836f-13c61d786358", APIVersion:"v1", ResourceVersion:"199", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' ephemeral-cluster-control-plane_c6760b1d-05ef-4977-b08c-85f8bbc475c6 became leader
I1113 23:39:18.196530       1 event.go:278] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"kube-controller-manager", UID:"fe9d5ee7-d0e6-4d0f-9b8c-77f85df8c739", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"200", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' ephemeral-cluster-control-plane_c6760b1d-05ef-4977-b08c-85f8bbc475c6 became leader
I1113 23:39:18.457301       1 plugins.go:100] No cloud provider specified.
I1113 23:39:18.458475       1 shared_informer.go:223] Waiting for caches to sync for tokens
I1113 23:39:18.559568       1 shared_informer.go:230] Caches are synced for tokens 
I1113 23:39:18.834347       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
I1113 23:39:18.834426       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
I1113 23:39:18.834462       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
I1113 23:39:18.834505       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
I1113 23:39:18.834611       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I1113 23:39:18.834654       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
I1113 23:39:18.834738       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I1113 23:39:18.834776       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
I1113 23:39:18.834818       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
I1113 23:39:18.834864       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
I1113 23:39:18.835059       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
I1113 23:39:18.835153       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I1113 23:39:18.835266       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
I1113 23:39:18.836066       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
I1113 23:39:18.836205       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
I1113 23:39:18.836375       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
I1113 23:39:18.836405       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
W1113 23:39:18.836429       1 shared_informer.go:461] resyncPeriod 48682241396965 is smaller than resyncCheckPeriod 51205965949980 and the informer has already started. Changing it to 51205965949980
I1113 23:39:18.836685       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
I1113 23:39:18.836845       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
I1113 23:39:18.837031       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I1113 23:39:18.837236       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
I1113 23:39:18.837376       1 controllermanager.go:533] Started "resourcequota"
I1113 23:39:18.837501       1 resource_quota_controller.go:272] Starting resource quota controller
I1113 23:39:18.837525       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I1113 23:39:18.837649       1 resource_quota_monitor.go:303] QuotaMonitor running
I1113 23:39:19.375113       1 garbagecollector.go:133] Starting garbage collector controller
I1113 23:39:19.375136       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I1113 23:39:19.375351       1 graph_builder.go:282] GraphBuilder running
I1113 23:39:19.375666       1 controllermanager.go:533] Started "garbagecollector"
I1113 23:39:19.401301       1 controllermanager.go:533] Started "csrsigning"
I1113 23:39:19.401596       1 certificate_controller.go:119] Starting certificate controller "csrsigning"
I1113 23:39:19.401643       1 shared_informer.go:223] Waiting for caches to sync for certificate-csrsigning
I1113 23:39:19.401747       1 dynamic_serving_content.go:130] Starting csr-controller::/etc/kubernetes/pki/ca.crt::/etc/kubernetes/pki/ca.key
I1113 23:39:19.430494       1 controllermanager.go:533] Started "pv-protection"
W1113 23:39:19.430529       1 core.go:243] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes.
W1113 23:39:19.430538       1 controllermanager.go:525] Skipping "route"
W1113 23:39:19.430549       1 controllermanager.go:525] Skipping "root-ca-cert-publisher"
I1113 23:39:19.430732       1 pv_protection_controller.go:83] Starting PV protection controller
I1113 23:39:19.430814       1 shared_informer.go:223] Waiting for caches to sync for PV protection
I1113 23:39:19.556761       1 controllermanager.go:533] Started "horizontalpodautoscaling"
I1113 23:39:19.556902       1 horizontal.go:169] Starting HPA controller
I1113 23:39:19.556913       1 shared_informer.go:223] Waiting for caches to sync for HPA
I1113 23:39:19.577113       1 controllermanager.go:533] Started "bootstrapsigner"
I1113 23:39:19.577385       1 shared_informer.go:223] Waiting for caches to sync for bootstrap_signer
I1113 23:39:19.598699       1 node_lifecycle_controller.go:384] Sending events to api server.
I1113 23:39:19.598979       1 taint_manager.go:163] Sending events to api server.
I1113 23:39:19.599239       1 node_lifecycle_controller.go:512] Controller will reconcile labels.
I1113 23:39:19.599443       1 controllermanager.go:533] Started "nodelifecycle"
I1113 23:39:19.599845       1 node_lifecycle_controller.go:546] Starting node controller
I1113 23:39:19.599927       1 shared_informer.go:223] Waiting for caches to sync for taint
I1113 23:39:19.828214       1 controllermanager.go:533] Started "ttl"
I1113 23:39:19.828243       1 ttl_controller.go:118] Starting TTL controller
I1113 23:39:19.828254       1 shared_informer.go:223] Waiting for caches to sync for TTL
I1113 23:39:19.990014       1 request.go:621] Throttling request took 1.052988468s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/apiextensions.k8s.io/v1?timeout=32s
I1113 23:39:20.085688       1 controllermanager.go:533] Started "tokencleaner"
I1113 23:39:20.085732       1 tokencleaner.go:118] Starting token cleaner controller
I1113 23:39:20.085737       1 shared_informer.go:223] Waiting for caches to sync for token_cleaner
I1113 23:39:20.085742       1 shared_informer.go:230] Caches are synced for token_cleaner 
I1113 23:39:20.229291       1 node_lifecycle_controller.go:78] Sending events to api server
E1113 23:39:20.229314       1 core.go:229] failed to start cloud node lifecycle controller: no cloud provider provided
W1113 23:39:20.229322       1 controllermanager.go:525] Skipping "cloud-node-lifecycle"
I1113 23:39:20.483435       1 controllermanager.go:533] Started "serviceaccount"
I1113 23:39:20.484040       1 serviceaccounts_controller.go:117] Starting service account controller
I1113 23:39:20.487089       1 shared_informer.go:223] Waiting for caches to sync for service account
I1113 23:39:20.878530       1 controllermanager.go:533] Started "disruption"
I1113 23:39:20.878649       1 disruption.go:331] Starting disruption controller
I1113 23:39:20.878663       1 shared_informer.go:223] Waiting for caches to sync for disruption
I1113 23:39:21.128850       1 controllermanager.go:533] Started "statefulset"
I1113 23:39:21.128983       1 stateful_set.go:146] Starting stateful set controller
I1113 23:39:21.129004       1 shared_informer.go:223] Waiting for caches to sync for stateful set
I1113 23:39:21.278734       1 node_ipam_controller.go:94] Sending events to api server.
I1113 23:39:31.283378       1 range_allocator.go:82] Sending events to api server.
I1113 23:39:31.283675       1 range_allocator.go:116] No Secondary Service CIDR provided. Skipping filtering out secondary service addresses.
I1113 23:39:31.283723       1 controllermanager.go:533] Started "nodeipam"
I1113 23:39:31.283925       1 node_ipam_controller.go:162] Starting ipam controller
I1113 23:39:31.283949       1 shared_informer.go:223] Waiting for caches to sync for node
E1113 23:39:31.309865       1 core.go:89] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
W1113 23:39:31.309891       1 controllermanager.go:525] Skipping "service"
I1113 23:39:31.336292       1 controllermanager.go:533] Started "persistentvolume-binder"
I1113 23:39:31.336494       1 pv_controller_base.go:295] Starting persistent volume controller
I1113 23:39:31.336525       1 shared_informer.go:223] Waiting for caches to sync for persistent volume
I1113 23:39:31.363894       1 controllermanager.go:533] Started "endpoint"
I1113 23:39:31.364092       1 endpoints_controller.go:182] Starting endpoint controller
I1113 23:39:31.364102       1 shared_informer.go:223] Waiting for caches to sync for endpoint
I1113 23:39:31.393275       1 controllermanager.go:533] Started "daemonset"
I1113 23:39:31.393798       1 daemon_controller.go:257] Starting daemon sets controller
I1113 23:39:31.393829       1 shared_informer.go:223] Waiting for caches to sync for daemon sets
I1113 23:39:31.444515       1 controllermanager.go:533] Started "cronjob"
I1113 23:39:31.445045       1 cronjob_controller.go:97] Starting CronJob Manager
I1113 23:39:31.487414       1 controllermanager.go:533] Started "replicaset"
I1113 23:39:31.489075       1 replica_set.go:181] Starting replicaset controller
I1113 23:39:31.489099       1 shared_informer.go:223] Waiting for caches to sync for ReplicaSet
I1113 23:39:31.501110       1 controllermanager.go:533] Started "csrapproving"
I1113 23:39:31.501525       1 certificate_controller.go:119] Starting certificate controller "csrapproving"
I1113 23:39:31.501546       1 shared_informer.go:223] Waiting for caches to sync for certificate-csrapproving
I1113 23:39:31.687741       1 controllermanager.go:533] Started "clusterrole-aggregation"
I1113 23:39:31.687814       1 clusterroleaggregation_controller.go:149] Starting ClusterRoleAggregator
I1113 23:39:31.687828       1 shared_informer.go:223] Waiting for caches to sync for ClusterRoleAggregator
I1113 23:39:31.939088       1 controllermanager.go:533] Started "replicationcontroller"
I1113 23:39:31.939127       1 replica_set.go:181] Starting replicationcontroller controller
I1113 23:39:31.939480       1 shared_informer.go:223] Waiting for caches to sync for ReplicationController
I1113 23:39:32.197198       1 controllermanager.go:533] Started "namespace"
I1113 23:39:32.197273       1 namespace_controller.go:200] Starting namespace controller
I1113 23:39:32.197280       1 shared_informer.go:223] Waiting for caches to sync for namespace
I1113 23:39:32.437475       1 controllermanager.go:533] Started "deployment"
I1113 23:39:32.437960       1 deployment_controller.go:153] Starting deployment controller
I1113 23:39:32.438788       1 shared_informer.go:223] Waiting for caches to sync for deployment
I1113 23:39:32.689355       1 controllermanager.go:533] Started "job"
I1113 23:39:32.689454       1 job_controller.go:144] Starting job controller
I1113 23:39:32.689462       1 shared_informer.go:223] Waiting for caches to sync for job
I1113 23:39:32.937212       1 controllermanager.go:533] Started "attachdetach"
W1113 23:39:32.937246       1 controllermanager.go:525] Skipping "ttl-after-finished"
I1113 23:39:32.937453       1 attach_detach_controller.go:338] Starting attach detach controller
I1113 23:39:32.937479       1 shared_informer.go:223] Waiting for caches to sync for attach detach
I1113 23:39:33.188044       1 controllermanager.go:533] Started "persistentvolume-expander"
I1113 23:39:33.188145       1 expand_controller.go:319] Starting expand controller
I1113 23:39:33.188158       1 shared_informer.go:223] Waiting for caches to sync for expand
I1113 23:39:33.437124       1 controllermanager.go:533] Started "pvc-protection"
I1113 23:39:33.437203       1 pvc_protection_controller.go:101] Starting PVC protection controller
I1113 23:39:33.437213       1 shared_informer.go:223] Waiting for caches to sync for PVC protection
I1113 23:39:33.688521       1 controllermanager.go:533] Started "endpointslice"
I1113 23:39:33.688578       1 endpointslice_controller.go:213] Starting endpoint slice controller
I1113 23:39:33.688586       1 shared_informer.go:223] Waiting for caches to sync for endpoint_slice
I1113 23:39:33.937604       1 controllermanager.go:533] Started "podgc"
I1113 23:39:33.937684       1 gc_controller.go:89] Starting GC controller
I1113 23:39:33.937772       1 shared_informer.go:223] Waiting for caches to sync for GC
I1113 23:39:34.086227       1 controllermanager.go:533] Started "csrcleaner"
I1113 23:39:34.086655       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I1113 23:39:34.086702       1 cleaner.go:82] Starting CSR cleaner controller
I1113 23:39:34.098539       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
W1113 23:39:34.124592       1 actual_state_of_world.go:506] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="ephemeral-cluster-control-plane" does not exist
I1113 23:39:34.128517       1 shared_informer.go:230] Caches are synced for TTL 
I1113 23:39:34.131595       1 shared_informer.go:230] Caches are synced for PV protection 
I1113 23:39:34.138538       1 shared_informer.go:230] Caches are synced for GC 
I1113 23:39:34.139319       1 shared_informer.go:230] Caches are synced for deployment 
I1113 23:39:34.140970       1 shared_informer.go:230] Caches are synced for ReplicationController 
I1113 23:39:34.147086       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"41226a99-c83a-4e4a-8343-db4631630117", APIVersion:"apps/v1", ResourceVersion:"184", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-66bff467f8 to 2
I1113 23:39:34.151618       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"local-path-storage", Name:"local-path-provisioner", UID:"2c052cb1-6eb8-4503-9bb1-273386b9ebe9", APIVersion:"apps/v1", ResourceVersion:"251", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set local-path-provisioner-bd4bb6b75 to 1
I1113 23:39:34.157209       1 shared_informer.go:230] Caches are synced for HPA 
I1113 23:39:34.184731       1 shared_informer.go:230] Caches are synced for node 
I1113 23:39:34.184769       1 range_allocator.go:172] Starting range CIDR allocator
I1113 23:39:34.184774       1 shared_informer.go:223] Waiting for caches to sync for cidrallocator
I1113 23:39:34.184779       1 shared_informer.go:230] Caches are synced for cidrallocator 
I1113 23:39:34.187509       1 shared_informer.go:230] Caches are synced for service account 
I1113 23:39:34.188377       1 shared_informer.go:230] Caches are synced for ClusterRoleAggregator 
I1113 23:39:34.190820       1 shared_informer.go:230] Caches are synced for endpoint_slice 
I1113 23:39:34.193246       1 shared_informer.go:230] Caches are synced for ReplicaSet 
I1113 23:39:34.197968       1 shared_informer.go:230] Caches are synced for daemon sets 
I1113 23:39:34.201516       1 shared_informer.go:230] Caches are synced for taint 
I1113 23:39:34.201641       1 node_lifecycle_controller.go:1433] Initializing eviction metric for zone: 
I1113 23:39:34.201698       1 shared_informer.go:230] Caches are synced for certificate-csrapproving 
W1113 23:39:34.201839       1 node_lifecycle_controller.go:1048] Missing timestamp for Node ephemeral-cluster-control-plane. Assuming now as a timestamp.
I1113 23:39:34.201894       1 node_lifecycle_controller.go:1199] Controller detected that all Nodes are not-Ready. Entering master disruption mode.
I1113 23:39:34.201934       1 event.go:278] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ephemeral-cluster-control-plane", UID:"ea8b5e4d-67ee-4303-8553-fdc978179e3a", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ephemeral-cluster-control-plane event: Registered Node ephemeral-cluster-control-plane in Controller
I1113 23:39:34.201973       1 taint_manager.go:187] Starting NoExecuteTaintManager
I1113 23:39:34.202975       1 shared_informer.go:230] Caches are synced for namespace 
I1113 23:39:34.203202       1 shared_informer.go:230] Caches are synced for certificate-csrsigning 
I1113 23:39:34.251351       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"10c449ee-95a9-4460-a570-689053b5f50d", APIVersion:"apps/v1", ResourceVersion:"358", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-66bff467f8-tfhxq
I1113 23:39:34.251568       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"local-path-storage", Name:"local-path-provisioner-bd4bb6b75", UID:"7fe48e77-0779-4bcd-8b04-0f27e409fc7c", APIVersion:"apps/v1", ResourceVersion:"359", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: local-path-provisioner-bd4bb6b75-l7v6q
I1113 23:39:34.268822       1 range_allocator.go:373] Set node ephemeral-cluster-control-plane PodCIDR to [10.244.0.0/24]
I1113 23:39:34.302284       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"10c449ee-95a9-4460-a570-689053b5f50d", APIVersion:"apps/v1", ResourceVersion:"358", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-66bff467f8-fdl46
I1113 23:39:34.323313       1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"ea913dfe-07e0-4c3c-b65d-03f535ce5331", APIVersion:"apps/v1", ResourceVersion:"189", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-cqq24
I1113 23:39:34.330393       1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kindnet", UID:"8a9812d5-e312-4974-9bb6-717051444032", APIVersion:"apps/v1", ResourceVersion:"219", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kindnet-g7pqw
I1113 23:39:34.338803       1 shared_informer.go:230] Caches are synced for PVC protection 
I1113 23:39:34.345135       1 shared_informer.go:230] Caches are synced for persistent volume 
E1113 23:39:34.376714       1 daemon_controller.go:292] kube-system/kube-proxy failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy", UID:"ea913dfe-07e0-4c3c-b65d-03f535ce5331", ResourceVersion:"189", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63740907557, loc:(*time.Location)(0x3ab5da0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"1"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubeadm", Operation:"Update", APIVersion:"apps/v1", Time:(*v1.Time)(0xc0017e2ec0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0017e2ee0)}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc0017e2f00), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc000420b40), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e2f20), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e2f40), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"k8s.gcr.io/kube-proxy:v1.18.2", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf", "--hostname-override=$(NODE_NAME)"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"NODE_NAME", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc0017e2f80)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc00176b680), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc000dbcf78), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc000126150), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"CriticalAddonsOnly", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"system-node-critical", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0xc00000f4b0)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0xc000dbcfc8)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
I1113 23:39:34.389003       1 shared_informer.go:230] Caches are synced for expand 
E1113 23:39:34.454518       1 daemon_controller.go:292] kube-system/kindnet failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kindnet", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/apps/v1/namespaces/kube-system/daemonsets/kindnet", UID:"8a9812d5-e312-4974-9bb6-717051444032", ResourceVersion:"219", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63740907559, loc:(*time.Location)(0x3ab5da0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"kindnet", "k8s-app":"kindnet", "tier":"node"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"1"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl", Operation:"Update", APIVersion:"apps/v1", Time:(*v1.Time)(0xc0017e2fe0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0017e3000)}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc0017e3020), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"kindnet", "k8s-app":"kindnet", "tier":"node"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"cni-cfg", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e3040), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e3060), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e3080), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kindnet-cni", Image:"kindest/kindnetd:0.5.4", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"HOST_IP", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc0017e30a0)}, v1.EnvVar{Name:"POD_IP", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc0017e30e0)}, v1.EnvVar{Name:"POD_SUBNET", Value:"10.244.0.0/16", ValueFrom:(*v1.EnvVarSource)(nil)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"cni-cfg", ReadOnly:false, MountPath:"/etc/cni/net.d", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc00176b810), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc000dbd1d8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kindnet", DeprecatedServiceAccount:"kindnet", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc0001261c0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0xc00000f4b8)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0xc000dbd270)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "kindnet": the object has been modified; please apply your changes to the latest version and try again
I1113 23:39:34.479203       1 shared_informer.go:230] Caches are synced for disruption 
I1113 23:39:34.479244       1 disruption.go:339] Sending events to api server.
I1113 23:39:34.488925       1 shared_informer.go:230] Caches are synced for bootstrap_signer 
I1113 23:39:34.532041       1 shared_informer.go:230] Caches are synced for stateful set 
I1113 23:39:34.537706       1 shared_informer.go:230] Caches are synced for attach detach 
I1113 23:39:34.589811       1 shared_informer.go:230] Caches are synced for job 
I1113 23:39:34.699165       1 shared_informer.go:230] Caches are synced for garbage collector 
I1113 23:39:34.737761       1 shared_informer.go:230] Caches are synced for resource quota 
I1113 23:39:34.764350       1 shared_informer.go:230] Caches are synced for endpoint 
I1113 23:39:34.775973       1 shared_informer.go:230] Caches are synced for garbage collector 
I1113 23:39:34.776046       1 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I1113 23:39:34.788200       1 shared_informer.go:230] Caches are synced for resource quota 
I1113 23:40:19.211725       1 node_lifecycle_controller.go:1226] Controller detected that some Nodes are Ready. Exiting master disruption mode.
I1113 23:40:35.620004       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for providers.clusterctl.cluster.x-k8s.io
I1113 23:40:35.620054       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I1113 23:40:35.721050       1 shared_informer.go:230] Caches are synced for resource quota 
I1113 23:40:35.811421       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I1113 23:40:35.811708       1 shared_informer.go:230] Caches are synced for garbage collector 
I1113 23:40:42.152920       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager", Name:"cert-manager-cainjector", UID:"329cc880-1c8e-4b25-a8c7-a93f2b665fe6", APIVersion:"apps/v1", ResourceVersion:"735", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set cert-manager-cainjector-5ffff9dd7c to 1
I1113 23:40:42.256533       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"cert-manager", Name:"cert-manager-cainjector-5ffff9dd7c", UID:"2d883cad-7625-43c7-a1bc-27603341cae7", APIVersion:"apps/v1", ResourceVersion:"736", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: cert-manager-cainjector-5ffff9dd7c-tn949
I1113 23:40:42.339748       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager", Name:"cert-manager", UID:"48cef02c-e5fd-403d-949a-1d4cda9c426e", APIVersion:"apps/v1", ResourceVersion:"741", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set cert-manager-578cd6d964 to 1
I1113 23:40:42.388125       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"cert-manager", Name:"cert-manager-578cd6d964", UID:"cafa3d0d-a065-4573-9b87-907621713de6", APIVersion:"apps/v1", ResourceVersion:"743", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: cert-manager-578cd6d964-x4zdx
I1113 23:40:42.534217       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"cert-manager", Name:"cert-manager-webhook", UID:"59e97f16-fbec-4603-809a-4a5ef22f5a8a", APIVersion:"apps/v1", ResourceVersion:"759", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set cert-manager-webhook-556b9d7dfd to 1
I1113 23:40:42.570568       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"cert-manager", Name:"cert-manager-webhook-556b9d7dfd", UID:"db5f8f92-cf3d-417c-8200-a508eb43e252", APIVersion:"apps/v1", ResourceVersion:"762", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: cert-manager-webhook-556b9d7dfd-c9p6j
I1113 23:41:06.035972       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capi-webhook-system", Name:"capi-controller-manager", UID:"87a76e56-f7a4-4f89-b80e-f83a2669a7dd", APIVersion:"apps/v1", ResourceVersion:"942", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capi-controller-manager-5c785c685c to 1
I1113 23:41:06.083979       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capi-webhook-system", Name:"capi-controller-manager-5c785c685c", UID:"23dfa7da-82e8-4684-9f90-4e24047483a8", APIVersion:"apps/v1", ResourceVersion:"943", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capi-controller-manager-5c785c685c-9zw9m
E1113 23:41:06.528854       1 resource_quota_controller.go:408] unable to retrieve the complete list of server APIs: exp.cluster.x-k8s.io/v1alpha3: the server could not find the requested resource
I1113 23:41:06.530146       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for machinedeployments.cluster.x-k8s.io
I1113 23:41:06.530668       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for issuers.cert-manager.io
I1113 23:41:06.531642       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for clusterresourcesets.addons.cluster.x-k8s.io
I1113 23:41:06.531899       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for certificaterequests.cert-manager.io
I1113 23:41:06.531995       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for machinehealthchecks.cluster.x-k8s.io
I1113 23:41:06.533756       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for clusterresourcesetbindings.addons.cluster.x-k8s.io
I1113 23:41:06.533930       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for machines.cluster.x-k8s.io
I1113 23:41:06.533983       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for certificates.cert-manager.io
I1113 23:41:06.534036       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for challenges.acme.cert-manager.io
I1113 23:41:06.534129       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for clusters.cluster.x-k8s.io
I1113 23:41:06.534181       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for orders.acme.cert-manager.io
I1113 23:41:06.534216       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for machinesets.cluster.x-k8s.io
I1113 23:41:06.534468       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I1113 23:41:07.429556       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I1113 23:41:07.529063       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capi-system", Name:"capi-controller-manager", UID:"eb7143c7-cac4-47d5-8ec2-ef14899e4992", APIVersion:"apps/v1", ResourceVersion:"996", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capi-controller-manager-67859f6b78 to 1
I1113 23:41:07.655214       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capi-system", Name:"capi-controller-manager-67859f6b78", UID:"0c87ade4-ca15-43bf-882a-38a6b94fcaa4", APIVersion:"apps/v1", ResourceVersion:"998", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capi-controller-manager-67859f6b78-k2dwf
I1113 23:41:08.834801       1 shared_informer.go:230] Caches are synced for resource quota 
I1113 23:41:08.930138       1 shared_informer.go:230] Caches are synced for garbage collector 
I1113 23:41:09.297168       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capi-webhook-system", Name:"capi-kubeadm-bootstrap-controller-manager", UID:"80886053-5635-4bda-bd9c-7dabe485bef6", APIVersion:"apps/v1", ResourceVersion:"1043", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capi-kubeadm-bootstrap-controller-manager-77658d7745 to 1
I1113 23:41:09.324770       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capi-webhook-system", Name:"capi-kubeadm-bootstrap-controller-manager-77658d7745", UID:"ad2b0f8a-878e-493e-9fe2-6028c4b745dc", APIVersion:"apps/v1", ResourceVersion:"1044", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capi-kubeadm-bootstrap-controller-manager-77658d7745-8bw2g
I1113 23:41:11.401769       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capi-kubeadm-bootstrap-system", Name:"capi-kubeadm-bootstrap-controller-manager", UID:"7410db07-a36a-41af-8e5f-80180c7312e3", APIVersion:"apps/v1", ResourceVersion:"1090", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capi-kubeadm-bootstrap-controller-manager-556678c94 to 1
I1113 23:41:11.484641       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capi-kubeadm-bootstrap-system", Name:"capi-kubeadm-bootstrap-controller-manager-556678c94", UID:"74a18148-110d-4f89-892f-327a726d3c0f", APIVersion:"apps/v1", ResourceVersion:"1091", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capi-kubeadm-bootstrap-controller-manager-556678c94-vndvn
I1113 23:41:13.454657       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capi-webhook-system", Name:"capi-kubeadm-control-plane-controller-manager", UID:"90bdcd03-e38b-4edc-bc12-4ad8f76bc246", APIVersion:"apps/v1", ResourceVersion:"1129", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capi-kubeadm-control-plane-controller-manager-74dcf8b9c to 1
I1113 23:41:13.485610       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capi-webhook-system", Name:"capi-kubeadm-control-plane-controller-manager-74dcf8b9c", UID:"b66b9417-bf10-4262-ab40-94c072744018", APIVersion:"apps/v1", ResourceVersion:"1131", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capi-kubeadm-control-plane-controller-manager-74dcf8b9c-792jg
I1113 23:41:16.122125       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capi-kubeadm-control-plane-system", Name:"capi-kubeadm-control-plane-controller-manager", UID:"b2f5cc5b-b357-4833-8780-a6ffabde2d01", APIVersion:"apps/v1", ResourceVersion:"1194", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capi-kubeadm-control-plane-controller-manager-556d47dffd to 1
I1113 23:41:16.258951       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capi-kubeadm-control-plane-system", Name:"capi-kubeadm-control-plane-controller-manager-556d47dffd", UID:"44569c04-10e5-4943-9806-327304c6a794", APIVersion:"apps/v1", ResourceVersion:"1197", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capi-kubeadm-control-plane-controller-manager-556d47dffd-vtc6q
I1113 23:41:18.797460       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"capd-system", Name:"capd-controller-manager", UID:"02d33980-a701-402f-957d-2b82b5ee277d", APIVersion:"apps/v1", ResourceVersion:"1276", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set capd-controller-manager-5498cfdfb5 to 1
I1113 23:41:18.847273       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"capd-system", Name:"capd-controller-manager-5498cfdfb5", UID:"80cd9499-5b43-4f19-9438-3da9a48237c6", APIVersion:"apps/v1", ResourceVersion:"1277", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: capd-controller-manager-5498cfdfb5-qmt2t
I1113 23:41:22.871429       1 namespace_controller.go:185] Namespace has been deleted cert-manager-test
I1113 23:41:39.886785       1 request.go:621] Throttling request took 1.037145395s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:41:39.890112       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for dockermachines.infrastructure.cluster.x-k8s.io
W1113 23:41:39.890140       1 shared_informer.go:461] resyncPeriod 65957631203381 is smaller than resyncCheckPeriod 70094519979271 and the informer has already started. Changing it to 70094519979271
I1113 23:41:39.890161       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for machinepools.exp.cluster.x-k8s.io
I1113 23:41:39.890176       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for dockermachinepools.exp.infrastructure.cluster.x-k8s.io
I1113 23:41:39.890192       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for dockermachinetemplates.infrastructure.cluster.x-k8s.io
I1113 23:41:39.890216       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for dockerclusters.infrastructure.cluster.x-k8s.io
I1113 23:41:39.890230       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for kubeadmconfigs.bootstrap.cluster.x-k8s.io
I1113 23:41:39.890251       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io
I1113 23:41:39.902532       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for kubeadmcontrolplanes.controlplane.cluster.x-k8s.io
I1113 23:41:39.902718       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I1113 23:41:40.406079       1 shared_informer.go:230] Caches are synced for resource quota 
I1113 23:41:41.038343       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I1113 23:41:41.038611       1 shared_informer.go:230] Caches are synced for garbage collector 
I1113 23:42:11.459684       1 request.go:621] Throttling request took 1.048675529s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s
I1113 23:42:42.517965       1 request.go:621] Throttling request took 1.048060096s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/autoscaling/v1?timeout=32s
I1113 23:43:13.572362       1 request.go:621] Throttling request took 1.044361332s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authentication.k8s.io/v1beta1?timeout=32sI1113 23:43:44.629527       1 request.go:621] Throttling request took 1.044119657s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/acme.cert-manager.io/v1alpha2?timeout=32sI1113 23:44:15.684960       1 request.go:621] Throttling request took 1.037014898s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s
I1113 23:44:46.740137       1 request.go:621] Throttling request took 1.043146186s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:45:17.748077       1 request.go:621] Throttling request took 1.002244299s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1?timeout=32s
I1113 23:45:48.852737       1 request.go:621] Throttling request took 1.04321958s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:46:19.905478       1 request.go:621] Throttling request took 1.046585977s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1beta1?timeout=32s
I1113 23:46:50.959438       1 request.go:621] Throttling request took 1.046870654s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/rbac.authorization.k8s.io/v1?timeout=32s
I1113 23:47:22.011995       1 request.go:621] Throttling request took 1.046812411s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/apiextensions.k8s.io/v1?timeout=32s
I1113 23:47:53.064626       1 request.go:621] Throttling request took 1.0438192s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:48:24.118361       1 request.go:621] Throttling request took 1.048321291s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:48:55.170741       1 request.go:621] Throttling request took 1.047581034s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/coordination.k8s.io/v1beta1?timeout=32s
I1113 23:49:26.226832       1 request.go:621] Throttling request took 1.046915554s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:49:57.280230       1 request.go:621] Throttling request took 1.047817964s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authorization.k8s.io/v1beta1?timeout=32s
I1113 23:50:28.336685       1 request.go:621] Throttling request took 1.046812327s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1beta1?timeout=32s
I1113 23:50:59.388260       1 request.go:621] Throttling request took 1.047332666s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/extensions/v1beta1?timeout=32s
I1113 23:51:30.444926       1 request.go:621] Throttling request took 1.048349624s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:52:01.498084       1 request.go:621] Throttling request took 1.046993194s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1?timeout=32s
I1113 23:52:32.553060       1 request.go:621] Throttling request took 1.048715633s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:53:03.611564       1 request.go:621] Throttling request took 1.052399012s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:53:34.665867       1 request.go:621] Throttling request took 1.047601327s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1?timeout=32s
I1113 23:54:05.719297       1 request.go:621] Throttling request took 1.048435986s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authorization.k8s.io/v1beta1?timeout=32s
I1113 23:54:36.774054       1 request.go:621] Throttling request took 1.049384303s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:55:07.815338       1 request.go:621] Throttling request took 1.02712536s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authentication.k8s.io/v1beta1?timeout=32s
I1113 23:55:38.887737       1 request.go:621] Throttling request took 1.050191201s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/cert-manager.io/v1alpha2?timeout=32s
I1113 23:56:09.939748       1 request.go:621] Throttling request took 1.047001466s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:56:40.991531       1 request.go:621] Throttling request took 1.038877079s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:57:12.043906       1 request.go:621] Throttling request took 1.033597238s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1113 23:57:43.097761       1 request.go:621] Throttling request took 1.048583793s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1113 23:58:14.150154       1 request.go:621] Throttling request took 1.044994505s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:58:45.202950       1 request.go:621] Throttling request took 1.047185263s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
I1113 23:59:16.258259       1 request.go:621] Throttling request took 1.035651873s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1113 23:59:47.311211       1 request.go:621] Throttling request took 1.0446615s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:00:18.367253       1 request.go:621] Throttling request took 1.049595209s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1114 00:00:49.420423       1 request.go:621] Throttling request took 1.048504364s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/scheduling.k8s.io/v1?timeout=32s
I1114 00:01:20.476619       1 request.go:621] Throttling request took 1.049545485s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/coordination.k8s.io/v1?timeout=32s
I1114 00:01:51.535868       1 request.go:621] Throttling request took 1.035793733s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1114 00:02:22.590907       1 request.go:621] Throttling request took 1.047975407s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:02:53.646120       1 request.go:621] Throttling request took 1.044823047s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:03:24.700284       1 request.go:621] Throttling request took 1.047969468s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/controlplane.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:03:55.753527       1 request.go:621] Throttling request took 1.047363515s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:04:26.808501       1 request.go:621] Throttling request took 1.045754815s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1?timeout=32s
I1114 00:04:57.863214       1 request.go:621] Throttling request took 1.046350741s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/cluster.x-k8s.io/v1alpha2?timeout=32s
I1114 00:05:28.877857       1 request.go:621] Throttling request took 1.007590124s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.cluster.x-k8s.io/v1alpha3?timeout=32sI1114 00:05:59.973350       1 request.go:621] Throttling request took 1.046881833s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:06:31.028848       1 request.go:621] Throttling request took 1.049159412s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:07:02.084510       1 request.go:621] Throttling request took 1.048670935s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/clusterctl.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:07:33.137544       1 request.go:621] Throttling request took 1.044195661s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:08:04.147343       1 request.go:621] Throttling request took 1.001485324s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
I1114 00:08:35.244614       1 request.go:621] Throttling request took 1.046856284s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:09:06.297667       1 request.go:621] Throttling request took 1.047528764s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:09:37.377391       1 request.go:621] Throttling request took 1.032759773s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1114 00:10:08.429992       1 request.go:621] Throttling request took 1.047516926s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/autoscaling/v1?timeout=32s
I1114 00:10:39.484235       1 request.go:621] Throttling request took 1.041470144s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:11:10.541100       1 request.go:621] Throttling request took 1.048271643s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/policy/v1beta1?timeout=32s
I1114 00:11:41.594128       1 request.go:621] Throttling request took 1.046793396s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:12:12.646644       1 request.go:621] Throttling request took 1.044696807s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/batch/v1?timeout=32s
I1114 00:12:43.699560       1 request.go:621] Throttling request took 1.04300129s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/addons.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:13:14.754455       1 request.go:621] Throttling request took 1.049308517s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/clusterctl.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:13:45.808989       1 request.go:621] Throttling request took 1.047109094s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:14:16.865393       1 request.go:621] Throttling request took 1.036668077s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:14:47.880670       1 request.go:621] Throttling request took 1.005609901s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authentication.k8s.io/v1?timeout=32s
I1114 00:15:18.978381       1 request.go:621] Throttling request took 1.047453869s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/autoscaling/v2beta2?timeout=32s
I1114 00:15:50.031501       1 request.go:621] Throttling request took 1.04616549s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:16:21.085196       1 request.go:621] Throttling request took 1.047949888s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:16:52.140526       1 request.go:621] Throttling request took 1.04238199s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authorization.k8s.io/v1beta1?timeout=32s
I1114 00:17:23.195738       1 request.go:621] Throttling request took 1.035163861s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/acme.cert-manager.io/v1alpha2?timeout=32sI1114 00:17:54.249723       1 request.go:621] Throttling request took 1.041673413s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:18:25.302738       1 request.go:621] Throttling request took 1.035691965s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:18:56.309342       1 request.go:621] Throttling request took 1.001344711s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:19:27.417198       1 request.go:621] Throttling request took 1.042068554s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:19:58.470367       1 request.go:621] Throttling request took 1.045745372s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
I1114 00:20:29.523583       1 request.go:621] Throttling request took 1.048434729s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:21:00.576203       1 request.go:621] Throttling request took 1.047043446s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:21:31.631473       1 request.go:621] Throttling request took 1.048806787s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:22:02.685133       1 request.go:621] Throttling request took 1.047535005s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/events.k8s.io/v1beta1?timeout=32s
I1114 00:22:33.740924       1 request.go:621] Throttling request took 1.050369252s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.cluster.x-k8s.io/v1alpha3?timeout=32sI1114 00:23:04.793784       1 request.go:621] Throttling request took 1.048246902s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/apiextensions.k8s.io/v1?timeout=32s
I1114 00:23:35.799009       1 request.go:621] Throttling request took 1.000452442s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:24:06.899494       1 request.go:621] Throttling request took 1.048252861s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/scheduling.k8s.io/v1beta1?timeout=32s
I1114 00:24:37.952282       1 request.go:621] Throttling request took 1.04621863s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/autoscaling/v1?timeout=32s
I1114 00:25:09.015190       1 request.go:621] Throttling request took 1.047728259s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/bootstrap.cluster.x-k8s.io/v1alpha2?timeout=32s
I1114 00:25:40.069505       1 request.go:621] Throttling request took 1.045890919s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:26:11.122497       1 request.go:621] Throttling request took 1.044266095s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/bootstrap.cluster.x-k8s.io/v1alpha2?timeout=32s
I1114 00:26:42.182564       1 request.go:621] Throttling request took 1.052107344s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:27:13.235034       1 request.go:621] Throttling request took 1.047077773s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/storage.k8s.io/v1?timeout=32s
I1114 00:27:44.287952       1 request.go:621] Throttling request took 1.046983269s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:28:15.343677       1 request.go:621] Throttling request took 1.047657145s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/authorization.k8s.io/v1beta1?timeout=32s
I1114 00:28:46.400030       1 request.go:621] Throttling request took 1.051401645s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/controlplane.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:29:17.453188       1 request.go:621] Throttling request took 1.048054329s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:29:48.511699       1 request.go:621] Throttling request took 1.048931064s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:30:19.572370       1 request.go:621] Throttling request took 1.033881595s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/coordination.k8s.io/v1?timeout=32s
I1114 00:30:50.629676       1 request.go:621] Throttling request took 1.047586033s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/exp.infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:31:21.684608       1 request.go:621] Throttling request took 1.048103306s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/networking.k8s.io/v1?timeout=32s
I1114 00:31:52.741429       1 request.go:621] Throttling request took 1.048807407s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/cluster.x-k8s.io/v1alpha3?timeout=32s
I1114 00:32:23.793510       1 request.go:621] Throttling request took 1.046194596s, request: GET:https://ephemeral-cluster-control-plane:6443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
rkjain commented 4 years ago 
❯ kubectl describe cluster --kubeconfig ~/.airship/kubeconfig
Name:         target-cluster
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cluster.x-k8s.io/v1alpha3","kind":"Cluster","metadata":{"annotations":{},"name":"target-cluster","namespace":"default"},"sp...
API Version:  cluster.x-k8s.io/v1alpha3
Kind:         Cluster
Metadata:
  Creation Timestamp:  2020-11-14T00:20:46Z
  Finalizers:
    cluster.cluster.x-k8s.io
  Generation:  1
  Managed Fields:
    API Version:  cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:clusterNetwork:
          .:
          f:pods:
            .:
            f:cidrBlocks:
          f:serviceDomain:
          f:services:
            .:
            f:cidrBlocks:
        f:controlPlaneRef:
          .:
          f:apiVersion:
          f:kind:
          f:name:
        f:infrastructureRef:
          .:
          f:apiVersion:
          f:kind:
          f:name:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2020-11-14T00:20:46Z
    API Version:  cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cluster.cluster.x-k8s.io":
      f:status:
        .:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Time:            2020-11-14T00:20:48Z
  Resource Version:  13826
  Self Link:         /apis/cluster.x-k8s.io/v1alpha3/namespaces/default/clusters/target-cluster
  UID:               e4c6d4a4-7d44-40ad-88b0-78d08712834d
Spec:
  Cluster Network:
    Pods:
      Cidr Blocks:
        172.17.0.0/16
    Service Domain:  cluster.local
    Services:
      Cidr Blocks:
        10.0.0.0/24
  Control Plane Endpoint:
    Host:  
    Port:  0
  Control Plane Ref:
    API Version:  controlplane.cluster.x-k8s.io/v1alpha3
    Kind:         KubeadmControlPlane
    Name:         target-cluster-control-plane
    Namespace:    default
  Infrastructure Ref:
    API Version:  infrastructure.cluster.x-k8s.io/v1alpha3
    Kind:         DockerCluster
    Name:         target-cluster
    Namespace:    default
Status:
  Conditions:
    Last Transition Time:  2020-11-14T00:20:48Z
    Reason:                WaitingForControlPlane
    Severity:              Info
    Status:                False
    Type:                  Ready
    Last Transition Time:  2020-11-14T00:20:48Z
    Reason:                WaitingForControlPlane
    Severity:              Info
    Status:                False
    Type:                  ControlPlaneReady
    Last Transition Time:  2020-11-14T00:20:47Z
    Reason:                WaitingForInfrastructure
    Severity:              Info
    Status:                False
    Type:                  InfrastructureReady
  Observed Generation:     1
  Phase:                   Provisioning
Events:                    <none>
❯ 
❯ kubectl describe machinehealthcheck  --kubeconfig ~/.airship/kubeconfig
Name:         target-cluster-mhc-0
Namespace:    default
Labels:       cluster.x-k8s.io/cluster-name=target-cluster
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cluster.x-k8s.io/v1alpha3","kind":"MachineHealthCheck","metadata":{"annotations":{},"name":"target-cluster-mhc-0","namespac...
API Version:  cluster.x-k8s.io/v1alpha3
Kind:         MachineHealthCheck
Metadata:
  Creation Timestamp:  2020-11-14T00:20:46Z
  Generation:          1
  Managed Fields:
    API Version:  cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:clusterName:
        f:maxUnhealthy:
        f:selector:
          .:
          f:matchLabels:
            .:
            f:nodepool:
        f:unhealthyConditions:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2020-11-14T00:20:46Z
    API Version:  cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
    Manager:    manager
    Operation:  Update
    Time:       2020-11-14T00:20:46Z
  Owner References:
    API Version:     cluster.x-k8s.io/v1alpha3
    Kind:            Cluster
    Name:            target-cluster
    UID:             e4c6d4a4-7d44-40ad-88b0-78d08712834d
  Resource Version:  13805
  Self Link:         /apis/cluster.x-k8s.io/v1alpha3/namespaces/default/machinehealthchecks/target-cluster-mhc-0
  UID:               38d5875f-6293-44e3-b9a8-4023bc6300cf
Spec:
  Cluster Name:          target-cluster
  Max Unhealthy:         100%
  Node Startup Timeout:  10m0s
  Selector:
    Match Labels:
      Nodepool:  pool1
  Unhealthy Conditions:
    Status:   True
    Timeout:  30s
    Type:     E2ENodeUnhealthy
Events:
  Type     Reason          Age                  From                           Message
  ----     ------          ----                 ----                           -------
  Warning  ReconcileError  4m9s (x20 over 20m)  machinehealthcheck-controller  error fetching REST client config for remote cluster: failed to retrieve kubeconfig secret for Cluster default/target-cluster: Secret "target-cluster-kubeconfig" not found
rkjain commented 4 years ago 
❯ kubectl describe kubeadmcontrolplane  --kubeconfig ~/.airship/kubeconfig
Name:         target-cluster-control-plane
Namespace:    default
Labels:       cluster.x-k8s.io/cluster-name=target-cluster
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"KubeadmControlPlane","metadata":{"annotations":{},"name":"target-cluster-co...
API Version:  controlplane.cluster.x-k8s.io/v1alpha3
Kind:         KubeadmControlPlane
Metadata:
  Creation Timestamp:  2020-11-14T00:20:46Z
  Generation:          1
  Managed Fields:
    API Version:  controlplane.cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:infrastructureTemplate:
          .:
          f:apiVersion:
          f:kind:
          f:name:
        f:kubeadmConfigSpec:
          .:
          f:clusterConfiguration:
            .:
            f:apiServer:
              .:
              f:certSANs:
            f:controllerManager:
              .:
              f:extraArgs:
                .:
                f:enable-hostpath-provisioner:
          f:files:
          f:initConfiguration:
            .:
            f:nodeRegistration:
              .:
              f:criSocket:
              f:kubeletExtraArgs:
                .:
                f:eviction-hard:
          f:joinConfiguration:
            .:
            f:nodeRegistration:
              .:
              f:criSocket:
              f:kubeletExtraArgs:
                .:
                f:eviction-hard:
          f:postKubeadmCommands:
        f:replicas:
        f:version:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2020-11-14T00:20:46Z
    API Version:  controlplane.cluster.x-k8s.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:cluster.x-k8s.io/cluster-name:
        f:ownerReferences:
          .:
          k:{"uid":"e4c6d4a4-7d44-40ad-88b0-78d08712834d"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
    Manager:    manager
    Operation:  Update
    Time:       2020-11-14T00:20:48Z
  Owner References:
    API Version:           cluster.x-k8s.io/v1alpha3
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Cluster
    Name:                  target-cluster
    UID:                   e4c6d4a4-7d44-40ad-88b0-78d08712834d
  Resource Version:        13825
  Self Link:               /apis/controlplane.cluster.x-k8s.io/v1alpha3/namespaces/default/kubeadmcontrolplanes/target-cluster-control-plane
  UID:                     2200c32b-b56e-4e92-b875-ce9264e18e8f
Spec:
  Infrastructure Template:
    API Version:  infrastructure.cluster.x-k8s.io/v1alpha3
    Kind:         DockerMachineTemplate
    Name:         target-cluster-control-plane
    Namespace:    default
  Kubeadm Config Spec:
    Cluster Configuration:
      API Server:
        Cert SA Ns:
          localhost
          127.0.0.1
      Controller Manager:
        Extra Args:
          Enable - Hostpath - Provisioner:  true
      Dns:
      Etcd:
      Networking:
      Scheduler:
    Files:
      Content:  #!/bin/sh -x
su - root -c "sleep 10; kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f https://docs.projectcalico.org/v3.12/manifests/calico.yaml"

      Owner:        root:root
      Path:         /calico.sh
      Permissions:  0755
    Init Configuration:
      Local API Endpoint:
        Advertise Address:  
        Bind Port:          0
      Node Registration:
        Cri Socket:  /var/run/containerd/containerd.sock
        Kubelet Extra Args:
          Eviction - Hard:  nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
    Join Configuration:
      Discovery:
      Node Registration:
        Cri Socket:  /var/run/containerd/containerd.sock
        Kubelet Extra Args:
          Eviction - Hard:  nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
    Post Kubeadm Commands:
      sh /calico.sh
  Replicas:  1
  Version:   v1.18.6
Events:      <none>
rkjain commented 4 years ago 
❯ kubectl describe pod cert-manager-578cd6d964-x4zdx -n cert-manager  --kubeconfig ~/.airship/kubeconfig
Name:         cert-manager-578cd6d964-x4zdx
Namespace:    cert-manager
Priority:     0
Node:         ephemeral-cluster-control-plane/172.17.0.2
Start Time:   Fri, 13 Nov 2020 15:40:42 -0800
Labels:       app=cert-manager
              app.kubernetes.io/component=controller
              app.kubernetes.io/instance=cert-manager
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=cert-manager
              helm.sh/chart=cert-manager-v0.16.1
              pod-template-hash=578cd6d964
Annotations:  prometheus.io/path: /metrics
              prometheus.io/port: 9402
              prometheus.io/scrape: true
Status:       Running
IP:           10.244.0.6
IPs:
  IP:           10.244.0.6
Controlled By:  ReplicaSet/cert-manager-578cd6d964
Containers:
  cert-manager:
    Container ID:  containerd://aba800eb351e93414e591d5aa1fa5edae649dfe8c62a06c16c47e8c25892db11
    Image:         quay.io/jetstack/cert-manager-controller:v0.16.1
    Image ID:      quay.io/jetstack/cert-manager-controller@sha256:ec46154598a4841cc4ca3c8915cf75079743f73fd1686be0c7b154f580fb9ca1
    Port:          9402/TCP
    Host Port:     0/TCP
    Args:
      --v=2
      --cluster-resource-namespace=$(POD_NAMESPACE)
      --leader-election-namespace=kube-system
    State:          Running
      Started:      Fri, 13 Nov 2020 15:41:02 -0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-nb5bk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  cert-manager-token-nb5bk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-token-nb5bk
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
teoyaomiqui commented 4 years ago

Issue appears to be gone with updates to versions catalogue @rkjain please confirm, and we can close the issue

rkjain commented 4 years ago

Looks like this was a temporary issue. Today, I did not face the same problem with the latest code.

rkjain commented 3 years ago

The same problem happened again today.

2020-11-18 00:05:48.588816 | TASK [airshipctl-run-script : Run script CONTROLPLANE_COUNT=1 TEST_SITE=docker-test-site ./tools/deployment/provider_common/30_deploy_controlplane.sh]
2020-11-18 00:05:49.111971 | primary | + CONTROLPLANE_COUNT=1 TEST_SITE=docker-test-site ./tools/deployment/provider_common/30_deploy_controlplane.sh
2020-11-18 00:05:49.117574 | primary | create control plane
2020-11-18 00:05:57.610889 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:129: Getting kubeconfig context name from cluster map
2020-11-18 00:05:57.611029 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:134: Getting kubeconfig file information from kubeconfig provider
2020-11-18 00:05:57.611087 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:139: Filtering out documents that shouldn't be applied to kubernetes from document bundle
2020-11-18 00:05:57.611257 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:147: Using kubeconfig at '/home/zuul/.airship/kubeconfig' and context 'ephemeral-cluster'
2020-11-18 00:05:57.611328 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:118: WaitTimeout: 33m20s
2020-11-18 00:05:57.611385 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:76: Getting infos for bundle, inventory id is controlplane-ephemeral
2020-11-18 00:05:57.611473 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:106: Inventory Object config Map not found, auto generating Inventory object
2020-11-18 00:05:57.612012 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:113: Injecting Inventory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"controlplane-ephemeral"},"name":"airshipit-controlplane-ephemeral","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
2020-11-18 00:05:57.612307 | primary | [airshipctl] 2020/11/18 00:05:57 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:119: Making sure that inventory object namespace airshipit exists
2020-11-18 00:05:57.856196 | primary | cluster.cluster.x-k8s.io/target-cluster created
2020-11-18 00:05:57.866810 | primary | dockercluster.infrastructure.cluster.x-k8s.io/target-cluster created
2020-11-18 00:05:57.891998 | primary | dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane created
2020-11-18 00:05:57.917536 | primary | kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane created
2020-11-18 00:05:57.934566 | primary | machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 created
2020-11-18 00:05:57.934645 | primary | 5 resource(s) applied. 5 created, 0 unchanged, 0 configured
2020-11-18 00:05:57.934672 | primary | cluster.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
2020-11-18 00:05:57.934694 | primary | dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
2020-11-18 00:05:57.934715 | primary | dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
2020-11-18 00:05:57.934736 | primary | kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
2020-11-18 00:05:57.934757 | primary | machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is NotFound: Resource not found
2020-11-18 00:05:59.836134 | primary | cluster.cluster.x-k8s.io/target-cluster is InProgress:
2020-11-18 00:05:59.836263 | primary | dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is Current: Resource is current
2020-11-18 00:05:59.836294 | primary | dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
2020-11-18 00:05:59.836325 | primary | kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
2020-11-18 00:05:59.836347 | primary | machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is Current: Resource is current
2020-11-18 00:06:01.836097 | primary | cluster.cluster.x-k8s.io/target-cluster is InProgress:
2020-11-18 00:06:01.836237 | primary | dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is Current: Resource is Ready
2020-11-18 00:06:35.834963 | primary | cluster.cluster.x-k8s.io/target-cluster is InProgress: Scaling up to 1 replicas (actual 0)
2020-11-18 00:06:35.835065 | primary | kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is InProgress: Scaling up to 1 replicas (actual 0)
2020-11-18 00:08:35.834830 | primary | cluster.cluster.x-k8s.io/target-cluster is InProgress:
2020-11-18 00:08:35.834939 | primary | kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is InProgress:
2020-11-18 00:39:17.934835 | primary | [airshipctl] 2020/11/18 00:39:17 opendev.org/airship/airshipctl@/pkg/events/processor.go:77: Received error when applying errors to kubernetes timeout after 2000 seconds waiting for 5 resources to reach condition AllCurrent
2020-11-18 00:08:35.834939 | primary | kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is InProgress:
2020-11-18 00:39:17.934835 | primary | [airshipctl] 2020/11/18 00:39:17 opendev.org/airship/airshipctl@/pkg/events/processor.go:77: Received error when applying errors to kubernetes timeout after 2000 seconds waiting for 5 resources to reach condition AllCurrent

We tested this with multiple providers. And observed that if we take out the below code from clusterctl.yaml, everything works fine. 

images:
  cert-manager:
    repository: "quay.io/jetstack
jezogwza commented 3 years ago

This was fixed.

rkjain commented 3 years ago

The issue still exists. (https://review.opendev.org/#/c/763232/) The script CONTROLPLANE_COUNT=1 TEST_SITE=docker-test-site ./tools/deployment/provider_common/30_deploy_controlplane.sh

failed on zuul.

kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is InProgress: 
[airshipctl] 2020/11/18 20:14:24 opendev.org/airship/airshipctl@/pkg/events/processor.go:77: Received error when applying errors to kubernetes timeout after 2000 seconds waiting for 5 resources to reach condition AllCurrent
Error events received on channel, errors are:
[timeout after 2000 seconds waiting for 5 resources to reach condition AllCurrent]
rkjain commented 3 years ago

To test the scripts locally one can do the following:

# prereq
rm -rf ~/.airship
mkdir /tmp/airship; 
cd /tmp/airship; 
git clone https://review.opendev.org/airship/airshipctl;
git review -d 763232

# Edit as per provider
unset AIRSHIP_SRC KUBECONFIG  WORKERS_COUNT  CONTROLPLANE_COUNT  TEST_SITE  TARGET_CLUSTER_NAME KIND_EXPERIMENTAL_DOCKER_NETWORK
export AIRSHIP_SRC="/tmp/airship"
export KUBECONFIG="$HOME/.airship/kubeconfig"
export WORKERS_COUNT=${WORKERS_COUNT:-"2"}
export CONTROLPLANE_COUNT=${CONTROLPLANE_COUNT:-"1"}
export TEST_SITE=${TEST_SITE:-"docker-test-site"}
export TARGET_CLUSTER_NAME=${TARGET_CLUSTER_NAME:-"target-cluster"}
export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge
export PROVIDER=default

# Clean up
kind delete clusters --all
# Common Scripts
CLUSTER=ephemeral-cluster KIND_CONFIG=./tools/deployment/templates/kind-cluster-with-extramounts ./tools/document/start_kind.sh
./tools/deployment/provider_common/03-init-airship-config.sh
PROVIDER=default TEST_SITE=docker-test-site PROVIDER_MANIFEST=docker_manifest ./tools/deployment/26_deploy_capi_ephemeral_node.sh
CONTROLPLANE_COUNT=1 TEST_SITE=docker-test-site ./tools/deployment/provider_common/30_deploy_controlplane.sh
KUBECONFIG=/tmp/target-cluster.kubeconfig ./tools/deployment/provider_common/32_cluster_init_target_node.sh
./tools/deployment/provider_common/33_cluster_move_target_node.sh
WORKERS_COUNT=2 KUBECONFIG=/tmp/target-cluster.kubeconfig TEST_SITE=docker-test-site ./tools/deployment/provider_common/34_deploy_worker_node.sh
rkjain commented 3 years ago

Here is a log of the local run of the above scripts. And they work fine. However, the problem exists on Zuul, when the phase run control plane get stcuk

❯ kind delete clusters --all
Deleted clusters: []
❯ CLUSTER=ephemeral-cluster KIND_CONFIG=./tools/deployment/templates/kind-cluster-with-extramounts ./tools/document/start_kind.sh
cluster name: ephemeral-cluster
config file: ./tools/deployment/templates/kind-cluster-with-extramounts
Creating cluster "ephemeral-cluster" ...
WARNING: Overriding docker network due to KIND_EXPERIMENTAL_DOCKER_NETWORK
WARNING: Here be dragons! This is not supported currently.
 ✓ Ensuring node image (kindest/node:v1.18.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Waiting ≤ 2m0s for control-plane = Ready ⏳ 
 • Ready after 25s 💚
Set kubectl context to "kind-ephemeral-cluster"
You can now use your cluster with:

kubectl cluster-info --context kind-ephemeral-cluster --kubeconfig /home/rishabh/.airship/kubeconfig

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
node/ephemeral-cluster-control-plane condition met
Context "ephemeral-cluster" modified.
This cluster can be deleted via:
/usr/local/bin/kind delete cluster --name ephemeral-cluster
❯ ./tools/deployment/provider_common/03-init-airship-config.sh
+ export KUBECONFIG=/home/rishabh/.airship/kubeconfig
+ KUBECONFIG=/home/rishabh/.airship/kubeconfig
+ export AIRSHIP_SRC=/tmp/airship
+ AIRSHIP_SRC=/tmp/airship
+ export TEST_SITE=docker-test-site
+ TEST_SITE=docker-test-site
+ export PROVIDER_MANIFEST=docker_manifest
+ PROVIDER_MANIFEST=docker_manifest
+ rm -rf /home/rishabh/.airship/config
+ airshipctl config init
+ airshipctl config set-manifest docker_manifest --repo primary --phase --url https://opendev.org/airship/airshipctl --branch master --metadata-path manifests/site/docker-test-site/metadata.yaml --target-path /tmp/airship
Manifest "docker_manifest" created.
+ airshipctl config set-context ephemeral-cluster --manifest docker_manifest
Context "ephemeral-cluster" created.
+ airshipctl config get-context
contextKubeconf: ""
managementConfiguration: default
manifest: default

contextKubeconf: ephemeral-cluster
managementConfiguration: ""
manifest: docker_manifest

+ airshipctl config use-context ephemeral-cluster
Switched to context "ephemeral-cluster".
+ airshipctl phase plan
GROUP    PHASE
group1   
         clusterctl-init-ephemeral
         controlplane-ephemeral
         clusterctl-init-target
         clusterctl-move
         workers-target
❯ PROVIDER=default TEST_SITE=docker-test-site PROVIDER_MANIFEST=docker_manifest ./tools/deployment/26_deploy_capi_ephemeral_node.shPROVIDER=default TEST_SITE=docker-test-site PROVIDER_MANIFEST=docker_manifest ./tools/deployment/26_deploy_capi_ephemeral_node.sh
❯ PROVIDER=default TEST_SITE=docker-test-site PROVIDER_MANIFEST=docker_manifest ./tools/deployment/26_deploy_capi_ephemeral_node.sh
+ export PROVIDER=default
+ PROVIDER=default
+ export KUBECONFIG=/home/rishabh/.airship/kubeconfig
+ KUBECONFIG=/home/rishabh/.airship/kubeconfig
+ export KUBECONFIG_EPHEMERAL_CONTEXT=ephemeral-cluster
+ KUBECONFIG_EPHEMERAL_CONTEXT=ephemeral-cluster
+ '[' default = metal3 ']'
+ kubectl --kubeconfig /home/rishabh/.airship/kubeconfig --context ephemeral-cluster get pods -A
NAMESPACE            NAME                                                      READY   STATUS    RESTARTS   AGE
kube-system          coredns-66bff467f8-h7m2v                                  1/1     Running   0          34s
kube-system          coredns-66bff467f8-v8bg6                                  1/1     Running   0          34s
kube-system          etcd-ephemeral-cluster-control-plane                      1/1     Running   0          45s
kube-system          kindnet-wvbql                                             1/1     Running   0          34s
kube-system          kube-apiserver-ephemeral-cluster-control-plane            1/1     Running   0          45s
kube-system          kube-controller-manager-ephemeral-cluster-control-plane   1/1     Running   0          45s
kube-system          kube-proxy-qtm5g                                          1/1     Running   0          34s
kube-system          kube-scheduler-ephemeral-cluster-control-plane            1/1     Running   0          44s
local-path-storage   local-path-provisioner-bd4bb6b75-9mgft                    1/1     Running   0          34s
+ echo 'Deploy cluster-api components to ephemeral node'
Deploy cluster-api components to ephemeral node
+ '[' default = metal3 ']'
+ airshipctl phase run clusterctl-init-ephemeral --debug --kubeconfig /home/rishabh/.airship/kubeconfig
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPO_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CABPK_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPD_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPD_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPI_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPM3_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPO_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CABPK_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CACPK_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CACPK_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPI_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPM3_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/client/client.go:81: Starting cluster-api initiation
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/events/processor.go:61: Received event: {4 2020-11-20 12:06:25.48049386 -0800 PST m=+5.824056092 {InitType {[]} {<nil>} {ApplyEventResourceUpdate ServersideApplied <nil>} {ResourceUpdateEvent <nil> <nil>} {PruneEventResourceUpdate Pruned <nil>} {DeleteEventResourceUpdate Deleted <nil>}} {<nil>} {ResourceUpdateEvent <nil> <nil>} {0 starting clusterctl init executor} {0 } {0 }}
Installing the clusterctl inventory CRD
Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
Fetching providers
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider cluster-api of type CoreProvider
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: CoreProvider, name: cluster-api
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: CoreProvider, name: cluster-api
Fetching File="components.yaml" Provider="cluster-api" Version="v0.3.7"
[airshipctl] 2020/11/20 12:06:25 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capi/v0.3.7
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type BootstrapProvider
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: BootstrapProvider, name: kubeadm
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: BootstrapProvider, name: kubeadm
Fetching File="components.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cabpk/v0.3.7
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type ControlPlaneProvider
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: ControlPlaneProvider, name: kubeadm
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: ControlPlaneProvider, name: kubeadm
Fetching File="components.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:06:27 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cacpk/v0.3.7
[airshipctl] 2020/11/20 12:06:28 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider docker of type InfrastructureProvider
[airshipctl] 2020/11/20 12:06:28 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: InfrastructureProvider, name: docker
[airshipctl] 2020/11/20 12:06:28 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: InfrastructureProvider, name: docker
Fetching File="components.yaml" Provider="infrastructure-docker" Version="v0.3.11"
[airshipctl] 2020/11/20 12:06:28 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capd/v0.3.11
[airshipctl] 2020/11/20 12:06:28 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider cluster-api of type CoreProvider
Fetching File="metadata.yaml" Provider="cluster-api" Version="v0.3.7"
[airshipctl] 2020/11/20 12:06:28 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capi/v0.3.7
[airshipctl] 2020/11/20 12:06:29 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type BootstrapProvider
Fetching File="metadata.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:06:29 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cabpk/v0.3.7
[airshipctl] 2020/11/20 12:06:29 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type ControlPlaneProvider
Fetching File="metadata.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:06:29 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cacpk/v0.3.7
[airshipctl] 2020/11/20 12:06:30 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider docker of type InfrastructureProvider
Fetching File="metadata.yaml" Provider="infrastructure-docker" Version="v0.3.11"
[airshipctl] 2020/11/20 12:06:30 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capd/v0.3.11
Creating Namespace="cert-manager-test"
Installing cert-manager Version="v0.16.1"
Creating Namespace="cert-manager"
Creating CustomResourceDefinition="certificaterequests.cert-manager.io"
Creating CustomResourceDefinition="certificates.cert-manager.io"
Creating CustomResourceDefinition="challenges.acme.cert-manager.io"
Creating CustomResourceDefinition="clusterissuers.cert-manager.io"
Creating CustomResourceDefinition="issuers.cert-manager.io"
Creating CustomResourceDefinition="orders.acme.cert-manager.io"
Creating ServiceAccount="cert-manager-cainjector" Namespace="cert-manager"
Creating ServiceAccount="cert-manager" Namespace="cert-manager"
Creating ServiceAccount="cert-manager-webhook" Namespace="cert-manager"
Creating ClusterRole="cert-manager-cainjector"
Creating ClusterRole="cert-manager-controller-issuers"
Creating ClusterRole="cert-manager-controller-clusterissuers"
Creating ClusterRole="cert-manager-controller-certificates"
Creating ClusterRole="cert-manager-controller-orders"
Creating ClusterRole="cert-manager-controller-challenges"
Creating ClusterRole="cert-manager-controller-ingress-shim"
Creating ClusterRole="cert-manager-view"
Creating ClusterRole="cert-manager-edit"
Creating ClusterRoleBinding="cert-manager-cainjector"
Creating ClusterRoleBinding="cert-manager-controller-issuers"
Creating ClusterRoleBinding="cert-manager-controller-clusterissuers"
Creating ClusterRoleBinding="cert-manager-controller-certificates"
Creating ClusterRoleBinding="cert-manager-controller-orders"
Creating ClusterRoleBinding="cert-manager-controller-challenges"
Creating ClusterRoleBinding="cert-manager-controller-ingress-shim"
Creating Role="cert-manager-cainjector:leaderelection" Namespace="kube-system"
Creating Role="cert-manager:leaderelection" Namespace="kube-system"
Creating Role="cert-manager-webhook:dynamic-serving" Namespace="cert-manager"
Creating RoleBinding="cert-manager-cainjector:leaderelection" Namespace="kube-system"
Creating RoleBinding="cert-manager:leaderelection" Namespace="kube-system"
Creating RoleBinding="cert-manager-webhook:dynamic-serving" Namespace="cert-manager"
Creating Service="cert-manager" Namespace="cert-manager"
Creating Service="cert-manager-webhook" Namespace="cert-manager"
Creating Deployment="cert-manager-cainjector" Namespace="cert-manager"
Creating Deployment="cert-manager" Namespace="cert-manager"
Creating Deployment="cert-manager-webhook" Namespace="cert-manager"
Creating MutatingWebhookConfiguration="cert-manager-webhook"
Creating ValidatingWebhookConfiguration="cert-manager-webhook"
Waiting for cert-manager to be available...
Updating Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Certificate="selfsigned-cert" Namespace="cert-manager-test"
Deleting Namespace="cert-manager-test"
Deleting Issuer="test-selfsigned" Namespace="cert-manager-test"
Deleting Certificate="selfsigned-cert" Namespace="cert-manager-test"
Installing Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Creating shared objects Provider="cluster-api" Version="v0.3.7"
Creating Namespace="capi-webhook-system"
Creating CustomResourceDefinition="clusterresourcesetbindings.addons.cluster.x-k8s.io"
Creating CustomResourceDefinition="clusterresourcesets.addons.cluster.x-k8s.io"
Creating CustomResourceDefinition="clusters.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinedeployments.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinehealthchecks.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinepools.exp.cluster.x-k8s.io"
Creating CustomResourceDefinition="machines.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinesets.cluster.x-k8s.io"
Creating Service="capi-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Creating Issuer="capi-selfsigned-issuer" Namespace="capi-webhook-system"
Creating MutatingWebhookConfiguration="capi-mutating-webhook-configuration"
Creating ValidatingWebhookConfiguration="capi-validating-webhook-configuration"
Creating instance objects Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Creating Namespace="capi-system"
Creating Role="capi-leader-election-role" Namespace="capi-system"
Creating ClusterRole="capi-system-capi-aggregated-manager-role"
Creating ClusterRole="capi-system-capi-manager-role"
Creating ClusterRole="capi-system-capi-proxy-role"
Creating RoleBinding="capi-leader-election-rolebinding" Namespace="capi-system"
Creating ClusterRoleBinding="capi-system-capi-manager-rolebinding"
Creating ClusterRoleBinding="capi-system-capi-proxy-rolebinding"
Creating Service="capi-controller-manager-metrics-service" Namespace="capi-system"
Creating Deployment="capi-controller-manager" Namespace="capi-system"
Creating inventory entry Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-bootstrap-system"
Creating shared objects Provider="bootstrap-kubeadm" Version="v0.3.7"
Creating CustomResourceDefinition="kubeadmconfigs.bootstrap.cluster.x-k8s.io"
Creating CustomResourceDefinition="kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io"
Creating Service="capi-kubeadm-bootstrap-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-kubeadm-bootstrap-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-kubeadm-bootstrap-serving-cert" Namespace="capi-webhook-system"
Creating Issuer="capi-kubeadm-bootstrap-selfsigned-issuer" Namespace="capi-webhook-system"
Creating ValidatingWebhookConfiguration="capi-kubeadm-bootstrap-validating-webhook-configuration"
Creating instance objects Provider="bootstrap-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-bootstrap-system"
Creating Namespace="capi-kubeadm-bootstrap-system"
Creating Role="capi-kubeadm-bootstrap-leader-election-role" Namespace="capi-kubeadm-bootstrap-system"
Creating ClusterRole="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-manager-role"
Creating ClusterRole="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-proxy-role"
Creating RoleBinding="capi-kubeadm-bootstrap-leader-election-rolebinding" Namespace="capi-kubeadm-bootstrap-system"
Creating ClusterRoleBinding="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-manager-rolebinding"
Creating ClusterRoleBinding="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-proxy-rolebinding"
Creating Service="capi-kubeadm-bootstrap-controller-manager-metrics-service" Namespace="capi-kubeadm-bootstrap-system"
Creating Deployment="capi-kubeadm-bootstrap-controller-manager" Namespace="capi-kubeadm-bootstrap-system"
Creating inventory entry Provider="bootstrap-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-control-plane-system"
Creating shared objects Provider="control-plane-kubeadm" Version="v0.3.7"
Creating CustomResourceDefinition="kubeadmcontrolplanes.controlplane.cluster.x-k8s.io"
Creating Service="capi-kubeadm-control-plane-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-kubeadm-control-plane-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-kubeadm-control-plane-serving-cert" Namespace="capi-webhook-system"
Creating Issuer="capi-kubeadm-control-plane-selfsigned-issuer" Namespace="capi-webhook-system"
Creating MutatingWebhookConfiguration="capi-kubeadm-control-plane-mutating-webhook-configuration"
Creating ValidatingWebhookConfiguration="capi-kubeadm-control-plane-validating-webhook-configuration"
Creating instance objects Provider="control-plane-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-control-plane-system"
Creating Namespace="capi-kubeadm-control-plane-system"
Creating Role="capi-kubeadm-control-plane-leader-election-role" Namespace="capi-kubeadm-control-plane-system"
Creating Role="capi-kubeadm-control-plane-manager-role" Namespace="capi-kubeadm-control-plane-system"
Creating ClusterRole="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-aggregated-manager-role"
Creating ClusterRole="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-manager-role"
Creating ClusterRole="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-proxy-role"
Creating RoleBinding="capi-kubeadm-control-plane-leader-election-rolebinding" Namespace="capi-kubeadm-control-plane-system"
Creating ClusterRoleBinding="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-manager-rolebinding"
Creating ClusterRoleBinding="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-proxy-rolebinding"
Creating Service="capi-kubeadm-control-plane-controller-manager-metrics-service" Namespace="capi-kubeadm-control-plane-system"
Creating Deployment="capi-kubeadm-control-plane-controller-manager" Namespace="capi-kubeadm-control-plane-system"
Creating inventory entry Provider="control-plane-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v0.3.11" TargetNamespace="capd-system"
Creating shared objects Provider="infrastructure-docker" Version="v0.3.11"
Creating CustomResourceDefinition="dockerclusters.infrastructure.cluster.x-k8s.io"
Creating CustomResourceDefinition="dockermachinepools.exp.infrastructure.cluster.x-k8s.io"
Creating CustomResourceDefinition="dockermachines.infrastructure.cluster.x-k8s.io"
Creating CustomResourceDefinition="dockermachinetemplates.infrastructure.cluster.x-k8s.io"
Creating ValidatingWebhookConfiguration="capd-validating-webhook-configuration"
Creating instance objects Provider="infrastructure-docker" Version="v0.3.11" TargetNamespace="capd-system"
Creating Namespace="capd-system"
Creating Role="capd-leader-election-role" Namespace="capd-system"
Creating ClusterRole="capd-system-capd-manager-role"
Creating ClusterRole="capd-system-capd-proxy-role"
Creating RoleBinding="capd-leader-election-rolebinding" Namespace="capd-system"
Creating ClusterRoleBinding="capd-system-capd-manager-rolebinding"
Creating ClusterRoleBinding="capd-system-capd-proxy-rolebinding"
Creating Service="capd-controller-manager-metrics-service" Namespace="capd-system"
Creating Service="capd-webhook-service" Namespace="capd-system"
Creating Deployment="capd-controller-manager" Namespace="capd-system"
Creating Certificate="capd-serving-cert" Namespace="capd-system"
Creating Issuer="capd-selfsigned-issuer" Namespace="capd-system"
Creating inventory entry Provider="infrastructure-docker" Version="v0.3.11" TargetNamespace="capd-system"
[airshipctl] 2020/11/20 12:07:12 opendev.org/airship/airshipctl@/pkg/events/processor.go:61: Received event: {4 2020-11-20 12:07:12.713160997 -0800 PST m=+53.056723234 {InitType {[]} {<nil>} {ApplyEventResourceUpdate ServersideApplied <nil>} {ResourceUpdateEvent <nil> <nil>} {PruneEventResourceUpdate Pruned <nil>} {DeleteEventResourceUpdate Deleted <nil>}} {<nil>} {ResourceUpdateEvent <nil> <nil>} {1 clusterctl init completed successfully} {0 } {0 }}
+ echo 'Waiting for clusterapi pods to come up'
Waiting for clusterapi pods to come up
+ kubectl --kubeconfig /home/rishabh/.airship/kubeconfig --context ephemeral-cluster wait --for=condition=available deploy --all --timeout=1000s -A
deployment.apps/capd-controller-manager condition met
deployment.apps/capi-kubeadm-bootstrap-controller-manager condition met
deployment.apps/capi-kubeadm-control-plane-controller-manager condition met
deployment.apps/capi-controller-manager condition met
deployment.apps/capi-controller-manager condition met
deployment.apps/capi-kubeadm-bootstrap-controller-manager condition met
deployment.apps/capi-kubeadm-control-plane-controller-manager condition met
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
deployment.apps/coredns condition met
deployment.apps/local-path-provisioner condition met
+ kubectl --kubeconfig /home/rishabh/.airship/kubeconfig --context ephemeral-cluster get pods --all-namespaces
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-5498cfdfb5-ltskj                         2/2     Running   0          49s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-556678c94-xpr2c        2/2     Running   0          57s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-556d47dffd-25f2q   2/2     Running   0          53s
capi-system                         capi-controller-manager-67859f6b78-4fw74                         2/2     Running   0          60s
capi-webhook-system                 capi-controller-manager-5c785c685c-z7nqt                         2/2     Running   0          62s
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-77658d7745-t4tvv       2/2     Running   0          59s
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-74dcf8b9c-sxngl    2/2     Running   0          56s
cert-manager                        cert-manager-578cd6d964-282cm                                    1/1     Running   0          87s
cert-manager                        cert-manager-cainjector-5ffff9dd7c-cpjq7                         1/1     Running   0          87s
cert-manager                        cert-manager-webhook-556b9d7dfd-cj6bm                            1/1     Running   0          87s
kube-system                         coredns-66bff467f8-h7m2v                                         1/1     Running   0          2m16s
kube-system                         coredns-66bff467f8-v8bg6                                         1/1     Running   0          2m16s
kube-system                         etcd-ephemeral-cluster-control-plane                             1/1     Running   0          2m27s
kube-system                         kindnet-wvbql                                                    1/1     Running   0          2m16s
kube-system                         kube-apiserver-ephemeral-cluster-control-plane                   1/1     Running   0          2m27s
kube-system                         kube-controller-manager-ephemeral-cluster-control-plane          1/1     Running   0          2m27s
kube-system                         kube-proxy-qtm5g                                                 1/1     Running   0          2m16s
kube-system                         kube-scheduler-ephemeral-cluster-control-plane                   1/1     Running   0          2m26s
local-path-storage                  local-path-provisioner-bd4bb6b75-9mgft                           1/1     Running   0          2m16s
❯ CONTROLPLANE_COUNT=1 TEST_SITE=docker-test-site ./tools/deployment/provider_common/30_deploy_controlplane.sh
create control plane
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:129: Getting kubeconfig context name from cluster map
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:134: Getting kubeconfig file information from kubeconfig provider
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:139: Filtering out documents that shouldn't be applied to kubernetes from document bundle
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:147: Using kubeconfig at '/home/rishabh/.airship/kubeconfig' and context 'ephemeral-cluster'
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:118: WaitTimeout: 33m20s
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:76: Getting infos for bundle, inventory id is controlplane-ephemeral
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:106: Inventory Object config Map not found, auto generating Inventory object
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:113: Injecting Inventory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"controlplane-ephemeral"},"name":"airshipit-controlplane-ephemeral","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
[airshipctl] 2020/11/20 12:08:22 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:119: Making sure that inventory object namespace airshipit exists
cluster.cluster.x-k8s.io/target-cluster created
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane created
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 created
5 resource(s) applied. 5 created, 0 unchanged, 0 configured
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is NotFound: Resource not found
cluster.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is NotFound: Resource not found
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is NotFound: Resource not found
cluster.cluster.x-k8s.io/target-cluster is InProgress: 
dockercluster.infrastructure.cluster.x-k8s.io/target-cluster is Current: Resource is Ready
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is current
machinehealthcheck.cluster.x-k8s.io/target-cluster-mhc-0 is Current: Resource is current
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is InProgress: 0 of 1 completed
cluster.cluster.x-k8s.io/target-cluster is InProgress: 0 of 1 completed
cluster.cluster.x-k8s.io/target-cluster is Current: Resource is Ready
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/target-cluster-control-plane is Current: Resource is Ready
all resources has reached the Current status
Generate kubeconfig
Generate kubeconfig: /tmp/target-cluster.kubeconfig
add context target-cluster
Context "target-cluster" created.
Check nodes status
node/target-cluster-control-plane-sq28n condition met
NAME                                 STATUS   ROLES    AGE   VERSION
target-cluster-control-plane-sq28n   Ready    master   48s   v1.18.6
Waiting for  pods to come up
pod/calico-kube-controllers-5569bdd565-9qd8n condition met
pod/calico-node-9mssq condition met
pod/coredns-66bff467f8-9dsfn condition met
pod/coredns-66bff467f8-t7kdw condition met
pod/etcd-target-cluster-control-plane-sq28n condition met
pod/kube-apiserver-target-cluster-control-plane-sq28n condition met
pod/kube-controller-manager-target-cluster-control-plane-sq28n condition met
pod/kube-proxy-xs277 condition met
pod/kube-scheduler-target-cluster-control-plane-sq28n condition met
NAMESPACE     NAME                                                         READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5569bdd565-9qd8n                     1/1     Running   0          79s
kube-system   calico-node-9mssq                                            1/1     Running   0          79s
kube-system   coredns-66bff467f8-9dsfn                                     1/1     Running   0          79s
kube-system   coredns-66bff467f8-t7kdw                                     1/1     Running   0          79s
kube-system   etcd-target-cluster-control-plane-sq28n                      1/1     Running   0          86s
kube-system   kube-apiserver-target-cluster-control-plane-sq28n            1/1     Running   0          86s
kube-system   kube-controller-manager-target-cluster-control-plane-sq28n   1/1     Running   0          86s
kube-system   kube-proxy-xs277                                             1/1     Running   0          79s
kube-system   kube-scheduler-target-cluster-control-plane-sq28n            1/1     Running   0          86s
NAME                                 PROVIDERID                                      PHASE
target-cluster-control-plane-sq28n   docker:////target-cluster-control-plane-sq28n   Running
Get cluster state for target workload cluster 
NAME             PHASE
target-cluster   Provisioned
Target Cluster Kubeconfig
/tmp/target-cluster.kubeconfig
❯ KUBECONFIG=/tmp/target-cluster.kubeconfig ./tools/deployment/provider_common/32_cluster_init_target_node.sh
+ export KUBECONFIG=/tmp/target-cluster.kubeconfig
+ KUBECONFIG=/tmp/target-cluster.kubeconfig
+ export KUBECONFIG_TARGET_CONTEXT=target-cluster
+ KUBECONFIG_TARGET_CONTEXT=target-cluster
+ CONTROL_PLANE_NODES=($(kubectl --kubeconfig $KUBECONFIG get --no-headers=true nodes | grep cluster-control-plane | awk '{print $1}'))
++ kubectl --kubeconfig /tmp/target-cluster.kubeconfig get --no-headers=true nodes
++ grep cluster-control-plane
++ awk '{print $1}'
+ for i in "${CONTROL_PLANE_NODES}"
+ echo untainting node target-cluster-control-plane-sq28n
untainting node target-cluster-control-plane-sq28n
+ kubectl taint node target-cluster-control-plane-sq28n node-role.kubernetes.io/master- --kubeconfig /tmp/target-cluster.kubeconfig --request-timeout 10s
node/target-cluster-control-plane-sq28n untainted
+ echo 'Deploy CAPI components to target cluster'
Deploy CAPI components to target cluster
+ airshipctl phase run clusterctl-init-target --debug --kubeconfig /tmp/target-cluster.kubeconfig
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CABPK_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CACPK_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPD_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPI_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPM3_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CABPK_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CACPK_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPD_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPI_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPM3_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPO_AUTH_PROXY is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/reader.go:109: Verifying that variable CONTAINER_CAPO_MANAGER is allowed to be appended
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/clusterctl/client/client.go:81: Starting cluster-api initiation
[airshipctl] 2020/11/20 12:11:03 opendev.org/airship/airshipctl@/pkg/events/processor.go:61: Received event: {4 2020-11-20 12:11:03.932119582 -0800 PST m=+7.151684220 {InitType {[]} {<nil>} {ApplyEventResourceUpdate ServersideApplied <nil>} {ResourceUpdateEvent <nil> <nil>} {PruneEventResourceUpdate Pruned <nil>} {DeleteEventResourceUpdate Deleted <nil>}} {<nil>} {ResourceUpdateEvent <nil> <nil>} {0 starting clusterctl init executor} {0 } {0 }}
Installing the clusterctl inventory CRD
Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
Fetching providers
[airshipctl] 2020/11/20 12:11:04 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider cluster-api of type CoreProvider
[airshipctl] 2020/11/20 12:11:04 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: CoreProvider, name: cluster-api
[airshipctl] 2020/11/20 12:11:04 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: CoreProvider, name: cluster-api
Fetching File="components.yaml" Provider="cluster-api" Version="v0.3.7"
[airshipctl] 2020/11/20 12:11:04 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capi/v0.3.7
[airshipctl] 2020/11/20 12:11:05 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type BootstrapProvider
[airshipctl] 2020/11/20 12:11:05 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: BootstrapProvider, name: kubeadm
[airshipctl] 2020/11/20 12:11:05 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: BootstrapProvider, name: kubeadm
Fetching File="components.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:11:05 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cabpk/v0.3.7
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type ControlPlaneProvider
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: ControlPlaneProvider, name: kubeadm
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: ControlPlaneProvider, name: kubeadm
Fetching File="components.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cacpk/v0.3.7
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider docker of type InfrastructureProvider
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository_client.go:37: Setting up airshipctl provider Components client
Provider type: InfrastructureProvider, name: docker
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/components_client.go:38: Getting airshipctl provider components, skipping variable substitution: false.
Provider type: InfrastructureProvider, name: docker
Fetching File="components.yaml" Provider="infrastructure-docker" Version="v0.3.11"
[airshipctl] 2020/11/20 12:11:07 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capd/v0.3.11
[airshipctl] 2020/11/20 12:11:08 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider cluster-api of type CoreProvider
Fetching File="metadata.yaml" Provider="cluster-api" Version="v0.3.7"
[airshipctl] 2020/11/20 12:11:08 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capi/v0.3.7
[airshipctl] 2020/11/20 12:11:08 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type BootstrapProvider
Fetching File="metadata.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:11:08 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cabpk/v0.3.7
[airshipctl] 2020/11/20 12:11:09 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider kubeadm of type ControlPlaneProvider
Fetching File="metadata.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
[airshipctl] 2020/11/20 12:11:09 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/cacpk/v0.3.7
[airshipctl] 2020/11/20 12:11:10 opendev.org/airship/airshipctl@/pkg/clusterctl/client/factory.go:86: Creating airshipctl repository implementation interface for provider docker of type InfrastructureProvider
Fetching File="metadata.yaml" Provider="infrastructure-docker" Version="v0.3.11"
[airshipctl] 2020/11/20 12:11:10 opendev.org/airship/airshipctl@/pkg/clusterctl/implementations/repository.go:83: Building cluster-api provider component documents from kustomize path at /tmp/airship/airshipctl/manifests/function/capd/v0.3.11
Creating Namespace="cert-manager-test"
Installing cert-manager Version="v0.16.1"
Creating Namespace="cert-manager"
Creating CustomResourceDefinition="certificaterequests.cert-manager.io"
Creating CustomResourceDefinition="certificates.cert-manager.io"
Creating CustomResourceDefinition="challenges.acme.cert-manager.io"
Creating CustomResourceDefinition="clusterissuers.cert-manager.io"
Creating CustomResourceDefinition="issuers.cert-manager.io"
Creating CustomResourceDefinition="orders.acme.cert-manager.io"
Creating ServiceAccount="cert-manager-cainjector" Namespace="cert-manager"
Creating ServiceAccount="cert-manager" Namespace="cert-manager"
Creating ServiceAccount="cert-manager-webhook" Namespace="cert-manager"
Creating ClusterRole="cert-manager-cainjector"
Creating ClusterRole="cert-manager-controller-issuers"
Creating ClusterRole="cert-manager-controller-clusterissuers"
Creating ClusterRole="cert-manager-controller-certificates"
Creating ClusterRole="cert-manager-controller-orders"
Creating ClusterRole="cert-manager-controller-challenges"
Creating ClusterRole="cert-manager-controller-ingress-shim"
Creating ClusterRole="cert-manager-view"
Creating ClusterRole="cert-manager-edit"
Creating ClusterRoleBinding="cert-manager-cainjector"
Creating ClusterRoleBinding="cert-manager-controller-issuers"
Creating ClusterRoleBinding="cert-manager-controller-clusterissuers"
Creating ClusterRoleBinding="cert-manager-controller-certificates"
Creating ClusterRoleBinding="cert-manager-controller-orders"
Creating ClusterRoleBinding="cert-manager-controller-challenges"
Creating ClusterRoleBinding="cert-manager-controller-ingress-shim"
Creating Role="cert-manager-cainjector:leaderelection" Namespace="kube-system"
Creating Role="cert-manager:leaderelection" Namespace="kube-system"
Creating Role="cert-manager-webhook:dynamic-serving" Namespace="cert-manager"
Creating RoleBinding="cert-manager-cainjector:leaderelection" Namespace="kube-system"
Creating RoleBinding="cert-manager:leaderelection" Namespace="kube-system"
Creating RoleBinding="cert-manager-webhook:dynamic-serving" Namespace="cert-manager"
Creating Service="cert-manager" Namespace="cert-manager"
Creating Service="cert-manager-webhook" Namespace="cert-manager"
Creating Deployment="cert-manager-cainjector" Namespace="cert-manager"
Creating Deployment="cert-manager" Namespace="cert-manager"
Creating Deployment="cert-manager-webhook" Namespace="cert-manager"
Creating MutatingWebhookConfiguration="cert-manager-webhook"
Creating ValidatingWebhookConfiguration="cert-manager-webhook"
Waiting for cert-manager to be available...
Updating Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Issuer="test-selfsigned" Namespace="cert-manager-test"
Creating Certificate="selfsigned-cert" Namespace="cert-manager-test"
Deleting Namespace="cert-manager-test"
Deleting Issuer="test-selfsigned" Namespace="cert-manager-test"
Deleting Certificate="selfsigned-cert" Namespace="cert-manager-test"
Installing Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Creating shared objects Provider="cluster-api" Version="v0.3.7"
Creating Namespace="capi-webhook-system"
Creating CustomResourceDefinition="clusterresourcesetbindings.addons.cluster.x-k8s.io"
Creating CustomResourceDefinition="clusterresourcesets.addons.cluster.x-k8s.io"
Creating CustomResourceDefinition="clusters.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinedeployments.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinehealthchecks.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinepools.exp.cluster.x-k8s.io"
Creating CustomResourceDefinition="machines.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinesets.cluster.x-k8s.io"
Creating Service="capi-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Creating Issuer="capi-selfsigned-issuer" Namespace="capi-webhook-system"
Creating MutatingWebhookConfiguration="capi-mutating-webhook-configuration"
Creating ValidatingWebhookConfiguration="capi-validating-webhook-configuration"
Creating instance objects Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Creating Namespace="capi-system"
Creating Role="capi-leader-election-role" Namespace="capi-system"
Creating ClusterRole="capi-system-capi-aggregated-manager-role"
Creating ClusterRole="capi-system-capi-manager-role"
Creating ClusterRole="capi-system-capi-proxy-role"
Creating RoleBinding="capi-leader-election-rolebinding" Namespace="capi-system"
Creating ClusterRoleBinding="capi-system-capi-manager-rolebinding"
Creating ClusterRoleBinding="capi-system-capi-proxy-rolebinding"
Creating Service="capi-controller-manager-metrics-service" Namespace="capi-system"
Creating Deployment="capi-controller-manager" Namespace="capi-system"
Creating inventory entry Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-bootstrap-system"
Creating shared objects Provider="bootstrap-kubeadm" Version="v0.3.7"
Creating CustomResourceDefinition="kubeadmconfigs.bootstrap.cluster.x-k8s.io"
Creating CustomResourceDefinition="kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io"
Creating Service="capi-kubeadm-bootstrap-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-kubeadm-bootstrap-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-kubeadm-bootstrap-serving-cert" Namespace="capi-webhook-system"
Creating Issuer="capi-kubeadm-bootstrap-selfsigned-issuer" Namespace="capi-webhook-system"
Creating ValidatingWebhookConfiguration="capi-kubeadm-bootstrap-validating-webhook-configuration"
Creating instance objects Provider="bootstrap-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-bootstrap-system"
Creating Namespace="capi-kubeadm-bootstrap-system"
Creating Role="capi-kubeadm-bootstrap-leader-election-role" Namespace="capi-kubeadm-bootstrap-system"
Creating ClusterRole="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-manager-role"
Creating ClusterRole="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-proxy-role"
Creating RoleBinding="capi-kubeadm-bootstrap-leader-election-rolebinding" Namespace="capi-kubeadm-bootstrap-system"
Creating ClusterRoleBinding="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-manager-rolebinding"
Creating ClusterRoleBinding="capi-kubeadm-bootstrap-system-capi-kubeadm-bootstrap-proxy-rolebinding"
Creating Service="capi-kubeadm-bootstrap-controller-manager-metrics-service" Namespace="capi-kubeadm-bootstrap-system"
Creating Deployment="capi-kubeadm-bootstrap-controller-manager" Namespace="capi-kubeadm-bootstrap-system"
Creating inventory entry Provider="bootstrap-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-control-plane-system"
Creating shared objects Provider="control-plane-kubeadm" Version="v0.3.7"
Creating CustomResourceDefinition="kubeadmcontrolplanes.controlplane.cluster.x-k8s.io"
Creating Service="capi-kubeadm-control-plane-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-kubeadm-control-plane-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-kubeadm-control-plane-serving-cert" Namespace="capi-webhook-system"
Creating Issuer="capi-kubeadm-control-plane-selfsigned-issuer" Namespace="capi-webhook-system"
Creating MutatingWebhookConfiguration="capi-kubeadm-control-plane-mutating-webhook-configuration"
Creating ValidatingWebhookConfiguration="capi-kubeadm-control-plane-validating-webhook-configuration"
Creating instance objects Provider="control-plane-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-control-plane-system"
Creating Namespace="capi-kubeadm-control-plane-system"
Creating Role="capi-kubeadm-control-plane-leader-election-role" Namespace="capi-kubeadm-control-plane-system"
Creating Role="capi-kubeadm-control-plane-manager-role" Namespace="capi-kubeadm-control-plane-system"
Creating ClusterRole="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-aggregated-manager-role"
Creating ClusterRole="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-manager-role"
Creating ClusterRole="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-proxy-role"
Creating RoleBinding="capi-kubeadm-control-plane-leader-election-rolebinding" Namespace="capi-kubeadm-control-plane-system"
Creating ClusterRoleBinding="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-manager-rolebinding"
Creating ClusterRoleBinding="capi-kubeadm-control-plane-system-capi-kubeadm-control-plane-proxy-rolebinding"
Creating Service="capi-kubeadm-control-plane-controller-manager-metrics-service" Namespace="capi-kubeadm-control-plane-system"
Creating Deployment="capi-kubeadm-control-plane-controller-manager" Namespace="capi-kubeadm-control-plane-system"
Creating inventory entry Provider="control-plane-kubeadm" Version="v0.3.7" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v0.3.11" TargetNamespace="capd-system"
Creating shared objects Provider="infrastructure-docker" Version="v0.3.11"
Creating CustomResourceDefinition="dockerclusters.infrastructure.cluster.x-k8s.io"
Creating CustomResourceDefinition="dockermachinepools.exp.infrastructure.cluster.x-k8s.io"
Creating CustomResourceDefinition="dockermachines.infrastructure.cluster.x-k8s.io"
Creating CustomResourceDefinition="dockermachinetemplates.infrastructure.cluster.x-k8s.io"
Creating ValidatingWebhookConfiguration="capd-validating-webhook-configuration"
Creating instance objects Provider="infrastructure-docker" Version="v0.3.11" TargetNamespace="capd-system"
Creating Namespace="capd-system"
Creating Role="capd-leader-election-role" Namespace="capd-system"
Creating ClusterRole="capd-system-capd-manager-role"
Creating ClusterRole="capd-system-capd-proxy-role"
Creating RoleBinding="capd-leader-election-rolebinding" Namespace="capd-system"
Creating ClusterRoleBinding="capd-system-capd-manager-rolebinding"
Creating ClusterRoleBinding="capd-system-capd-proxy-rolebinding"
Creating Service="capd-controller-manager-metrics-service" Namespace="capd-system"
Creating Service="capd-webhook-service" Namespace="capd-system"
Creating Deployment="capd-controller-manager" Namespace="capd-system"
Creating Certificate="capd-serving-cert" Namespace="capd-system"
Creating Issuer="capd-selfsigned-issuer" Namespace="capd-system"
Creating inventory entry Provider="infrastructure-docker" Version="v0.3.11" TargetNamespace="capd-system"
[airshipctl] 2020/11/20 12:12:14 opendev.org/airship/airshipctl@/pkg/events/processor.go:61: Received event: {4 2020-11-20 12:12:14.084241881 -0800 PST m=+77.303806555 {InitType {[]} {<nil>} {ApplyEventResourceUpdate ServersideApplied <nil>} {ResourceUpdateEvent <nil> <nil>} {PruneEventResourceUpdate Pruned <nil>} {DeleteEventResourceUpdate Deleted <nil>}} {<nil>} {ResourceUpdateEvent <nil> <nil>} {1 clusterctl init completed successfully} {0 } {0 }}
+ echo 'Waiting for pods to be ready'
Waiting for pods to be ready
+ kubectl --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster wait --all-namespaces --for=condition=Ready pods --all --timeout=600s
pod/capd-controller-manager-5498cfdfb5-6jv8g condition met
pod/capi-kubeadm-bootstrap-controller-manager-556678c94-x992m condition met
pod/capi-kubeadm-control-plane-controller-manager-556d47dffd-l5hl8 condition met
pod/capi-controller-manager-67859f6b78-snc8c condition met
pod/capi-controller-manager-5c785c685c-tk2jh condition met
pod/capi-kubeadm-bootstrap-controller-manager-77658d7745-k5vjf condition met
pod/capi-kubeadm-control-plane-controller-manager-74dcf8b9c-twfdp condition met
pod/cert-manager-578cd6d964-9vdsm condition met
pod/cert-manager-cainjector-5ffff9dd7c-bkn2c condition met
pod/cert-manager-webhook-556b9d7dfd-7zvqx condition met
pod/calico-kube-controllers-5569bdd565-9qd8n condition met
pod/calico-node-9mssq condition met
pod/coredns-66bff467f8-9dsfn condition met
pod/coredns-66bff467f8-t7kdw condition met
pod/etcd-target-cluster-control-plane-sq28n condition met
pod/kube-apiserver-target-cluster-control-plane-sq28n condition met
pod/kube-controller-manager-target-cluster-control-plane-sq28n condition met
pod/kube-proxy-xs277 condition met
pod/kube-scheduler-target-cluster-control-plane-sq28n condition met
+ kubectl --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster get pods --all-namespaces
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-5498cfdfb5-6jv8g                         2/2     Running   0          43s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-556678c94-x992m        2/2     Running   0          52s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-556d47dffd-l5hl8   2/2     Running   0          47s
capi-system                         capi-controller-manager-67859f6b78-snc8c                         2/2     Running   0          57s
capi-webhook-system                 capi-controller-manager-5c785c685c-tk2jh                         2/2     Running   0          59s
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-77658d7745-k5vjf       2/2     Running   0          56s
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-74dcf8b9c-twfdp    2/2     Running   0          51s
cert-manager                        cert-manager-578cd6d964-9vdsm                                    1/1     Running   0          100s
cert-manager                        cert-manager-cainjector-5ffff9dd7c-bkn2c                         1/1     Running   0          100s
cert-manager                        cert-manager-webhook-556b9d7dfd-7zvqx                            1/1     Running   0          100s
kube-system                         calico-kube-controllers-5569bdd565-9qd8n                         1/1     Running   0          3m24s
kube-system                         calico-node-9mssq                                                1/1     Running   0          3m24s
kube-system                         coredns-66bff467f8-9dsfn                                         1/1     Running   0          3m24s
kube-system                         coredns-66bff467f8-t7kdw                                         1/1     Running   0          3m24s
kube-system                         etcd-target-cluster-control-plane-sq28n                          1/1     Running   0          3m31s
kube-system                         kube-apiserver-target-cluster-control-plane-sq28n                1/1     Running   0          3m31s
kube-system                         kube-controller-manager-target-cluster-control-plane-sq28n       1/1     Running   0          3m31s
kube-system                         kube-proxy-xs277                                                 1/1     Running   0          3m24s
kube-system                         kube-scheduler-target-cluster-control-plane-sq28n                1/1     Running   0          3m31s
❯ ./tools/deployment/provider_common/33_cluster_move_target_node.sh./tools/deployment/provider_common/33_cluster_move_target_node.sh
zsh: no such file or directory: ./tools/deployment/provider_common/33_cluster_move_target_node.sh./tools/deployment/provider_common/33_cluster_move_target_node.sh
❯ ./tools/deployment/provider_common/33_cluster_move_target_node.sh
+ export TIMEOUT=3600
+ TIMEOUT=3600
+ export KUBECONFIG=/home/rishabh/.airship/kubeconfig
+ KUBECONFIG=/home/rishabh/.airship/kubeconfig
+ export KUBECONFIG_EPHEMERAL_CONTEXT=ephemeral-cluster
+ KUBECONFIG_EPHEMERAL_CONTEXT=ephemeral-cluster
+ export KUBECONFIG_TARGET_CONTEXT=target-cluster
+ KUBECONFIG_TARGET_CONTEXT=target-cluster
+ export TARGET_KUBECONFIG=/tmp/target-cluster.kubeconfig
+ TARGET_KUBECONFIG=/tmp/target-cluster.kubeconfig
+ echo 'Move Cluster Object to Target Cluster'
Move Cluster Object to Target Cluster
+ KUBECONFIG=/home/rishabh/.airship/kubeconfig:/tmp/target-cluster.kubeconfig
+ kubectl config view --merge --flatten
+ airshipctl phase run clusterctl-move --kubeconfig /tmp/merged_target_ephemeral.kubeconfig
[airshipctl] 2020/11/20 12:13:13 command 'clusterctl move' is going to be executed
[airshipctl] 2020/11/20 12:13:13 Received event: {4 2020-11-20 12:13:13.268312974 -0800 PST m=+7.555888217 {InitType {[]} {<nil>} {ApplyEventResourceUpdate ServersideApplied <nil>} {ResourceUpdateEvent <nil> <nil>} {PruneEventResourceUpdate Pruned <nil>} {DeleteEventResourceUpdate Deleted <nil>}} {<nil>} {ResourceUpdateEvent <nil> <nil>} {2 starting clusterctl move executor} {0 } {0 }}
[airshipctl] 2020/11/20 12:13:16 Received event: {4 2020-11-20 12:13:16.427097164 -0800 PST m=+10.714672384 {InitType {[]} {<nil>} {ApplyEventResourceUpdate ServersideApplied <nil>} {ResourceUpdateEvent <nil> <nil>} {PruneEventResourceUpdate Pruned <nil>} {DeleteEventResourceUpdate Deleted <nil>}} {<nil>} {ResourceUpdateEvent <nil> <nil>} {3 clusterctl move completed successfully} {0 } {0 }}
+ echo 'Waiting for pods to be ready'
Waiting for pods to be ready
+ kubectl --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster wait --all-namespaces --for=condition=Ready pods --all --timeout=3000s
pod/capd-controller-manager-5498cfdfb5-6jv8g condition met
pod/capi-kubeadm-bootstrap-controller-manager-556678c94-x992m condition met
pod/capi-kubeadm-control-plane-controller-manager-556d47dffd-l5hl8 condition met
pod/capi-controller-manager-67859f6b78-snc8c condition met
pod/capi-controller-manager-5c785c685c-tk2jh condition met
pod/capi-kubeadm-bootstrap-controller-manager-77658d7745-k5vjf condition met
pod/capi-kubeadm-control-plane-controller-manager-74dcf8b9c-twfdp condition met
pod/cert-manager-578cd6d964-9vdsm condition met
pod/cert-manager-cainjector-5ffff9dd7c-bkn2c condition met
pod/cert-manager-webhook-556b9d7dfd-7zvqx condition met
pod/calico-kube-controllers-5569bdd565-9qd8n condition met
pod/calico-node-9mssq condition met
pod/coredns-66bff467f8-9dsfn condition met
pod/coredns-66bff467f8-t7kdw condition met
pod/etcd-target-cluster-control-plane-sq28n condition met
pod/kube-apiserver-target-cluster-control-plane-sq28n condition met
pod/kube-controller-manager-target-cluster-control-plane-sq28n condition met
pod/kube-proxy-xs277 condition met
pod/kube-scheduler-target-cluster-control-plane-sq28n condition met
+ kubectl --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster get pods --all-namespaces
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-5498cfdfb5-6jv8g                         2/2     Running   0          66s
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-556678c94-x992m        2/2     Running   0          75s
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-556d47dffd-l5hl8   2/2     Running   0          70s
capi-system                         capi-controller-manager-67859f6b78-snc8c                         2/2     Running   0          80s
capi-webhook-system                 capi-controller-manager-5c785c685c-tk2jh                         2/2     Running   0          82s
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-77658d7745-k5vjf       2/2     Running   0          79s
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-74dcf8b9c-twfdp    2/2     Running   0          74s
cert-manager                        cert-manager-578cd6d964-9vdsm                                    1/1     Running   0          2m3s
cert-manager                        cert-manager-cainjector-5ffff9dd7c-bkn2c                         1/1     Running   0          2m3s
cert-manager                        cert-manager-webhook-556b9d7dfd-7zvqx                            1/1     Running   0          2m3s
kube-system                         calico-kube-controllers-5569bdd565-9qd8n                         1/1     Running   0          3m47s
kube-system                         calico-node-9mssq                                                1/1     Running   0          3m47s
kube-system                         coredns-66bff467f8-9dsfn                                         1/1     Running   0          3m47s
kube-system                         coredns-66bff467f8-t7kdw                                         1/1     Running   0          3m47s
kube-system                         etcd-target-cluster-control-plane-sq28n                          1/1     Running   0          3m54s
kube-system                         kube-apiserver-target-cluster-control-plane-sq28n                1/1     Running   0          3m54s
kube-system                         kube-controller-manager-target-cluster-control-plane-sq28n       1/1     Running   0          3m54s
kube-system                         kube-proxy-xs277                                                 1/1     Running   0          3m47s
kube-system                         kube-scheduler-target-cluster-control-plane-sq28n                1/1     Running   0          3m54s
++ date +%s
+ end=1605906798
+ echo 'Waiting 3600 seconds for crds to be created.'
Waiting 3600 seconds for crds to be created.
+ true
+ kubectl --request-timeout 20s --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster get cluster target-cluster -o json
+ jq .status.controlPlaneReady
+ grep -q true
++ date +%s
+ now=1605903198
+ '[' 1605903198 -gt 1605906798 ']'
+ echo -n .
.+ sleep 15
+ true
+ kubectl --request-timeout 20s --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster get cluster target-cluster -o json
+ jq .status.controlPlaneReady
+ grep -q true
+ echo -e '\nGet CRD status'

Get CRD status
+ kubectl --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster get machines
NAME                                 PROVIDERID                                      PHASE
target-cluster-control-plane-sq28n   docker:////target-cluster-control-plane-sq28n   Running
+ kubectl --kubeconfig /tmp/target-cluster.kubeconfig --context target-cluster get clusters
NAME             PHASE
target-cluster   Provisioned
+ break
❯ WORKERS_COUNT=2 KUBECONFIG=/tmp/target-cluster.kubeconfig TEST_SITE=docker-test-site \
> ./tools/deployment/provider_common/34_deploy_worker_node.sh
+ export TIMEOUT=3600
+ TIMEOUT=3600
+ export KUBECONFIG=/tmp/target-cluster.kubeconfig
+ KUBECONFIG=/tmp/target-cluster.kubeconfig
+ export TARGET_KUBECONFIG=/tmp/target-cluster.kubeconfig
+ TARGET_KUBECONFIG=/tmp/target-cluster.kubeconfig
+ export KUBECONFIG_TARGET_CONTEXT=target-cluster
+ KUBECONFIG_TARGET_CONTEXT=target-cluster
+ export TEST_SITE=docker-test-site
+ TEST_SITE=docker-test-site
+ export WORKERS_COUNT=2
+ WORKERS_COUNT=2
+ export AIRSHIP_SRC=/tmp/airship
+ AIRSHIP_SRC=/tmp/airship
+ sed -i '/value.*/s//value": 2 }/g' /tmp/airship/airshipctl/manifests/site/docker-test-site/target/workers/machine_count.json
+ echo 'Stop/Delete ephemeral node'
Stop/Delete ephemeral node
+ kind delete cluster --name ephemeral-cluster
Deleting cluster "ephemeral-cluster" ...
+ echo 'Deploy worker node'
Deploy worker node
+ airshipctl phase run workers-target --debug --kubeconfig /tmp/target-cluster.kubeconfig
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:129: Getting kubeconfig context name from cluster map
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:134: Getting kubeconfig file information from kubeconfig provider
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:139: Filtering out documents that shouldn't be applied to kubernetes from document bundle
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:147: Using kubeconfig at '/tmp/target-cluster.kubeconfig' and context 'target-cluster'
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/executor.go:118: WaitTimeout: 33m20s
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:76: Getting infos for bundle, inventory id is workers-target
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:106: Inventory Object config Map not found, auto generating Inventory object
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:113: Injecting Inventory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"workers-target"},"name":"airshipit-workers-target","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
[airshipctl] 2020/11/20 12:13:56 opendev.org/airship/airshipctl@/pkg/k8s/applier/applier.go:119: Making sure that inventory object namespace airshipit exists
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/target-cluster-md-0 created
machinedeployment.cluster.x-k8s.io/target-cluster-md-0 created
3 resource(s) applied. 3 created, 0 unchanged, 0 configured
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-md-0 is NotFound: Resource not found
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/target-cluster-md-0 is NotFound: Resource not found
machinedeployment.cluster.x-k8s.io/target-cluster-md-0 is NotFound: Resource not found
dockermachinetemplate.infrastructure.cluster.x-k8s.io/target-cluster-md-0 is Current: Resource is current
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/target-cluster-md-0 is Current: Resource is current
machinedeployment.cluster.x-k8s.io/target-cluster-md-0 is Current: Resource is current
all resources has reached the Current status
+ kubectl wait --for=condition=ready node --all --timeout=1000s --context target-cluster --kubeconfig /tmp/target-cluster.kubeconfig -A
node/target-cluster-control-plane-sq28n condition met
❯ kubectl get machines --kubeconfig /tmp/target-cluster.kubeconfig -w
NAME                                   PROVIDERID                                      PHASE
target-cluster-control-plane-sq28n     docker:////target-cluster-control-plane-sq28n   Running
target-cluster-md-0-84db44cdff-qthvr                                                   Provisioning
target-cluster-md-0-84db44cdff-sjpw7                                                   Provisioning
target-cluster-md-0-84db44cdff-qthvr                                                   Provisioning
target-cluster-md-0-84db44cdff-sjpw7                                                   Provisioning
target-cluster-md-0-84db44cdff-qthvr   docker:////target-cluster-md-0-84db44cdff-qthvr   Provisioning
target-cluster-md-0-84db44cdff-qthvr   docker:////target-cluster-md-0-84db44cdff-qthvr   Running
target-cluster-md-0-84db44cdff-sjpw7   docker:////target-cluster-md-0-84db44cdff-sjpw7   Provisioning
target-cluster-md-0-84db44cdff-qthvr   docker:////target-cluster-md-0-84db44cdff-qthvr   Running
target-cluster-md-0-84db44cdff-sjpw7   docker:////target-cluster-md-0-84db44cdff-sjpw7   Running
target-cluster-md-0-84db44cdff-qthvr   docker:////target-cluster-md-0-84db44cdff-qthvr   Running
target-cluster-md-0-84db44cdff-sjpw7   docker:////target-cluster-md-0-84db44cdff-sjpw7   Running
target-cluster-md-0-84db44cdff-sjpw7   docker:////target-cluster-md-0-84db44cdff-sjpw7   Running
^C%                                                                                                                                                                                         
❯ kubectl get machines --kubeconfig /tmp/target-cluster.kubeconfig
NAME                                   PROVIDERID                                        PHASE
target-cluster-control-plane-sq28n     docker:////target-cluster-control-plane-sq28n     Running
target-cluster-md-0-84db44cdff-qthvr   docker:////target-cluster-md-0-84db44cdff-qthvr   Running
target-cluster-md-0-84db44cdff-sjpw7   docker:////target-cluster-md-0-84db44cdff-sjpw7   Running
mattmceuen commented 3 years ago

I think we need to see the output from the problematic run to be able to dig into this deeper @rkjain. Can you add the debugging commands to the deployment script within your patchset?

ratnopamc commented 3 years ago

Hey @mattmceuen,

All tests pass locally without any issues consistently. Therefore, we can close this issue #413. Will rerun the phases in Zuul and if the issue persists, will open a new issue.

mattmceuen commented 3 years ago

I'll go ahead and close this, we can open a new issue or reopen this one with more debugging info if the issue occurs.

ratnopamc commented 3 years ago

Hi @mattmceuen

This particular issue is fixed now. PS 763232

The fix was to set environment variable KIND_EXPERIMENTAL_DOCKER_NETWORK to "bridge". This was not being set properly in the scripts and because of that the deployment of controlplane was never able to reach target state and the status of the machine was always provisioning. When the subsequent cluster move was attempted, it failed with the below error as a result -

[error during clusterctl move: [cannot start the move operation while the control plane for "/, Kind=" default/target-cluster is not yet initialized, cannot start the move operation while "/, Kind=" default/target-cluster-control-plane-l9f8k is still provisioning the node]]

After the fix, phase run commands including cluster move work fine in Zuul and target cluster for capd is deployed successfully.

airshipbot commented 3 years ago

Issue reopened due to new activity on Gerrit.

airshipbot commented 3 years ago

Related Change #763232

Subject: Integrate common infra provider zuul scripts Link: https://review.opendev.org/c/airship/airshipctl/+/763232 Status: NEW Owner: Rishabh Kumar Jain (rishabh.k.jain@ericsson.com)

This change will close this issue when merged.

Approvals

Code-Review
! None
Verified
+1 Zuul
+1 ATT Airship2.0 CI
Workflow
! None

Last Updated: 2020-12-10 19:13:14 CST

airshipbot commented 3 years ago

The Change that closes this issue was merged.

Golvanig commented 3 years ago

Hello, I just got the same issue with today 2021-02-12 version. I installed the developer version "https://opendev.org/airship/airshipctl/src/branch/master/docs/source/developers.md" I had the following message:

"[airshipctl] 2021/02/12 17:31:53 opendev.org/airship/airshipctl@/pkg/events/processor.go:84: Received error when applying errors to kubernetes timeout after 2000 seconds waiting for 8 resources to reach condition AllCurrent Error events received on channel, errors are: [timeout after 2000 seconds waiting for 8 resources to reach condition AllCurrent]'

in the sequence: sudo -E ./tools/deployment/30_deploy_controlplane.sh

Previous steps seem correct. I can of course give you more information traces (if you ask me what you need exactly - Learning AirshipCtl :))

Thanks Thierry

ratnopamc commented 3 years ago

Hi @Golvanig, please follow the https://github.com/airshipit/airshipctl/blob/master/docs/source/developers.md guide. This captures steps to deploy k8s cluster on bare metal host using airshipctl. If you encounter any issue while following https://github.com/airshipit/airshipctl/blob/master/docs/source/developers.md guide, please feel free to ask in the airship slack channel or open a new issue. We'd be happy to help.

Golvanig commented 3 years ago

Thanks eratnch, This is exactly what I followed - I mean the step by step process, and got the issue equivalent to #413.

To complete the information, I ran also the "./tools/gate/20_run_gate_runner.sh", which gathers all the steps in one, and there it works.