search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.95k stars 4.65k forks source link

kops rolling-update flaw in change-detection logic #1831

Closed nicolasbelanger closed 7 years ago

nicolasbelanger commented 7 years ago

It looks like the kops rolling-update cluster <clusterName> is not detecting change when adding the following into a cluster manifest:

  docker:
    bridgeIP: <someBridgeIP>

Upon kops update cluster <clusterName> --yes, I expected kops rolling-update cluster <clusterName> to show a NEEDUPDATE > 0, but it was not the case. Same applies when trying to add something like:

spec:
  kubeAPIServer:
    runtimeConfig:
      batch/v2alpha1: "true"
      apps/v1alpha1: "true"

FYI:

# kops version
Version 1.5.1 (git-01deca8)

and apiVersion: kops/v1alpha2

chrislovecnm commented 7 years ago

I think this is a flaw in upgrade cluster or nodeup.

Can you give us a:

kops get cluster mycluster -o yaml --full after you have run upgrade? Also a -v 10 on upgrade would be helpful.

nicolasbelanger commented 7 years ago

Result for kops get cluster mycluster -o yaml --full

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2017-02-08T14:50:12Z"
  name: <---8<--->
spec:
  api:
    dns: {}
  channel: stable
  cloudProvider: aws
  clusterDNSDomain: cluster.local
  configBase: s3://<---8<--->
  configStore: s3://<---8<--->
  dnsZone: <---8<--->
  docker:
    bridge: ""
    bridgeIP: 172.18.0.1/16
    ipMasq: false
    ipTables: false
    logLevel: warn
    storage: overlay,aufs
    version: 1.12.3
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
    name: main
  - etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
    name: events
  keyStore: s3://<---8<--->
  kubeAPIServer:
    address: 127.0.0.1
    admissionControl:
    - NamespaceLifecycle
    - LimitRanger
    - ServiceAccount
    - PersistentVolumeLabel
    - DefaultStorageClass
    - ResourceQuota
    allowPrivileged: true
    anonymousAuth: false
    apiServerCount: 1
    basicAuthFile: /srv/kubernetes/basic_auth.csv
    clientCAFile: /srv/kubernetes/ca.crt
    cloudProvider: aws
    etcdServers:
    - http://127.0.0.1:4001
    etcdServersOverrides:
    - /events#http://127.0.0.1:4002
    image: gcr.io/google_containers/kube-apiserver:v1.5.2
    kubeletPreferredAddressTypes:
    - InternalIP
    - Hostname
    - ExternalIP
    - LegacyHostIP
    logLevel: 2
    pathSrvKubernetes: /srv/kubernetes
    pathSrvSshproxy: /srv/sshproxy
    runtimeConfig:
      batch/v2alpha1: "true"
    securePort: 443
    serviceClusterIPRange: 100.64.0.0/13
    storageBackend: etcd2
    tlsCertFile: /srv/kubernetes/server.cert
    tlsPrivateKeyFile: /srv/kubernetes/server.key
    tokenAuthFile: /srv/kubernetes/known_tokens.csv
  kubeControllerManager:
    allocateNodeCIDRs: true
    attachDetachReconcileSyncPeriod: 1m0s
    cloudProvider: aws
    clusterCIDR: 100.96.0.0/11
    clusterName: <---8<--->
    configureCloudRoutes: true
    image: gcr.io/google_containers/kube-controller-manager:v1.5.2
    leaderElection:
      leaderElect: true
    logLevel: 2
    master: 127.0.0.1:8080
    pathSrvKubernetes: /srv/kubernetes
    rootCAFile: /srv/kubernetes/ca.crt
    serviceAccountPrivateKeyFile: /srv/kubernetes/server.key
  kubeDNS:
    domain: cluster.local
    image: gcr.io/google_containers/kubedns-amd64:1.3
    replicas: 2
    serverIP: 100.64.0.10
  kubeProxy:
    cpuRequest: 100m
    image: gcr.io/google_containers/kube-proxy:v1.5.2
    logLevel: 2
    master: https://api.internal.<---8<--->
  kubeScheduler:
    image: gcr.io/google_containers/kube-scheduler:v1.5.2
    leaderElection:
      leaderElect: true
    logLevel: 2
    master: 127.0.0.1:8080
  kubelet:
    allowPrivileged: true
    apiServers: https://api.internal.<---8<--->
    babysitDaemons: true
    cgroupRoot: docker
    cloudProvider: aws
    clusterDNS: 100.64.0.10
    clusterDomain: cluster.local
    enableDebuggingHandlers: true
    evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    evictionPressureTransitionPeriod: "0"
    hostnameOverride: '@aws'
    logLevel: 2
    networkPluginMTU: 9001
    networkPluginName: kubenet
    nonMasqueradeCIDR: 100.64.0.0/10
    podManifestPath: /etc/kubernetes/manifests
  kubernetesApiAccess:
  - <---8<--->
  kubernetesVersion: 1.5.2
  masterInternalName: api.internal.<---8<--->
  masterKubelet:
    allowPrivileged: true
    apiServers: http://127.0.0.1:8080
    babysitDaemons: true
    cgroupRoot: docker
    cloudProvider: aws
    clusterDNS: 100.64.0.10
    clusterDomain: cluster.local
    enableDebuggingHandlers: true
    evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    evictionPressureTransitionPeriod: "0"
    hostnameOverride: '@aws'
    logLevel: 2
    networkPluginMTU: 9001
    networkPluginName: kubenet
    nonMasqueradeCIDR: 100.64.0.0/10
    podManifestPath: /etc/kubernetes/manifests
    registerSchedulable: false
  masterPublicName: api.<---8<--->
  networkCIDR: 172.20.0.0/16
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  secretStore: s3://<---8<--->
  serviceClusterIPRange: 100.64.0.0/13
  sshAccess:
  - <---8<--->
  subnets:
  - cidr: 172.20.32.0/19
    name: eu-west-1a
    type: Public
    zone: eu-west-1a
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

As for kops update cluster mycluster -v 10:

I0209 09:21:26.419605   15325 s3context.go:108] Found bucket "<---8<--->" in region "eu-west-1"
I0209 09:21:26.419699   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/config"
I0209 09:21:27.037235   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/instancegroup/"
I0209 09:21:27.296346   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/instancegroup: [s3://<---8<--->/<---8<--->/instancegroup/master-eu-west-1a s3://<---8<--->/<---8<--->/instancegroup/nodes]
I0209 09:21:27.296383   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/instancegroup/master-eu-west-1a"
I0209 09:21:27.399682   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/instancegroup/nodes"
I0209 09:21:27.508737   15325 channel.go:92] resolving "stable" against default channel location "https://raw.githubusercontent.com/kubernetes/kops/master/channels/"
I0209 09:21:27.508764   15325 channel.go:97] Loading channel from "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable"
I0209 09:21:27.508775   15325 context.go:114] Performing HTTP request: GET https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable
I0209 09:21:27.653213   15325 channel.go:106] Channel contents: spec:
  images:
    # We put the "legacy" version first, for kops versions that don't support versions ( < 1.5.0 )
    - name: kope.io/k8s-1.4-debian-jessie-amd64-hvm-ebs-2016-10-21
      providerID: aws
      kubernetesVersion: ">=1.4.0 <1.5.0"
    - name: kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09
      providerID: aws
      kubernetesVersion: ">=1.5.0"
  cluster:
    kubernetesVersion: v1.4.8
    networking:
      kubenet: {}
  kubernetesVersions:
  - range: ">=1.5.0"
    recommendedVersion: 1.5.2
    requiredVersion: 1.5.1
  - range: "<1.5.0"
    recommendedVersion: 1.4.8
    requiredVersion: 1.4.2
  kopsVersions:
  - range: ">=1.5.0-alpha1"
    recommendedVersion: 1.5.1
    #requiredVersion: 1.5.1
    kubernetesVersion: 1.5.2
  - range: "<1.5.0"
    recommendedVersion: 1.4.4
    #requiredVersion: 1.4.4
    kubernetesVersion: 1.4.8
I0209 09:21:27.653815   15325 populate_cluster_spec.go:337] Defaulted KubeControllerManager.ClusterCIDR to 100.96.0.0/11
I0209 09:21:27.653829   15325 populate_cluster_spec.go:344] Defaulted ServiceClusterIPRange to 100.64.0.0/13
I0209 09:21:27.653845   15325 aws_utils.go:38] Querying EC2 for all valid regions
I0209 09:21:27.834383   15325 aws_cloud.go:606] Querying EC2 for all valid zones in region "eu-west-1"
I0209 09:21:27.834585   15325 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0209 09:21:28.495005   15325 subnets.go:48] All subnets have CIDRs; skipping asssignment logic
I0209 09:21:28.495076   15325 aws_cloud.go:606] Querying EC2 for all valid zones in region "eu-west-1"
I0209 09:21:28.495221   15325 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0209 09:21:28.667078   15325 utils.go:104] Querying for all DNS zones to find match for "<---8<--->"
I0209 09:21:28.863232   15325 populate_cluster_spec.go:235] Defaulting DNS zone to: <---8<--->
I0209 09:21:28.863263   15325 tagbuilder.go:97] tags: [_aws _k8s_1_5 _networking_kubenet]
I0209 09:21:28.863316   15325 tree_walker.go:97] visit "config/_gce"
I0209 09:21:28.863331   15325 tree_walker.go:120] Skipping directory "config/_gce" as tag "_gce" not present
I0209 09:21:28.863337   15325 tree_walker.go:97] visit "config/components"
I0209 09:21:28.863351   15325 tree_walker.go:97] visit "config/components/kubelet"
I0209 09:21:28.863363   15325 tree_walker.go:97] visit "config/components/kubelet/_aws"
I0209 09:21:28.863371   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kubelet/_aws"
I0209 09:21:28.863380   15325 tree_walker.go:97] visit "config/components/kubelet/_aws/kubelet.aws.options"
I0209 09:21:28.863474   15325 tree_walker.go:97] visit "config/components/kubelet/_gce"
I0209 09:21:28.863485   15325 tree_walker.go:120] Skipping directory "config/components/kubelet/_gce" as tag "_gce" not present
I0209 09:21:28.863490   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet"
I0209 09:21:28.863500   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kubelet/_networking_kubenet"
I0209 09:21:28.863507   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/_k8s_1_4"
I0209 09:21:28.863516   15325 tree_walker.go:120] Skipping directory "config/components/kubelet/_networking_kubenet/_k8s_1_4" as tag "_k8s_1_4" not present
I0209 09:21:28.863522   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/_k8s_1_5"
I0209 09:21:28.863530   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kubelet/_networking_kubenet/_k8s_1_5"
I0209 09:21:28.863536   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/_k8s_1_5/kubelet.kubenet.1_5.options"
I0209 09:21:28.863582   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/kubelet.kubenet.options"
I0209 09:21:28.863747   15325 tree_walker.go:97] visit "config/components/kubelet/kubelet.options"
I0209 09:21:28.863875   15325 tree_walker.go:97] visit "config/components/docker"
I0209 09:21:28.863896   15325 tree_walker.go:97] visit "config/components/docker/_networking_kubenet"
I0209 09:21:28.863904   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/docker/_networking_kubenet"
I0209 09:21:28.863910   15325 tree_walker.go:97] visit "config/components/docker/_networking_kubenet/kubenet.options"
I0209 09:21:28.863961   15325 tree_walker.go:97] visit "config/components/docker/docker.options"
I0209 09:21:28.864027   15325 tree_walker.go:97] visit "config/components/docker/_e2e_storage_test_environment"
I0209 09:21:28.864037   15325 tree_walker.go:120] Skipping directory "config/components/docker/_e2e_storage_test_environment" as tag "_e2e_storage_test_environment" not present
I0209 09:21:28.864044   15325 tree_walker.go:97] visit "config/components/docker/_networking_cni"
I0209 09:21:28.864052   15325 tree_walker.go:120] Skipping directory "config/components/docker/_networking_cni" as tag "_networking_cni" not present
I0209 09:21:28.864057   15325 tree_walker.go:97] visit "config/components/kube-apiserver"
I0209 09:21:28.864071   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_aws"
I0209 09:21:28.864079   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_aws"
I0209 09:21:28.864085   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_aws/kube-apiserver.aws.options"
I0209 09:21:28.864166   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_gce"
I0209 09:21:28.864177   15325 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_gce" as tag "_gce" not present
I0209 09:21:28.864183   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_3"
I0209 09:21:28.864191   15325 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_3" as tag "_k8s_1_3" not present
I0209 09:21:28.864196   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_4"
I0209 09:21:28.864204   15325 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_4" as tag "_k8s_1_4" not present
I0209 09:21:28.864209   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5"
I0209 09:21:28.864218   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_k8s_1_5"
I0209 09:21:28.864225   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5/kube-apiserver.options"
I0209 09:21:28.864330   15325 tree_walker.go:97] visit "config/components/kube-apiserver/kube-apiserver.options"
I0209 09:21:28.864454   15325 tree_walker.go:97] visit "config/components/kube-controller-manager"
I0209 09:21:28.864474   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_aws"
I0209 09:21:28.864483   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-controller-manager/_aws"
I0209 09:21:28.864490   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_aws/kube-controller-manager.aws.options"
I0209 09:21:28.864617   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_gce"
I0209 09:21:28.864635   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_gce" as tag "_gce" not present
I0209 09:21:28.864641   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_classic"
I0209 09:21:28.864665   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_networking_classic" as tag "_networking_classic" not present
I0209 09:21:28.864671   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_cni"
I0209 09:21:28.864679   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_networking_cni" as tag "_networking_cni" not present
I0209 09:21:28.864685   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_kubenet"
I0209 09:21:28.864694   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-controller-manager/_networking_kubenet"
I0209 09:21:28.864701   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_kubenet/_networking_external"
I0209 09:21:28.864710   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_networking_kubenet/_networking_external" as tag "_networking_external" not present
I0209 09:21:28.864717   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_kubenet/kube-controller-manager.networking-kubenet.options"
I0209 09:21:28.864768   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/kube-controller-manager.options"
I0209 09:21:28.864851   15325 tree_walker.go:97] visit "config/components/kube-dns"
I0209 09:21:28.864864   15325 tree_walker.go:97] visit "config/components/kube-dns/kube-dns.options"
I0209 09:21:28.864943   15325 tree_walker.go:97] visit "config/components/kube-proxy"
I0209 09:21:28.864955   15325 tree_walker.go:97] visit "config/components/kube-proxy/kube-proxy.options"
I0209 09:21:28.865024   15325 tree_walker.go:97] visit "config/components/kube-scheduler"
I0209 09:21:28.865035   15325 tree_walker.go:97] visit "config/components/kube-scheduler/kube-scheduler.options"
I0209 09:21:28.865122   15325 tree_walker.go:97] visit "config/defaults.options"
I0209 09:21:28.865164   15325 tree_walker.go:97] visit "config/_aws"
I0209 09:21:28.865173   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/_aws"
I0209 09:21:28.865178   15325 tree_walker.go:97] visit "config/_aws/defaults.options"
I0209 09:21:28.865349   15325 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I0209 09:21:28.865442   15325 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I0209 09:21:28.865628   15325 options_loader.go:101] executing template components/kube-controller-manager/kube-controller-manager.options (tags=[])
I0209 09:21:28.865739   15325 options_loader.go:101] executing template components/kube-dns/kube-dns.options (tags=[])
I0209 09:21:28.865812   15325 options_loader.go:101] executing template components/kube-proxy/kube-proxy.options (tags=[])
I0209 09:21:28.865887   15325 options_loader.go:101] executing template components/kube-scheduler/kube-scheduler.options (tags=[])
I0209 09:21:28.865954   15325 options_loader.go:101] executing template components/kubelet/kubelet.options (tags=[])
I0209 09:21:28.866078   15325 options_loader.go:101] executing template defaults.options (tags=[])
I0209 09:21:28.866120   15325 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I0209 09:21:28.866146   15325 options_loader.go:101] executing template components/docker/_networking_kubenet/kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.866280   15325 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I0209 09:21:28.866316   15325 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_5/kube-apiserver.options (tags=[_k8s_1_5])
I0209 09:21:28.866958   15325 options_loader.go:101] executing template components/kube-controller-manager/_aws/kube-controller-manager.aws.options (tags=[_aws])
I0209 09:21:28.867043   15325 options_loader.go:101] executing template components/kube-controller-manager/_networking_kubenet/kube-controller-manager.networking-kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.867094   15325 options_loader.go:101] executing template components/kubelet/_aws/kubelet.aws.options (tags=[_aws])
I0209 09:21:28.867257   15325 options_loader.go:101] executing template components/kubelet/_networking_kubenet/kubelet.kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.867312   15325 options_loader.go:101] executing template components/kubelet/_networking_kubenet/_k8s_1_5/kubelet.kubenet.1_5.options (tags=[_networking_kubenet _k8s_1_5])
I0209 09:21:28.867364   15325 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I0209 09:21:28.867380   15325 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I0209 09:21:28.867387   15325 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I0209 09:21:28.867392   15325 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I0209 09:21:28.867397   15325 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I0209 09:21:28.867406   15325 kubecontrollermanager.go:79] Kubernetes version "1.5.2" supports AttachDetachReconcileSyncPeriod; will configure
I0209 09:21:28.867420   15325 kubecontrollermanager.go:84] AttachDetachReconcileSyncPeriod is not set; will set to default 1m0s
I0209 09:21:28.867702   15325 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I0209 09:21:28.867770   15325 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I0209 09:21:28.867935   15325 options_loader.go:101] executing template components/kube-controller-manager/kube-controller-manager.options (tags=[])
I0209 09:21:28.868050   15325 options_loader.go:101] executing template components/kube-dns/kube-dns.options (tags=[])
I0209 09:21:28.868128   15325 options_loader.go:101] executing template components/kube-proxy/kube-proxy.options (tags=[])
I0209 09:21:28.868195   15325 options_loader.go:101] executing template components/kube-scheduler/kube-scheduler.options (tags=[])
I0209 09:21:28.868268   15325 options_loader.go:101] executing template components/kubelet/kubelet.options (tags=[])
I0209 09:21:28.868414   15325 options_loader.go:101] executing template defaults.options (tags=[])
I0209 09:21:28.868455   15325 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I0209 09:21:28.868475   15325 options_loader.go:101] executing template components/docker/_networking_kubenet/kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.868501   15325 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I0209 09:21:28.868533   15325 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_5/kube-apiserver.options (tags=[_k8s_1_5])
I0209 09:21:28.868602   15325 options_loader.go:101] executing template components/kube-controller-manager/_aws/kube-controller-manager.aws.options (tags=[_aws])
I0209 09:21:28.868634   15325 options_loader.go:101] executing template components/kube-controller-manager/_networking_kubenet/kube-controller-manager.networking-kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.868663   15325 options_loader.go:101] executing template components/kubelet/_aws/kubelet.aws.options (tags=[_aws])
I0209 09:21:28.868705   15325 options_loader.go:101] executing template components/kubelet/_networking_kubenet/kubelet.kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.868740   15325 options_loader.go:101] executing template components/kubelet/_networking_kubenet/_k8s_1_5/kubelet.kubenet.1_5.options (tags=[_networking_kubenet _k8s_1_5])
I0209 09:21:28.868825   15325 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I0209 09:21:28.868834   15325 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I0209 09:21:28.868838   15325 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I0209 09:21:28.868843   15325 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I0209 09:21:28.868848   15325 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I0209 09:21:28.868854   15325 kubecontrollermanager.go:79] Kubernetes version "1.5.2" supports AttachDetachReconcileSyncPeriod; will configure
I0209 09:21:28.869104   15325 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I0209 09:21:28.869164   15325 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I0209 09:21:28.869316   15325 options_loader.go:101] executing template components/kube-controller-manager/kube-controller-manager.options (tags=[])
I0209 09:21:28.869427   15325 options_loader.go:101] executing template components/kube-dns/kube-dns.options (tags=[])
I0209 09:21:28.869494   15325 options_loader.go:101] executing template components/kube-proxy/kube-proxy.options (tags=[])
I0209 09:21:28.869553   15325 options_loader.go:101] executing template components/kube-scheduler/kube-scheduler.options (tags=[])
I0209 09:21:28.869699   15325 options_loader.go:101] executing template components/kubelet/kubelet.options (tags=[])
I0209 09:21:28.869858   15325 options_loader.go:101] executing template defaults.options (tags=[])
I0209 09:21:28.869891   15325 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I0209 09:21:28.869918   15325 options_loader.go:101] executing template components/docker/_networking_kubenet/kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.869943   15325 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I0209 09:21:28.869991   15325 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_5/kube-apiserver.options (tags=[_k8s_1_5])
I0209 09:21:28.870211   15325 options_loader.go:101] executing template components/kube-controller-manager/_aws/kube-controller-manager.aws.options (tags=[_aws])
I0209 09:21:28.870246   15325 options_loader.go:101] executing template components/kube-controller-manager/_networking_kubenet/kube-controller-manager.networking-kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.870287   15325 options_loader.go:101] executing template components/kubelet/_aws/kubelet.aws.options (tags=[_aws])
I0209 09:21:28.870326   15325 options_loader.go:101] executing template components/kubelet/_networking_kubenet/kubelet.kubenet.options (tags=[_networking_kubenet])
I0209 09:21:28.870352   15325 options_loader.go:101] executing template components/kubelet/_networking_kubenet/_k8s_1_5/kubelet.kubenet.1_5.options (tags=[_networking_kubenet _k8s_1_5])
I0209 09:21:28.870432   15325 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I0209 09:21:28.870441   15325 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I0209 09:21:28.870445   15325 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I0209 09:21:28.870450   15325 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I0209 09:21:28.870454   15325 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I0209 09:21:28.870460   15325 kubecontrollermanager.go:79] Kubernetes version "1.5.2" supports AttachDetachReconcileSyncPeriod; will configure
I0209 09:21:28.870752   15325 spec_builder.go:68] options: {
  "channel": "stable",
  "configBase": "s3://<---8<--->/<---8<--->",
  "cloudProvider": "aws",
  "kubernetesVersion": "1.5.2",
  "subnets": [
    {
      "name": "eu-west-1a",
      "zone": "eu-west-1a",
      "cidr": "172.20.32.0/19",
      "type": "Public"
    }
  ],
  "masterPublicName": "api.<---8<--->",
  "masterInternalName": "api.internal.<---8<--->",
  "networkCIDR": "172.20.0.0/16",
  "topology": {
    "masters": "public",
    "nodes": "public",
    "dns": {
      "type": "Public"
    }
  },
  "secretStore": "s3://<---8<--->/<---8<--->/secrets",
  "keyStore": "s3://<---8<--->/<---8<--->/pki",
  "configStore": "s3://<---8<--->/<---8<--->",
  "dnsZone": "<---8<--->",
  "clusterDNSDomain": "cluster.local",
  "serviceClusterIPRange": "100.64.0.0/13",
  "nonMasqueradeCIDR": "100.64.0.0/10",
  "sshAccess": [
    "<---8<--->/29"
  ],
  "kubernetesApiAccess": [
    "<---8<--->/29"
  ],
  "etcdClusters": [
    {
      "name": "main",
      "etcdMembers": [
        {
          "name": "a",
          "instanceGroup": "master-eu-west-1a"
        }
      ]
    },
    {
      "name": "events",
      "etcdMembers": [
        {
          "name": "a",
          "instanceGroup": "master-eu-west-1a"
        }
      ]
    }
  ],
  "docker": {
    "bridge": "",
    "logLevel": "warn",
    "ipTables": false,
    "ipMasq": false,
    "storage": "overlay,aufs",
    "bridgeIP": "172.19.0.1/16",
    "version": "1.12.3"
  },
  "kubeDNS": {
    "image": "gcr.io/google_containers/kubedns-amd64:1.3",
    "replicas": 2,
    "domain": "cluster.local",
    "serverIP": "100.64.0.10"
  },
  "kubeAPIServer": {
    "pathSrvKubernetes": "/srv/kubernetes",
    "pathSrvSshproxy": "/srv/sshproxy",
    "image": "gcr.io/google_containers/kube-apiserver:v1.5.2",
    "logLevel": 2,
    "cloudProvider": "aws",
    "securePort": 443,
    "address": "127.0.0.1",
    "etcdServers": [
      "http://127.0.0.1:4001"
    ],
    "etcdServersOverrides": [
      "/events#http://127.0.0.1:4002"
    ],
    "admissionControl": [
      "NamespaceLifecycle",
      "LimitRanger",
      "ServiceAccount",
      "PersistentVolumeLabel",
      "DefaultStorageClass",
      "ResourceQuota"
    ],
    "serviceClusterIPRange": "100.64.0.0/13",
    "clientCAFile": "/srv/kubernetes/ca.crt",
    "basicAuthFile": "/srv/kubernetes/basic_auth.csv",
    "tlsCertFile": "/srv/kubernetes/server.cert",
    "tlsPrivateKeyFile": "/srv/kubernetes/server.key",
    "tokenAuthFile": "/srv/kubernetes/known_tokens.csv",
    "allowPrivileged": true,
    "apiServerCount": 1,
    "runtimeConfig": {
      "batch/v2alpha1": "true"
    },
    "anonymousAuth": false,
    "kubeletPreferredAddressTypes": [
      "InternalIP",
      "Hostname",
      "ExternalIP",
      "LegacyHostIP"
    ],
    "storageBackend": "etcd2"
  },
  "kubeControllerManager": {
    "master": "127.0.0.1:8080",
    "logLevel": 2,
    "serviceAccountPrivateKeyFile": "/srv/kubernetes/server.key",
    "image": "gcr.io/google_containers/kube-controller-manager:v1.5.2",
    "pathSrvKubernetes": "/srv/kubernetes",
    "cloudProvider": "aws",
    "clusterName": "<---8<--->",
    "clusterCIDR": "100.96.0.0/11",
    "allocateNodeCIDRs": true,
    "configureCloudRoutes": true,
    "rootCAFile": "/srv/kubernetes/ca.crt",
    "leaderElection": {
      "leaderElect": true
    },
    "attachDetachReconcileSyncPeriod": "1m0s"
  },
  "kubeScheduler": {
    "master": "127.0.0.1:8080",
    "logLevel": 2,
    "image": "gcr.io/google_containers/kube-scheduler:v1.5.2",
    "leaderElection": {
      "leaderElect": true
    }
  },
  "kubeProxy": {
    "image": "gcr.io/google_containers/kube-proxy:v1.5.2",
    "cpuRequest": "100m",
    "logLevel": 2,
    "master": "https://api.internal.<---8<--->"
  },
  "kubelet": {
    "apiServers": "https://api.internal.<---8<--->",
    "logLevel": 2,
    "podManifestPath": "/etc/kubernetes/manifests",
    "hostnameOverride": "@aws",
    "allowPrivileged": true,
    "enableDebuggingHandlers": true,
    "clusterDomain": "cluster.local",
    "clusterDNS": "100.64.0.10",
    "networkPluginName": "kubenet",
    "cloudProvider": "aws",
    "cgroupRoot": "docker",
    "babysitDaemons": true,
    "nonMasqueradeCIDR": "100.64.0.0/10",
    "networkPluginMTU": 9001,
    "evictionHard": "memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%",
    "evictionPressureTransitionPeriod": "0"
  },
  "masterKubelet": {
    "apiServers": "http://127.0.0.1:8080",
    "logLevel": 2,
    "podManifestPath": "/etc/kubernetes/manifests",
    "hostnameOverride": "@aws",
    "allowPrivileged": true,
    "enableDebuggingHandlers": true,
    "clusterDomain": "cluster.local",
    "clusterDNS": "100.64.0.10",
    "networkPluginName": "kubenet",
    "cloudProvider": "aws",
    "cgroupRoot": "docker",
    "babysitDaemons": true,
    "registerSchedulable": false,
    "nonMasqueradeCIDR": "100.64.0.0/10",
    "networkPluginMTU": 9001,
    "evictionHard": "memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%",
    "evictionPressureTransitionPeriod": "0"
  },
  "networking": {
    "kubenet": {}
  },
  "api": {
    "dns": {}
  }
}
I0209 09:21:28.871847   15325 channel.go:157] RecommendedVersion="1.5.1", Have="1.5.1".  No upgrade needed.
I0209 09:21:28.871869   15325 channel.go:185] VersionRecommendationSpec does not specify RequiredVersion
I0209 09:21:28.871880   15325 channel.go:137] RecommendedVersion="1.5.2", Have="1.5.2".  No upgrade needed.
I0209 09:21:28.871887   15325 channel.go:177] RequiredVersion="1.5.1", Have="1.5.2".  No upgrade needed.
I0209 09:21:28.871958   15325 apply_cluster.go:196] Adding default kubelet release asset: https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubelet
I0209 09:21:28.871975   15325 context.go:114] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubelet.sha1
I0209 09:21:29.176349   15325 apply_cluster.go:593] Found hash "5e486d4a2700a3a61c4edfd97fb088984a7f734f" for "https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubelet"
I0209 09:21:29.176397   15325 apply_cluster.go:207] Adding default kubectl release asset: https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubectl
I0209 09:21:29.176410   15325 context.go:114] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubectl.sha1
I0209 09:21:29.279092   15325 apply_cluster.go:593] Found hash "10e675883b167140f78ddf7ed92f936dca291647" for "https://storage.googleapis.com/kubernetes-release/release/v1.5.2/bin/linux/amd64/kubectl"
I0209 09:21:29.279241   15325 networking.go:84] Adding default CNI asset: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
I0209 09:21:29.279261   15325 urls.go:40] Using default base url: "https://kubeupv2.s3.amazonaws.com/kops/1.5.1/"
I0209 09:21:29.279267   15325 urls.go:64] Using default nodeup location: "https://kubeupv2.s3.amazonaws.com/kops/1.5.1/linux/amd64/nodeup"
I0209 09:21:29.279294   15325 aws_cloud.go:606] Querying EC2 for all valid zones in region "eu-west-1"
I0209 09:21:29.279406   15325 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0209 09:21:29.571192   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/ssh/public/admin/"
I0209 09:21:29.708163   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/ssh/public/admin: [s3://<---8<--->/<---8<--->/pki/ssh/public/admin/5be9a70debf169cbba99fe1a6acd4d62]
I0209 09:21:29.708192   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/ssh/public/admin/5be9a70debf169cbba99fe1a6acd4d62"
I0209 09:21:29.849837   15325 dns.go:99] Doing DNS lookup to verify NS records for "<---8<--->"
I0209 09:21:29.922829   15325 dns.go:116] Found NS records for "<---8<--->": [ns-1247.awsdns-27.org. ns-1822.awsdns-35.co.uk. ns-35.awsdns-04.com. ns-617.awsdns-13.net.]
I0209 09:21:29.922884   15325 tagbuilder.go:97] tags: [_aws _k8s_1_5 _networking_kubenet]
I0209 09:21:29.922946   15325 tree_walker.go:97] visit "config/defaults.options"
I0209 09:21:29.922961   15325 tree_walker.go:97] visit "config/_aws"
I0209 09:21:29.922968   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/_aws"
I0209 09:21:29.922975   15325 tree_walker.go:97] visit "config/_aws/defaults.options"
I0209 09:21:29.922982   15325 tree_walker.go:97] visit "config/_gce"
I0209 09:21:29.922990   15325 tree_walker.go:120] Skipping directory "config/_gce" as tag "_gce" not present
I0209 09:21:29.922995   15325 tree_walker.go:97] visit "config/components"
I0209 09:21:29.923010   15325 tree_walker.go:97] visit "config/components/kubelet"
I0209 09:21:29.923042   15325 tree_walker.go:97] visit "config/components/kubelet/_aws"
I0209 09:21:29.923050   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kubelet/_aws"
I0209 09:21:29.923057   15325 tree_walker.go:97] visit "config/components/kubelet/_aws/kubelet.aws.options"
I0209 09:21:29.923064   15325 tree_walker.go:97] visit "config/components/kubelet/_gce"
I0209 09:21:29.923072   15325 tree_walker.go:120] Skipping directory "config/components/kubelet/_gce" as tag "_gce" not present
I0209 09:21:29.923077   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet"
I0209 09:21:29.923086   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kubelet/_networking_kubenet"
I0209 09:21:29.923093   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/_k8s_1_4"
I0209 09:21:29.923102   15325 tree_walker.go:120] Skipping directory "config/components/kubelet/_networking_kubenet/_k8s_1_4" as tag "_k8s_1_4" not present
I0209 09:21:29.923108   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/_k8s_1_5"
I0209 09:21:29.923116   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kubelet/_networking_kubenet/_k8s_1_5"
I0209 09:21:29.923123   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/_k8s_1_5/kubelet.kubenet.1_5.options"
I0209 09:21:29.923131   15325 tree_walker.go:97] visit "config/components/kubelet/_networking_kubenet/kubelet.kubenet.options"
I0209 09:21:29.923139   15325 tree_walker.go:97] visit "config/components/kubelet/kubelet.options"
I0209 09:21:29.923146   15325 tree_walker.go:97] visit "config/components/docker"
I0209 09:21:29.923157   15325 tree_walker.go:97] visit "config/components/docker/_e2e_storage_test_environment"
I0209 09:21:29.923165   15325 tree_walker.go:120] Skipping directory "config/components/docker/_e2e_storage_test_environment" as tag "_e2e_storage_test_environment" not present
I0209 09:21:29.923171   15325 tree_walker.go:97] visit "config/components/docker/_networking_cni"
I0209 09:21:29.923179   15325 tree_walker.go:120] Skipping directory "config/components/docker/_networking_cni" as tag "_networking_cni" not present
I0209 09:21:29.923185   15325 tree_walker.go:97] visit "config/components/docker/_networking_kubenet"
I0209 09:21:29.923192   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/docker/_networking_kubenet"
I0209 09:21:29.923198   15325 tree_walker.go:97] visit "config/components/docker/_networking_kubenet/kubenet.options"
I0209 09:21:29.923206   15325 tree_walker.go:97] visit "config/components/docker/docker.options"
I0209 09:21:29.923213   15325 tree_walker.go:97] visit "config/components/kube-apiserver"
I0209 09:21:29.923226   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_aws"
I0209 09:21:29.923234   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_aws"
I0209 09:21:29.923240   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_aws/kube-apiserver.aws.options"
I0209 09:21:29.923248   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_gce"
I0209 09:21:29.923255   15325 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_gce" as tag "_gce" not present
I0209 09:21:29.923260   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_3"
I0209 09:21:29.923267   15325 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_3" as tag "_k8s_1_3" not present
I0209 09:21:29.923273   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_4"
I0209 09:21:29.923280   15325 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_4" as tag "_k8s_1_4" not present
I0209 09:21:29.923285   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5"
I0209 09:21:29.923293   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_k8s_1_5"
I0209 09:21:29.923299   15325 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5/kube-apiserver.options"
I0209 09:21:29.923307   15325 tree_walker.go:97] visit "config/components/kube-apiserver/kube-apiserver.options"
I0209 09:21:29.923314   15325 tree_walker.go:97] visit "config/components/kube-controller-manager"
I0209 09:21:29.923328   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_classic"
I0209 09:21:29.923337   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_networking_classic" as tag "_networking_classic" not present
I0209 09:21:29.923342   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_cni"
I0209 09:21:29.923351   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_networking_cni" as tag "_networking_cni" not present
I0209 09:21:29.923356   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_kubenet"
I0209 09:21:29.923402   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-controller-manager/_networking_kubenet"
I0209 09:21:29.923505   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_kubenet/_networking_external"
I0209 09:21:29.923518   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_networking_kubenet/_networking_external" as tag "_networking_external" not present
I0209 09:21:29.923524   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_networking_kubenet/kube-controller-manager.networking-kubenet.options"
I0209 09:21:29.923534   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/kube-controller-manager.options"
I0209 09:21:29.923542   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_aws"
I0209 09:21:29.923551   15325 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-controller-manager/_aws"
I0209 09:21:29.923557   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_aws/kube-controller-manager.aws.options"
I0209 09:21:29.923565   15325 tree_walker.go:97] visit "config/components/kube-controller-manager/_gce"
I0209 09:21:29.923572   15325 tree_walker.go:120] Skipping directory "config/components/kube-controller-manager/_gce" as tag "_gce" not present
I0209 09:21:29.923578   15325 tree_walker.go:97] visit "config/components/kube-dns"
I0209 09:21:29.923586   15325 tree_walker.go:97] visit "config/components/kube-dns/kube-dns.options"
I0209 09:21:29.923593   15325 tree_walker.go:97] visit "config/components/kube-proxy"
I0209 09:21:29.923602   15325 tree_walker.go:97] visit "config/components/kube-proxy/kube-proxy.options"
I0209 09:21:29.923609   15325 tree_walker.go:97] visit "config/components/kube-scheduler"
I0209 09:21:29.923617   15325 tree_walker.go:97] visit "config/components/kube-scheduler/kube-scheduler.options"
I0209 09:21:29.923627   15325 tree_walker.go:97] visit "cloudup/_gce"
I0209 09:21:29.923636   15325 tree_walker.go:120] Skipping directory "cloudup/_gce" as tag "_gce" not present
I0209 09:21:29.923641   15325 tree_walker.go:97] visit "cloudup/pki"
I0209 09:21:29.923650   15325 tree_walker.go:97] visit "cloudup/pki/kubecfg"
I0209 09:21:29.923661   15325 loader.go:300] Reading cloudup/pki/kubecfg
I0209 09:21:29.923930   15325 loader.go:396] Built pki/kubecfg:keypair/kubecfg => *fitasks.Keypair {"Name":null,"subject":"cn=kubecfg","type":"client","alternateNames":null,"alternateNameTasks":null}
I0209 09:21:29.923973   15325 tree_walker.go:97] visit "cloudup/pki/kubelet"
I0209 09:21:29.923983   15325 loader.go:300] Reading cloudup/pki/kubelet
I0209 09:21:29.924164   15325 loader.go:396] Built pki/kubelet:keypair/kubelet => *fitasks.Keypair {"Name":null,"subject":"cn=kubelet","type":"client","alternateNames":null,"alternateNameTasks":null}
I0209 09:21:29.924179   15325 tree_walker.go:97] visit "cloudup/pki/master"
I0209 09:21:29.924187   15325 loader.go:300] Reading cloudup/pki/master
I0209 09:21:29.924404   15325 loader.go:396] Built pki/master:keypair/master => *fitasks.Keypair {"Name":null,"subject":"cn=kubernetes-master","type":"server","alternateNames":["kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster.local","api.<---8<--->","api.internal.<---8<--->","100.64.0.1"],"alternateNameTasks":null}
I0209 09:21:29.924420   15325 tree_walker.go:97] visit "cloudup/resources"
I0209 09:21:29.924434   15325 tree_walker.go:97] visit "cloudup/resources/cloudinit.yaml.template"
I0209 09:21:29.924461   15325 loader.go:282] loading (templated) resource "cloudinit.yaml"
I0209 09:21:29.924468   15325 tree_walker.go:97] visit "cloudup/resources/cluster-name.template"
I0209 09:21:29.925152   15325 loader.go:282] loading (templated) resource "cluster-name"
I0209 09:21:29.925175   15325 tree_walker.go:97] visit "cloudup/resources/addons"
I0209 09:21:29.925225   15325 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io"
I0209 09:21:29.925246   15325 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/v1.4.0.yaml.template"
I0209 09:21:29.925401   15325 loader.go:282] loading (templated) resource "addons/kube-dns.addons.k8s.io/v1.4.0.yaml"
I0209 09:21:29.925416   15325 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/v1.5.1.yaml.template"
I0209 09:21:29.925642   15325 loader.go:282] loading (templated) resource "addons/kube-dns.addons.k8s.io/v1.5.1.yaml"
I0209 09:21:29.925656   15325 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io"
I0209 09:21:29.925671   15325 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io/addon.yaml"
I0209 09:21:29.925701   15325 loader.go:290] loading resource "addons/limit-range.addons.k8s.io/addon.yaml"
I0209 09:21:29.925707   15325 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I0209 09:21:29.925734   15325 loader.go:290] loading resource "addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I0209 09:21:29.925740   15325 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io"
I0209 09:21:29.925750   15325 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io/v1.0.20161116.yaml"
I0209 09:21:29.925795   15325 loader.go:290] loading resource "addons/networking.kope.io/v1.0.20161116.yaml"
I0209 09:21:29.925802   15325 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org"
I0209 09:21:29.925813   15325 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org/v2.0.yaml.template"
I0209 09:21:29.925940   15325 loader.go:282] loading (templated) resource "addons/networking.projectcalico.org/v2.0.yaml"
I0209 09:21:29.925957   15325 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave"
I0209 09:21:29.925966   15325 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave/v1.8.2.yaml"
I0209 09:21:29.926021   15325 loader.go:290] loading resource "addons/networking.weave/v1.8.2.yaml"
I0209 09:21:29.926029   15325 tree_walker.go:97] visit "cloudup/resources/addons/storage-aws.addons.k8s.io"
I0209 09:21:29.926039   15325 tree_walker.go:97] visit "cloudup/resources/addons/storage-aws.addons.k8s.io/v1.5.0.yaml"
I0209 09:21:29.926127   15325 loader.go:290] loading resource "addons/storage-aws.addons.k8s.io/v1.5.0.yaml"
I0209 09:21:29.926136   15325 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io"
I0209 09:21:29.926148   15325 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io/addon.yaml"
I0209 09:21:29.926184   15325 loader.go:290] loading resource "addons/core.addons.k8s.io/addon.yaml"
I0209 09:21:29.926190   15325 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io/v1.4.0.yaml"
I0209 09:21:29.926256   15325 loader.go:290] loading resource "addons/core.addons.k8s.io/v1.4.0.yaml"
I0209 09:21:29.926266   15325 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io"
I0209 09:21:29.926282   15325 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/addon.yaml"
I0209 09:21:29.926307   15325 loader.go:290] loading resource "addons/dns-controller.addons.k8s.io/addon.yaml"
I0209 09:21:29.926312   15325 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/v1.4.0.yaml"
I0209 09:21:29.926355   15325 loader.go:290] loading resource "addons/dns-controller.addons.k8s.io/v1.4.0.yaml"
I0209 09:21:29.926364   15325 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/v1.4.1.yaml.template"
I0209 09:21:29.926485   15325 loader.go:282] loading (templated) resource "addons/dns-controller.addons.k8s.io/v1.4.1.yaml"
I0209 09:21:29.926496   15325 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/v1.5.1.yaml.template"
I0209 09:21:29.926542   15325 loader.go:282] loading (templated) resource "addons/dns-controller.addons.k8s.io/v1.5.1.yaml"
I0209 09:21:29.926549   15325 tree_walker.go:97] visit "cloudup/tokens"
I0209 09:21:29.926559   15325 tree_walker.go:97] visit "cloudup/tokens/tokens.yaml"
I0209 09:21:29.926565   15325 loader.go:300] Reading cloudup/tokens/tokens.yaml
I0209 09:21:29.926825   15325 loader.go:396] Built tokens/tokens.yaml:secret/admin => *fitasks.Secret {"Name":null}
I0209 09:21:29.926854   15325 loader.go:396] Built tokens/tokens.yaml:secret/kube => *fitasks.Secret {"Name":null}
I0209 09:21:29.926871   15325 loader.go:396] Built tokens/tokens.yaml:secret/system-controller_manager => *fitasks.Secret {"Name":"system:controller_manager"}
I0209 09:21:29.926887   15325 loader.go:396] Built tokens/tokens.yaml:secret/system-dns => *fitasks.Secret {"Name":"system:dns"}
I0209 09:21:29.926898   15325 loader.go:396] Built tokens/tokens.yaml:secret/system-monitoring => *fitasks.Secret {"Name":"system:monitoring"}
I0209 09:21:29.926907   15325 loader.go:396] Built tokens/tokens.yaml:secret/kube-proxy => *fitasks.Secret {"Name":null}
I0209 09:21:29.926916   15325 loader.go:396] Built tokens/tokens.yaml:secret/kubelet => *fitasks.Secret {"Name":null}
I0209 09:21:29.926926   15325 loader.go:396] Built tokens/tokens.yaml:secret/system-logging => *fitasks.Secret {"Name":"system:logging"}
I0209 09:21:29.926937   15325 loader.go:396] Built tokens/tokens.yaml:secret/system-scheduler => *fitasks.Secret {"Name":"system:scheduler"}
I0209 09:21:29.927701   15325 iam_builder.go:210] Ignoring location "s3://<---8<--->/<---8<--->/pki/" because found parent "s3://<---8<--->/<---8<--->/"
I0209 09:21:29.927716   15325 iam_builder.go:210] Ignoring location "s3://<---8<--->/<---8<--->/secrets/" because found parent "s3://<---8<--->/<---8<--->/"
I0209 09:21:29.927724   15325 iam_builder.go:215] Found root location "s3://<---8<--->/<---8<--->/"
I0209 09:21:29.927838   15325 iam_builder.go:210] Ignoring location "s3://<---8<--->/<---8<--->/pki/" because found parent "s3://<---8<--->/<---8<--->/"
I0209 09:21:29.927875   15325 iam_builder.go:210] Ignoring location "s3://<---8<--->/<---8<--->/secrets/" because found parent "s3://<---8<--->/<---8<--->/"
I0209 09:21:29.927883   15325 iam_builder.go:215] Found root location "s3://<---8<--->/<---8<--->/"
I0209 09:21:29.928803   15325 topological_sort.go:62] Dependencies:
I0209 09:21:29.928811   15325 topological_sort.go:64]   SecurityGroup/masters.<---8<--->:   [VPC/<---8<--->]
I0209 09:21:29.928819   15325 topological_sort.go:64]   VPC/<---8<--->: []
I0209 09:21:29.928823   15325 topological_sort.go:64]   IAMInstanceProfile/nodes.<---8<--->:    []
I0209 09:21:29.928828   15325 topological_sort.go:64]   secret/system-logging:  []
I0209 09:21:29.928832   15325 topological_sort.go:64]   IAMRolePolicy/additional.masters.<---8<--->:    [IAMRole/masters.<---8<--->]
I0209 09:21:29.928837   15325 topological_sort.go:64]   secret/system-scheduler:    []
I0209 09:21:29.928842   15325 topological_sort.go:64]   LaunchConfiguration/master-eu-west-1a.masters.<---8<--->:   [SSHKey/kubernetes.<---8<--->-<---8<---> SecurityGroup/masters.<---8<---> IAMInstanceProfile/masters.<---8<--->]
I0209 09:21:29.928848   15325 topological_sort.go:64]   Route/0.0.0.0/0:    [RouteTable/<---8<---> InternetGateway/<---8<--->]
I0209 09:21:29.928855   15325 topological_sort.go:64]   RouteTable/<---8<--->:  [VPC/<---8<--->]
I0209 09:21:29.928859   15325 topological_sort.go:64]   secret/kube-proxy:  []
I0209 09:21:29.928864   15325 topological_sort.go:64]   IAMInstanceProfileRole/nodes.<---8<--->:    [IAMInstanceProfile/nodes.<---8<---> IAMRole/nodes.<---8<--->]
I0209 09:21:29.928869   15325 topological_sort.go:64]   <---8<--->-addons-core.addons.k8s.io:   []
I0209 09:21:29.928874   15325 topological_sort.go:64]   EBSVolume/a.etcd-events.<---8<--->: []
I0209 09:21:29.928879   15325 topological_sort.go:64]   SecurityGroupRule/all-master-to-master: [SecurityGroup/masters.<---8<---> SecurityGroup/masters.<---8<--->]
I0209 09:21:29.928884   15325 topological_sort.go:64]   <---8<--->-addons-storage-aws.addons.k8s.io:    []
I0209 09:21:29.928888   15325 topological_sort.go:64]   RouteTableAssociation/eu-west-1a.<---8<--->:    [RouteTable/<---8<---> Subnet/eu-west-1a.<---8<--->]
I0209 09:21:29.928894   15325 topological_sort.go:64]   EBSVolume/a.etcd-main.<---8<--->:   []
I0209 09:21:29.928898   15325 topological_sort.go:64]   SecurityGroup/nodes.<---8<--->: [VPC/<---8<--->]
I0209 09:21:29.928903   15325 topological_sort.go:64]   IAMRolePolicy/masters.<---8<--->:   [IAMRole/masters.<---8<--->]
I0209 09:21:29.928908   15325 topological_sort.go:64]   secret/admin:   []
I0209 09:21:29.928912   15325 topological_sort.go:64]   IAMInstanceProfileRole/masters.<---8<--->:  [IAMInstanceProfile/masters.<---8<---> IAMRole/masters.<---8<--->]
I0209 09:21:29.928917   15325 topological_sort.go:64]   secret/system-controller_manager:   []
I0209 09:21:29.928922   15325 topological_sort.go:64]   SecurityGroupRule/all-master-to-node:   [SecurityGroup/nodes.<---8<---> SecurityGroup/masters.<---8<--->]
I0209 09:21:29.928927   15325 topological_sort.go:64]   VPCDHCPOptionsAssociation/<---8<--->:   [VPC/<---8<---> DHCPOptions/<---8<--->]
I0209 09:21:29.928932   15325 topological_sort.go:64]   IAMRolePolicy/additional.nodes.<---8<--->:  [IAMRole/nodes.<---8<--->]
I0209 09:21:29.928937   15325 topological_sort.go:64]   secret/kubelet: []
I0209 09:21:29.929206   15325 topological_sort.go:64]   SecurityGroupRule/master-egress:    [SecurityGroup/masters.<---8<--->]
I0209 09:21:29.929261   15325 topological_sort.go:64]   keypair/kubecfg:    []
I0209 09:21:29.929353   15325 topological_sort.go:64]   <---8<--->-addons-limit-range.addons.k8s.io:    []
I0209 09:21:29.929361   15325 topological_sort.go:64]   SSHKey/kubernetes.<---8<--->-<---8<--->:    []
I0209 09:21:29.929368   15325 topological_sort.go:64]   Subnet/eu-west-1a.<---8<--->:   [VPC/<---8<--->]
I0209 09:21:29.929376   15325 topological_sort.go:64]   <---8<--->-addons-kube-dns.addons.k8s.io:   []
I0209 09:21:29.929382   15325 topological_sort.go:64]   SecurityGroupRule/node-to-master-tcp-4194:  [SecurityGroup/masters.<---8<---> SecurityGroup/nodes.<---8<--->]
I0209 09:21:29.929390   15325 topological_sort.go:64]   SecurityGroupRule/node-egress:  [SecurityGroup/nodes.<---8<--->]
I0209 09:21:29.929397   15325 topological_sort.go:64]   <---8<--->-addons-bootstrap:    []
I0209 09:21:29.929404   15325 topological_sort.go:64]   InternetGateway/<---8<--->: [VPC/<---8<--->]
I0209 09:21:29.929411   15325 topological_sort.go:64]   secret/system-dns:  []
I0209 09:21:29.929472   15325 topological_sort.go:64]   IAMRolePolicy/nodes.<---8<--->: [IAMRole/nodes.<---8<--->]
I0209 09:21:29.929481   15325 topological_sort.go:64]   DHCPOptions/<---8<--->: []
I0209 09:21:29.929487   15325 topological_sort.go:64]   AutoscalingGroup/nodes.<---8<--->:  [Subnet/eu-west-1a.<---8<---> LaunchConfiguration/nodes.<---8<--->]
I0209 09:21:29.929496   15325 topological_sort.go:64]   SecurityGroupRule/ssh-external-to-node-<---8<--->/29:   [SecurityGroup/nodes.<---8<--->]
I0209 09:21:29.929503   15325 topological_sort.go:64]   secret/system-monitoring:   []
I0209 09:21:29.929510   15325 topological_sort.go:64]   AutoscalingGroup/master-eu-west-1a.masters.<---8<--->:  [Subnet/eu-west-1a.<---8<---> LaunchConfiguration/master-eu-west-1a.masters.<---8<--->]
I0209 09:21:29.929521   15325 topological_sort.go:64]   secret/kube:    []
I0209 09:21:29.929529   15325 topological_sort.go:64]   SecurityGroupRule/https-external-to-master-<---8<--->/29:   [SecurityGroup/masters.<---8<--->]
I0209 09:21:29.929535   15325 topological_sort.go:64]   IAMRole/masters.<---8<--->: []
I0209 09:21:29.929540   15325 topological_sort.go:64]   IAMInstanceProfile/masters.<---8<--->:  []
I0209 09:21:29.929546   15325 topological_sort.go:64]   keypair/master: []
I0209 09:21:29.929553   15325 topological_sort.go:64]   SecurityGroupRule/node-to-master-tcp-443:   [SecurityGroup/masters.<---8<---> SecurityGroup/nodes.<---8<--->]
I0209 09:21:29.929558   15325 topological_sort.go:64]   keypair/kubelet:    []
I0209 09:21:29.929563   15325 topological_sort.go:64]   <---8<--->-addons-dns-controller.addons.k8s.io: []
I0209 09:21:29.929567   15325 topological_sort.go:64]   SecurityGroupRule/ssh-external-to-master-<---8<--->/29: [SecurityGroup/masters.<---8<--->]
I0209 09:21:29.929573   15325 topological_sort.go:64]   LaunchConfiguration/nodes.<---8<--->:   [SSHKey/kubernetes.<---8<--->-<---8<---> SecurityGroup/nodes.<---8<---> IAMInstanceProfile/nodes.<---8<--->]
I0209 09:21:29.929578   15325 topological_sort.go:64]   SecurityGroupRule/all-node-to-node: [SecurityGroup/nodes.<---8<---> SecurityGroup/nodes.<---8<--->]
I0209 09:21:29.929585   15325 topological_sort.go:64]   IAMRole/nodes.<---8<--->:   []
I0209 09:21:29.929688   15325 executor.go:91] Tasks: 0 done / 55 total; 27 can run
I0209 09:21:29.929865   15325 executor.go:157] Executing task "secret/kubelet": *fitasks.Secret {"Name":"kubelet"}
I0209 09:21:29.929934   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/kubelet"
I0209 09:21:29.929809   15325 executor.go:157] Executing task "<---8<--->-addons-limit-range.addons.k8s.io": *fitasks.ManagedFile {"Name":"<---8<--->-addons-limit-range.addons.k8s.io","Location":"addons/limit-range.addons.k8s.io/v1.5.0.yaml","Contents":{"Name":"addons/limit-range.addons.k8s.io/v1.5.0.yaml","Resource":{}}}
I0209 09:21:29.930010   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I0209 09:21:29.930317   15325 executor.go:157] Executing task "secret/system-controller_manager": *fitasks.Secret {"Name":"system:controller_manager"}
I0209 09:21:29.930390   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/system:controller_manager"
I0209 09:21:29.930680   15325 executor.go:157] Executing task "VPC/<---8<--->": *awstasks.VPC {"Name":"<---8<--->","ID":null,"CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false}
I0209 09:21:29.930867   15325 executor.go:157] Executing task "IAMRole/masters.<---8<--->": *awstasks.IAMRole {"ID":null,"Name":"masters.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}}
I0209 09:21:29.930987   15325 request_logger.go:45] AWS request: ec2/DescribeVpcs
I0209 09:21:29.931086   15325 executor.go:157] Executing task "secret/system-monitoring": *fitasks.Secret {"Name":"system:monitoring"}
I0209 09:21:29.931161   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/system:monitoring"
I0209 09:21:29.931303   15325 request_logger.go:45] AWS request: iam/GetRole
I0209 09:21:29.931438   15325 executor.go:157] Executing task "SSHKey/kubernetes.<---8<--->-<---8<--->": *awstasks.SSHKey {"Name":"kubernetes.<---8<--->-<---8<--->","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":null}
I0209 09:21:29.931486   15325 executor.go:157] Executing task "EBSVolume/a.etcd-main.<---8<--->": *awstasks.EBSVolume {"Name":"a.etcd-main.<---8<--->","ID":null,"AvailabilityZone":"eu-west-1a","VolumeType":"gp2","SizeGB":20,"KmsKeyId":null,"Encrypted":false,"Tags":{"k8s.io/etcd/main":"a/a","k8s.io/role/master":"1"}}
I0209 09:21:29.931691   15325 sshkey.go:200] Computed SSH key fingerprint as "<---8<--->"
I0209 09:21:29.931732   15325 request_logger.go:45] AWS request: ec2/DescribeVolumes
I0209 09:21:29.931797   15325 request_logger.go:45] AWS request: ec2/DescribeKeyPairs
I0209 09:21:29.931853   15325 executor.go:157] Executing task "secret/system-logging": *fitasks.Secret {"Name":"system:logging"}
I0209 09:21:29.931901   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/system:logging"
I0209 09:21:29.931907   15325 executor.go:157] Executing task "<---8<--->-addons-core.addons.k8s.io": *fitasks.ManagedFile {"Name":"<---8<--->-addons-core.addons.k8s.io","Location":"addons/core.addons.k8s.io/v1.4.0.yaml","Contents":{"Name":"addons/core.addons.k8s.io/v1.4.0.yaml","Resource":{}}}
I0209 09:21:29.931958   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/addons/core.addons.k8s.io/v1.4.0.yaml"
I0209 09:21:29.932108   15325 executor.go:157] Executing task "keypair/master": *fitasks.Keypair {"Name":"master","subject":"cn=kubernetes-master","type":"server","alternateNames":["kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster.local","api.<---8<--->","api.internal.<---8<--->","100.64.0.1"],"alternateNameTasks":null}
I0209 09:21:29.932134   15325 executor.go:157] Executing task "keypair/kubelet": *fitasks.Keypair {"Name":"kubelet","subject":"cn=kubelet","type":"client","alternateNames":null,"alternateNameTasks":null}
I0209 09:21:29.932163   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/issued/master/"
I0209 09:21:29.932171   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/issued/kubelet/"
I0209 09:21:29.932355   15325 executor.go:157] Executing task "secret/kube": *fitasks.Secret {"Name":"kube"}
I0209 09:21:29.932385   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/kube"
I0209 09:21:29.932370   15325 executor.go:157] Executing task "secret/system-scheduler": *fitasks.Secret {"Name":"system:scheduler"}
I0209 09:21:29.932407   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/system:scheduler"
I0209 09:21:29.932531   15325 executor.go:157] Executing task "IAMInstanceProfile/masters.<---8<--->": *awstasks.IAMInstanceProfile {"Name":"masters.<---8<--->","ID":null}
I0209 09:21:29.932561   15325 executor.go:157] Executing task "IAMInstanceProfile/nodes.<---8<--->": *awstasks.IAMInstanceProfile {"Name":"nodes.<---8<--->","ID":null}
I0209 09:21:29.932641   15325 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0209 09:21:29.932645   15325 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0209 09:21:29.932675   15325 executor.go:157] Executing task "secret/kube-proxy": *fitasks.Secret {"Name":"kube-proxy"}
I0209 09:21:29.929818   15325 executor.go:157] Executing task "<---8<--->-addons-dns-controller.addons.k8s.io": *fitasks.ManagedFile {"Name":"<---8<--->-addons-dns-controller.addons.k8s.io","Location":"addons/dns-controller.addons.k8s.io/v1.5.1.yaml","Contents":{"Name":"addons/dns-controller.addons.k8s.io/v1.5.1.yaml","Resource":{}}}
I0209 09:21:29.932813   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/addons/dns-controller.addons.k8s.io/v1.5.1.yaml"
I0209 09:21:29.932778   15325 executor.go:157] Executing task "EBSVolume/a.etcd-events.<---8<--->": *awstasks.EBSVolume {"Name":"a.etcd-events.<---8<--->","ID":null,"AvailabilityZone":"eu-west-1a","VolumeType":"gp2","SizeGB":20,"KmsKeyId":null,"Encrypted":false,"Tags":{"k8s.io/etcd/events":"a/a","k8s.io/role/master":"1"}}
I0209 09:21:29.933763   15325 request_logger.go:45] AWS request: ec2/DescribeVolumes
I0209 09:21:29.932795   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/kube-proxy"
I0209 09:21:29.933085   15325 executor.go:157] Executing task "secret/admin": *fitasks.Secret {"Name":"admin"}
I0209 09:21:29.934053   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/admin"
I0209 09:21:29.932920   15325 executor.go:157] Executing task "keypair/kubecfg": *fitasks.Keypair {"Name":"kubecfg","subject":"cn=kubecfg","type":"client","alternateNames":null,"alternateNameTasks":null}
I0209 09:21:29.934358   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/issued/kubecfg/"
I0209 09:21:29.932928   15325 executor.go:157] Executing task "IAMRole/nodes.<---8<--->": *awstasks.IAMRole {"ID":null,"Name":"nodes.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}}
I0209 09:21:29.934607   15325 request_logger.go:45] AWS request: iam/GetRole
I0209 09:21:29.933050   15325 executor.go:157] Executing task "<---8<--->-addons-bootstrap": *fitasks.ManagedFile {"Name":"<---8<--->-addons-bootstrap","Location":"addons/bootstrap-channel.yaml","Contents":{"Name":"","Resource":{}}}
I0209 09:21:29.934677   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/addons/bootstrap-channel.yaml"
I0209 09:21:29.932673   15325 executor.go:157] Executing task "DHCPOptions/<---8<--->": *awstasks.DHCPOptions {"Name":"<---8<--->","ID":null,"DomainName":"eu-west-1.compute.internal","DomainNameServers":"AmazonProvidedDNS"}
I0209 09:21:29.934973   15325 request_logger.go:45] AWS request: ec2/DescribeDhcpOptions
I0209 09:21:29.932912   15325 executor.go:157] Executing task "secret/system-dns": *fitasks.Secret {"Name":"system:dns"}
I0209 09:21:29.935012   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/secrets/system:dns"
I0209 09:21:29.932890   15325 executor.go:157] Executing task "<---8<--->-addons-kube-dns.addons.k8s.io": *fitasks.ManagedFile {"Name":"<---8<--->-addons-kube-dns.addons.k8s.io","Location":"addons/kube-dns.addons.k8s.io/v1.5.1.yaml","Contents":{"Name":"addons/kube-dns.addons.k8s.io/v1.5.1.yaml","Resource":{}}}
I0209 09:21:29.935177   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/addons/kube-dns.addons.k8s.io/v1.5.1.yaml"
I0209 09:21:29.932901   15325 executor.go:157] Executing task "<---8<--->-addons-storage-aws.addons.k8s.io": *fitasks.ManagedFile {"Name":"<---8<--->-addons-storage-aws.addons.k8s.io","Location":"addons/storage-aws.addons.k8s.io/v1.5.0.yaml","Contents":{"Name":"addons/storage-aws.addons.k8s.io/v1.5.0.yaml","Resource":{}}}
I0209 09:21:29.935327   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/addons/storage-aws.addons.k8s.io/v1.5.0.yaml"
I0209 09:21:30.126890   15325 iamrole.go:94] actual RolePolicyDocument was json-equal to expected; returning expected value
I0209 09:21:30.126929   15325 iamrole.go:102] found matching IAMRole "<---8<--->"
I0209 09:21:30.183107   15325 iamrole.go:94] actual RolePolicyDocument was json-equal to expected; returning expected value
I0209 09:21:30.183128   15325 iamrole.go:102] found matching IAMRole "<---8<--->"
I0209 09:21:30.219242   15325 vpc.go:78] found matching VPC *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":null,"Shared":null}
I0209 09:21:30.219398   15325 request_logger.go:45] AWS request: ec2/DescribeVpcAttribute
I0209 09:21:30.516919   15325 ebsvolume.go:90] found existing volume
I0209 09:21:30.516950   15325 changes.go:148] comparing maps: Name a.etcd-main.<---8<---> a.etcd-main.<---8<--->
I0209 09:21:30.516957   15325 changes.go:148] comparing maps: k8s.io/role/master 1 1
I0209 09:21:30.516962   15325 changes.go:148] comparing maps: KubernetesCluster <---8<---> <---8<--->
I0209 09:21:30.516966   15325 changes.go:148] comparing maps: k8s.io/etcd/main a/a a/a
I0209 09:21:30.547225   15325 sshkey.go:89] SSH key fingerprints match; assuming public keys match
I0209 09:21:30.566904   15325 dhcp_options.go:68] found existing DhcpOptions
I0209 09:21:30.574231   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/issued/kubecfg: [s3://<---8<--->/<---8<--->/pki/issued/kubecfg/6384750246061542071750814544.crt]
I0209 09:21:30.574274   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/issued/kubecfg/6384750246061542071750814544.crt"
I0209 09:21:30.586478   15325 ebsvolume.go:90] found existing volume
I0209 09:21:30.586508   15325 changes.go:148] comparing maps: k8s.io/etcd/events a/a a/a
I0209 09:21:30.586516   15325 changes.go:148] comparing maps: k8s.io/role/master 1 1
I0209 09:21:30.586520   15325 changes.go:148] comparing maps: KubernetesCluster <---8<---> <---8<--->
I0209 09:21:30.586525   15325 changes.go:148] comparing maps: Name a.etcd-events.<---8<---> a.etcd-events.<---8<--->
I0209 09:21:30.626411   15325 request_logger.go:45] AWS request: ec2/DescribeVpcAttribute
I0209 09:21:30.626487   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/issued/master: [s3://<---8<--->/<---8<--->/pki/issued/master/6384750248421029136559548902.crt]
I0209 09:21:30.626515   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/issued/master/6384750248421029136559548902.crt"
I0209 09:21:30.626711   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/issued/kubelet: [s3://<---8<--->/<---8<--->/pki/issued/kubelet/6384750245941121724961760448.crt]
I0209 09:21:30.626723   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/issued/kubelet/6384750245941121724961760448.crt"
I0209 09:21:30.690452   15325 ca.go:367] Parsing pem block: "CERTIFICATE"
I0209 09:21:30.690649   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/private/kubecfg/"
I0209 09:21:30.739988   15325 ca.go:367] Parsing pem block: "CERTIFICATE"
I0209 09:21:30.740168   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/private/kubelet/"
I0209 09:21:30.772555   15325 ca.go:367] Parsing pem block: "CERTIFICATE"
I0209 09:21:30.772753   15325 s3fs.go:199] Listing objects in S3 bucket "<---8<--->" with prefix "<---8<--->/pki/private/master/"
I0209 09:21:30.881993   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/private/kubecfg: [s3://<---8<--->/<---8<--->/pki/private/kubecfg/6384750246061542071750814544.key]
I0209 09:21:30.882021   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/private/kubecfg/6384750246061542071750814544.key"
I0209 09:21:30.965099   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/private/master: [s3://<---8<--->/<---8<--->/pki/private/master/6384750248421029136559548902.key]
I0209 09:21:30.965138   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/private/master/6384750248421029136559548902.key"
I0209 09:21:30.976104   15325 s3fs.go:225] Listed files in s3://<---8<--->/<---8<--->/pki/private/kubelet: [s3://<---8<--->/<---8<--->/pki/private/kubelet/6384750245941121724961760448.key]
I0209 09:21:30.976150   15325 s3fs.go:162] Reading file "s3://<---8<--->/<---8<--->/pki/private/kubelet/6384750245941121724961760448.key"
I0209 09:21:31.025865   15325 ca.go:385] Parsing pem block: "RSA PRIVATE KEY"
I0209 09:21:31.116284   15325 ca.go:385] Parsing pem block: "RSA PRIVATE KEY"
I0209 09:21:31.116421   15325 ca.go:385] Parsing pem block: "RSA PRIVATE KEY"
I0209 09:21:31.117339   15325 changes.go:173] comparing slices: 0 100.64.0.1 100.64.0.1
I0209 09:21:31.117362   15325 changes.go:173] comparing slices: 1 api.internal.<---8<---> api.internal.<---8<--->
I0209 09:21:31.117372   15325 changes.go:173] comparing slices: 2 api.<---8<---> api.<---8<--->
I0209 09:21:31.117380   15325 changes.go:173] comparing slices: 3 kubernetes kubernetes
I0209 09:21:31.117387   15325 changes.go:173] comparing slices: 4 kubernetes.default kubernetes.default
I0209 09:21:31.117414   15325 changes.go:173] comparing slices: 5 kubernetes.default.svc kubernetes.default.svc
I0209 09:21:31.117422   15325 changes.go:173] comparing slices: 6 kubernetes.default.svc.cluster.local kubernetes.default.svc.cluster.local
I0209 09:21:31.117511   15325 executor.go:91] Tasks: 27 done / 55 total; 12 can run
I0209 09:21:31.117666   15325 executor.go:157] Executing task "IAMInstanceProfileRole/nodes.<---8<--->": *awstasks.IAMInstanceProfileRole {"Name":"nodes.<---8<--->","InstanceProfile":{"Name":"nodes.<---8<--->","ID":"<---8<--->"},"Role":{"ID":"<---8<--->","Name":"nodes.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}}}
I0209 09:21:31.117914   15325 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0209 09:21:31.117631   15325 executor.go:157] Executing task "Subnet/eu-west-1a.<---8<--->": *awstasks.Subnet {"Name":"eu-west-1a.<---8<--->","ID":null,"VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}
I0209 09:21:31.118414   15325 request_logger.go:45] AWS request: ec2/DescribeSubnets
I0209 09:21:31.117588   15325 executor.go:157] Executing task "VPCDHCPOptionsAssociation/<---8<--->": *awstasks.VPCDHCPOptionsAssociation {"Name":"<---8<--->","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"DHCPOptions":{"Name":"<---8<--->","ID":"<---8<--->","DomainName":"eu-west-1.compute.internal","DomainNameServers":"AmazonProvidedDNS"}}
I0209 09:21:31.118590   15325 aws_cloud.go:527] Calling DescribeVPC for VPC "<---8<--->"
I0209 09:21:31.118753   15325 request_logger.go:45] AWS request: ec2/DescribeVpcs
I0209 09:21:31.117602   15325 executor.go:157] Executing task "IAMInstanceProfileRole/masters.<---8<--->": *awstasks.IAMInstanceProfileRole {"Name":"masters.<---8<--->","InstanceProfile":{"Name":"masters.<---8<--->","ID":"<---8<--->"},"Role":{"ID":"<---8<--->","Name":"masters.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}}}
I0209 09:21:31.119046   15325 request_logger.go:45] AWS request: iam/GetInstanceProfile
I0209 09:21:31.117559   15325 executor.go:157] Executing task "IAMRolePolicy/masters.<---8<--->": *awstasks.IAMRolePolicy {"ID":null,"Name":"masters.<---8<--->","Role":{"ID":"<---8<--->","Name":"masters.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0209 09:21:31.117620   15325 executor.go:157] Executing task "IAMRolePolicy/additional.nodes.<---8<--->": *awstasks.IAMRolePolicy {"ID":null,"Name":"additional.nodes.<---8<--->","Role":{"ID":"<---8<--->","Name":"nodes.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0209 09:21:31.117643   15325 executor.go:157] Executing task "IAMRolePolicy/nodes.<---8<--->": *awstasks.IAMRolePolicy {"ID":null,"Name":"nodes.<---8<--->","Role":{"ID":"<---8<--->","Name":"nodes.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0209 09:21:31.119339   15325 request_logger.go:45] AWS request: iam/GetRolePolicy
I0209 09:21:31.119354   15325 request_logger.go:45] AWS request: iam/GetRolePolicy
I0209 09:21:31.119395   15325 request_logger.go:45] AWS request: iam/GetRolePolicy
I0209 09:21:31.117648   15325 executor.go:157] Executing task "RouteTable/<---8<--->": *awstasks.RouteTable {"Name":"<---8<--->","ID":null,"VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false}}
I0209 09:21:31.117662   15325 executor.go:157] Executing task "IAMRolePolicy/additional.masters.<---8<--->": *awstasks.IAMRolePolicy {"ID":null,"Name":"additional.masters.<---8<--->","Role":{"ID":"<---8<--->","Name":"masters.<---8<--->","RolePolicyDocument":{"Name":"","Resource":{}}},"PolicyDocument":{"Name":"","Resource":{}}}
I0209 09:21:31.119569   15325 request_logger.go:45] AWS request: iam/GetRolePolicy
I0209 09:21:31.119625   15325 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0209 09:21:31.117615   15325 executor.go:157] Executing task "SecurityGroup/masters.<---8<--->": *awstasks.SecurityGroup {"Name":"masters.<---8<--->","ID":null,"Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]}
I0209 09:21:31.119943   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:31.117675   15325 executor.go:157] Executing task "SecurityGroup/nodes.<---8<--->": *awstasks.SecurityGroup {"Name":"nodes.<---8<--->","ID":null,"Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]}
I0209 09:21:31.120265   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:31.117568   15325 executor.go:157] Executing task "InternetGateway/<---8<--->": *awstasks.InternetGateway {"Name":"<---8<--->","ID":null,"VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"Shared":false}
I0209 09:21:31.120493   15325 request_logger.go:45] AWS request: ec2/DescribeInternetGateways
I0209 09:21:31.281213   15325 subnet.go:74] found matching subnet "<---8<--->"
I0209 09:21:31.473665   15325 routetable.go:58] found matching RouteTable "<---8<--->"
I0209 09:21:31.638941   15325 securitygroup.go:74] found matching SecurityGroup "<---8<--->"
I0209 09:21:31.639015   15325 changes.go:173] comparing slices: 0 port=22 port=22
I0209 09:21:31.639057   15325 changes.go:173] comparing slices: 1 port=443 port=443
I0209 09:21:31.639061   15325 changes.go:173] comparing slices: 2 port=4001 port=4001
I0209 09:21:31.639065   15325 changes.go:173] comparing slices: 3 port=4789 port=4789
I0209 09:21:31.639069   15325 changes.go:173] comparing slices: 4 port=179 port=179
I0209 09:21:31.639319   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:31.733670   15325 securitygroup.go:74] found matching SecurityGroup "<---8<--->"
I0209 09:21:31.733704   15325 changes.go:173] comparing slices: 0 port=22 port=22
I0209 09:21:31.733808   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:31.883864   15325 securitygroup.go:330] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"<---8<--->\",\n      UserId: \"119428626494\"\n    }]\n}" (did not match removal rules)
I0209 09:21:31.883917   15325 securitygroup.go:330] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"<---8<--->\",\n      UserId: \"119428626494\"\n    }]\n}" (did not match removal rules)
I0209 09:21:31.883933   15325 securitygroup.go:324] permission matches rule {"Port":22}: {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "<---8<--->/29"
    }],
  ToPort: 22
}
I0209 09:21:32.086809   15325 securitygroup.go:330] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"<---8<--->\",\n      UserId: \"119428626494\"\n    }]\n}" (did not match removal rules)
I0209 09:21:32.086879   15325 securitygroup.go:324] permission matches rule {"Port":22}: {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "<---8<--->/29"
    }],
  ToPort: 22
}
I0209 09:21:32.086940   15325 securitygroup.go:330] Ignoring security group permission "{\n  FromPort: 4194,\n  IpProtocol: \"tcp\",\n  ToPort: 4194,\n  UserIdGroupPairs: [{\n      GroupId: \"<---8<--->\",\n      UserId: \"119428626494\"\n    }]\n}" (did not match removal rules)
I0209 09:21:32.086981   15325 securitygroup.go:324] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "<---8<--->/29"
    }],
  ToPort: 443
}
I0209 09:21:32.087021   15325 securitygroup.go:324] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  ToPort: 443,
  UserIdGroupPairs: [{
      GroupId: "<---8<--->",
      UserId: "119428626494"
    }]
}
I0209 09:21:32.495414   15325 internetgateway.go:91] found matching InternetGateway "igw-936ec1f7"
I0209 09:21:32.495470   15325 executor.go:91] Tasks: 39 done / 55 total; 14 can run
I0209 09:21:32.495518   15325 executor.go:157] Executing task "SecurityGroupRule/node-to-master-tcp-443": *awstasks.SecurityGroupRule {"Name":"node-to-master-tcp-443","SecurityGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"CIDR":null,"Protocol":"tcp","FromPort":443,"ToPort":443,"SourceGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"Egress":null}
I0209 09:21:32.495689   15325 executor.go:157] Executing task "SecurityGroupRule/https-external-to-master-<---8<--->/29": *awstasks.SecurityGroupRule {"Name":"https-external-to-master-<---8<--->/29","SecurityGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"CIDR":"<---8<--->/29","Protocol":"tcp","FromPort":443,"ToPort":443,"SourceGroup":null,"Egress":null}
I0209 09:21:32.495773   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.495856   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.495913   15325 executor.go:157] Executing task "SecurityGroupRule/ssh-external-to-master-<---8<--->/29": *awstasks.SecurityGroupRule {"Name":"ssh-external-to-master-<---8<--->/29","SecurityGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"CIDR":"<---8<--->/29","Protocol":"tcp","FromPort":22,"ToPort":22,"SourceGroup":null,"Egress":null}
I0209 09:21:32.495997   15325 executor.go:157] Executing task "Route/0.0.0.0/0": *awstasks.Route {"Name":"0.0.0.0/0","RouteTable":{"Name":"<---8<--->","ID":"<---8<--->","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false}},"Instance":null,"CIDR":"0.0.0.0/0","InternetGateway":{"Name":"<---8<--->","ID":"igw-936ec1f7","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"Shared":false},"NatGateway":null}
I0209 09:21:32.496123   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.496206   15325 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0209 09:21:32.496245   15325 executor.go:157] Executing task "SecurityGroupRule/node-egress": *awstasks.SecurityGroupRule {"Name":"node-egress","SecurityGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I0209 09:21:32.496468   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.496429   15325 executor.go:157] Executing task "SecurityGroupRule/all-node-to-node": *awstasks.SecurityGroupRule {"Name":"all-node-to-node","SecurityGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"Egress":null}
I0209 09:21:32.496508   15325 executor.go:157] Executing task "LaunchConfiguration/nodes.<---8<--->": *awstasks.LaunchConfiguration {"Name":"nodes.<---8<--->","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"t2.medium","SSHKey":{"Name":"kubernetes.<---8<--->-<---8<--->","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"<---8<--->"},"SecurityGroups":[{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]}],"AdditionalSecurityGroupIDs":null,"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"nodes.<---8<--->","ID":"<---8<--->"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":null}
I0209 09:21:32.496671   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.496709   15325 executor.go:157] Executing task "SecurityGroupRule/node-to-master-tcp-4194": *awstasks.SecurityGroupRule {"Name":"node-to-master-tcp-4194","SecurityGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"CIDR":null,"Protocol":"tcp","FromPort":4194,"ToPort":4194,"SourceGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"Egress":null}
I0209 09:21:32.496763   15325 executor.go:157] Executing task "RouteTableAssociation/eu-west-1a.<---8<--->": *awstasks.RouteTableAssociation {"Name":"eu-west-1a.<---8<--->","ID":null,"RouteTable":{"Name":"<---8<--->","ID":"<---8<--->","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false}},"Subnet":{"Name":"eu-west-1a.<---8<--->","ID":"<---8<--->","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}}
I0209 09:21:32.496945   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.496967   15325 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I0209 09:21:32.497009   15325 executor.go:157] Executing task "SecurityGroupRule/all-master-to-master": *awstasks.SecurityGroupRule {"Name":"all-master-to-master","SecurityGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"Egress":null}
I0209 09:21:32.497053   15325 executor.go:157] Executing task "SecurityGroupRule/master-egress": *awstasks.SecurityGroupRule {"Name":"master-egress","SecurityGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I0209 09:21:32.497167   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.497203   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.497204   15325 executor.go:157] Executing task "SecurityGroupRule/all-master-to-node": *awstasks.SecurityGroupRule {"Name":"all-master-to-node","SecurityGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]},"Egress":null}
I0209 09:21:32.497374   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.497343   15325 executor.go:157] Executing task "SecurityGroupRule/ssh-external-to-node-<---8<--->/29": *awstasks.SecurityGroupRule {"Name":"ssh-external-to-node-<---8<--->/29","SecurityGroup":{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]},"CIDR":"<---8<--->/29","Protocol":"tcp","FromPort":22,"ToPort":22,"SourceGroup":null,"Egress":null}
I0209 09:21:32.497816   15325 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I0209 09:21:32.496753   15325 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
I0209 09:21:32.497006   15325 executor.go:157] Executing task "LaunchConfiguration/master-eu-west-1a.masters.<---8<--->": *awstasks.LaunchConfiguration {"Name":"master-eu-west-1a.masters.<---8<--->","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.<---8<--->-<---8<--->","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"<---8<--->"},"SecurityGroups":[{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]}],"AdditionalSecurityGroupIDs":null,"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"masters.<---8<--->","ID":"<---8<--->"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":null}
I0209 09:21:32.498111   15325 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
I0209 09:21:33.103330   15325 route.go:97] found route matching cidr 0.0.0.0/0
I0209 09:21:33.152384   15325 routetableassociation.go:78] found matching RouteTableAssociation "rtbassoc-8c1112eb"
I0209 09:21:33.224207   15325 launchconfiguration.go:100] found existing AutoscalingLaunchConfiguration: "master-eu-west-1a.masters.<---8<--->-20170208145154"
I0209 09:21:33.224284   15325 aws_cloud.go:558] Calling DescribeImages to resolve name "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0209 09:21:33.224451   15325 request_logger.go:45] AWS request: ec2/DescribeImages
I0209 09:21:33.288527   15325 launchconfiguration.go:100] found existing AutoscalingLaunchConfiguration: "nodes.<---8<--->-20170208145154"
I0209 09:21:33.288603   15325 aws_cloud.go:558] Calling DescribeImages to resolve name "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0209 09:21:33.288725   15325 request_logger.go:45] AWS request: ec2/DescribeImages
I0209 09:21:33.355537   15325 aws_cloud.go:601] Resolved image "ami-fa5c7489"
I0209 09:21:33.355567   15325 launchconfiguration.go:143] Returning matching ImageId as expected name: "ami-fa5c7489" -> "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0209 09:21:33.355638   15325 urls.go:85] Using default protokube location: "https://kubeupv2.s3.amazonaws.com/kops/1.5.1/images/protokube.tar.gz"
I0209 09:21:33.355651   15325 context.go:114] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.5.1/images/protokube.tar.gz.sha1
I0209 09:21:33.563037   15325 apply_cluster.go:593] Found hash "6805cba0ea13805b2fa439914679a083be7ac959" for "https://kubeupv2.s3.amazonaws.com/kops/1.5.1/images/protokube.tar.gz"
I0209 09:21:33.563496   15325 changes.go:173] comparing slices: 0 &{<nil> 0xc820c70508 <nil> *awstasks.VPC null []} &{0xc82065c540 0xc820e10ce8 0xc82065c560 *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false} [port=22 port=443 port=4001 port=4789 port=179]}
I0209 09:21:33.574808   15325 aws_cloud.go:601] Resolved image "ami-fa5c7489"
I0209 09:21:33.574840   15325 launchconfiguration.go:143] Returning matching ImageId as expected name: "ami-fa5c7489" -> "kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09"
I0209 09:21:33.574912   15325 context.go:114] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.5.1/images/protokube.tar.gz.sha1
I0209 09:21:33.597679   15325 apply_cluster.go:593] Found hash "6805cba0ea13805b2fa439914679a083be7ac959" for "https://kubeupv2.s3.amazonaws.com/kops/1.5.1/images/protokube.tar.gz"
I0209 09:21:33.598557   15325 changes.go:173] comparing slices: 0 &{<nil> 0xc820e14448 <nil> *awstasks.VPC null []} &{0xc82065c310 0xc820e7c268 0xc82065c380 *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false} [port=22]}
I0209 09:21:33.598630   15325 executor.go:91] Tasks: 53 done / 55 total; 2 can run
I0209 09:21:33.598653   15325 executor.go:157] Executing task "AutoscalingGroup/nodes.<---8<--->": *awstasks.AutoscalingGroup {"Name":"nodes.<---8<--->","MinSize":1,"MaxSize":1,"Subnets":[{"Name":"eu-west-1a.<---8<--->","ID":"<---8<--->","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}],"Tags":{"k8s.io/role/node":"1"},"LaunchConfiguration":{"Name":"nodes.<---8<--->","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"t2.medium","SSHKey":{"Name":"kubernetes.<---8<--->-<---8<--->","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"<---8<--->"},"SecurityGroups":[{"Name":"nodes.<---8<--->","ID":"<---8<--->","Description":"Security group for nodes","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22"]}],"AdditionalSecurityGroupIDs":null,"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"nodes.<---8<--->","ID":"<---8<--->"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":"nodes.<---8<--->-20170208145154"}}
I0209 09:21:33.599034   15325 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I0209 09:21:33.598962   15325 executor.go:157] Executing task "AutoscalingGroup/master-eu-west-1a.masters.<---8<--->": *awstasks.AutoscalingGroup {"Name":"master-eu-west-1a.masters.<---8<--->","MinSize":1,"MaxSize":1,"Subnets":[{"Name":"eu-west-1a.<---8<--->","ID":"<---8<--->","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}],"Tags":{"k8s.io/role/master":"1"},"LaunchConfiguration":{"Name":"master-eu-west-1a.masters.<---8<--->","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.<---8<--->-<---8<--->","PublicKey":{"Name":"","Resource":{}},"KeyFingerprint":"<---8<--->"},"SecurityGroups":[{"Name":"masters.<---8<--->","ID":"<---8<--->","Description":"Security group for masters","VPC":{"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"]}],"AdditionalSecurityGroupIDs":null,"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"masters.<---8<--->","ID":"<---8<--->"},"RootVolumeSize":20,"RootVolumeType":"gp2","SpotPrice":"","ID":"master-eu-west-1a.masters.<---8<--->-20170208145154"}}
I0209 09:21:33.599195   15325 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I0209 09:21:33.712994   15325 changes.go:173] comparing slices: 0 &{0xc82065cfe0 0xc820c3eb68 *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false} 0xc82065d020 0xc82065d030 0xc82065cde5} &{0xc82065cfe0 0xc820c3eb68 *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false} 0xc82065d020 0xc82065d030 0xc82065cde5}
I0209 09:21:33.713061   15325 changes.go:148] comparing maps: KubernetesCluster <---8<---> <---8<--->
I0209 09:21:33.713067   15325 changes.go:148] comparing maps: Name master-eu-west-1a.masters.<---8<---> master-eu-west-1a.masters.<---8<--->
I0209 09:21:33.713072   15325 changes.go:148] comparing maps: k8s.io/role/master 1 1
I0209 09:21:33.719251   15325 changes.go:173] comparing slices: 0 &{0xc82065cfe0 0xc820c3eb68 *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false} 0xc82065d020 0xc82065d030 0xc82065cde5} &{0xc82065cfe0 0xc820c3eb68 *awstasks.VPC {"Name":"<---8<--->","ID":"<---8<--->","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false} 0xc82065d020 0xc82065d030 0xc82065cde5}
I0209 09:21:33.719293   15325 changes.go:148] comparing maps: KubernetesCluster <---8<---> <---8<--->
I0209 09:21:33.719299   15325 changes.go:148] comparing maps: Name nodes.<---8<---> nodes.<---8<--->
I0209 09:21:33.719304   15325 changes.go:148] comparing maps: k8s.io/role/node 1 1
I0209 09:21:33.719338   15325 executor.go:91] Tasks: 55 done / 55 total; 0 can run
Will create resources:
  IAMRolePolicy/additional.masters.<---8<--->
    Role                    name:masters.<---8<---> id:<---8<--->

  IAMRolePolicy/additional.nodes.<---8<--->
    Role                    name:nodes.<---8<---> id:<---8<--->

I0209 09:21:33.719397   15325 context.go:77] deleting temp dir: "/tmp/deploy249785023"
Must specify --yes to apply changes
nicolasbelanger commented 7 years ago

^^^ That was on a new cluster ^^^ I did the same on a qa cluster, I then got caught to do sudo systemctl daemon-reload && sudo systemctl restart docker manually on the nodes. The config was not applied (though it was on file)...

chrislovecnm commented 7 years ago

The config was not applied (though it was on file)...

More context please.

nicolasbelanger commented 7 years ago

The changes were written on /etc/sysconfig/docker, but still not loaded, which led me to restart docker like ^^^

chrislovecnm commented 7 years ago

So this is a problem with nodeup on the launch of the instance.

ajsheppard commented 7 years ago

Is the bridgeIP parameter being respected at all? I tried changing this value and rebuilding my cluster from scratch and it still uses the docker default value.

nicolasbelanger commented 7 years ago

@ajsheppard Have you checked /etc/sysconfig/docker?

ajsheppard commented 7 years ago

Hi @nicolasbelanger the bip flag is getting passed to the /etc/sysconfig/docker file however it is reporting this error when I replace the instance in the asg.

Feb 23 23:33:11 ip-172-17-125-227 systemd[1]: Starting Docker Socket for the API.
Feb 23 23:33:11 ip-172-17-125-227 systemd[1]: Listening on Docker Socket for the API.
Feb 23 23:33:11 ip-172-17-125-227 systemd[1]: Starting Docker Application Container Engine...
Feb 23 23:33:11 ip-172-17-125-227 systemd[1]: protokube.service: main process exited, code=exited, status=125/n/a
Feb 23 23:33:11 ip-172-17-125-227 systemd[1]: Unit protokube.service entered failed state.
Feb 23 23:33:11 ip-172-17-125-227 docker[527]: /usr/bin/docker: An error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: read unix @->/var/run/docker.sock: read: connection reset by peer.
Feb 23 23:33:11 ip-172-17-125-227 docker[527]: See '/usr/bin/docker run --help'.
Feb 23 23:33:12 ip-172-17-125-227 dockerd[537]: time="2017-02-23T23:33:12.554444821Z" level=warning msg="Your kernel does not support swap memory limit."
Feb 23 23:33:12 ip-172-17-125-227 dockerd[537]: time="2017-02-23T23:33:12.554485009Z" level=warning msg="Your kernel does not support kernel memory limit."
Feb 23 23:33:12 ip-172-17-125-227 dockerd[537]: time="2017-02-23T23:33:12.610192721Z" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: failed to allocate gateway (192.168.20.0): Address already in use"
Feb 23 23:33:12 ip-172-17-125-227 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Feb 23 23:33:12 ip-172-17-125-227 systemd[1]: Failed to start Docker Application Container Engine.
Feb 23 23:33:12 ip-172-17-125-227 systemd[1]: Unit docker.service entered failed state.
Feb 23 23:33:12 ip-172-17-125-227 cloud-final[655]: I0223 23:33:12.977227     740 executor.go:91] Tasks: 60 done / 62 total; 1 can run
Feb 23 23:33:12 ip-172-17-125-227 cloud-final[655]: I0223 23:33:12.977264     740 executor.go:157] Executing task "Service/docker.service": Service: docker.service
Feb 23 23:33:12 ip-172-17-125-227 cloud-final[655]: I0223 23:33:12.977372     740 service.go:119] querying state of service "docker.service"
Feb 23 23:33:12 ip-172-17-125-227 cloud-final[655]: W0223 23:33:12.980450     740 service.go:194] Unknown ActiveState="activating"; will treat as not running
Feb 23 23:33:12 ip-172-17-125-227 cloud-final[655]: I0223 23:33:12.980484     740 changes.go:80] Field changed "Running" actual="false" expected="true"
Feb 23 23:33:12 ip-172-17-125-227 cloud-final[655]: I0223 23:33:12.980527     740 service.go:333] Restarting service "docker.service"
Feb 23 23:33:12 ip-172-17-125-227 systemd[1]: Stopping Docker Application Container Engine...
Feb 23 23:33:12 ip-172-17-125-227 systemd[1]: Stopping Docker Socket for the API.

The docker0 in ifconfig is still reporting with the default bridge cidr so maybe this is being created earlier in the instance initialisation.

chrislovecnm commented 7 years ago

Thanks for the awesome bug report! We always appreciate contributions, btw :) This smells like a bug in the update / validate code or in nodeup.

KashifSaadat commented 7 years ago

Hi @nicolasbelanger, a fix for this was recently merged into kops master branch so now changes to docker and the KubeAPIServer spec are correctly picked up and mark the relevant nodes for requiring a rolling update.

Are you happy for us to close this issue?

nicolasbelanger commented 7 years ago

Yep Thanks!