kops edit cluster trying to apply unwanted changes

[cnw004]

1. What kops version are you running? The command kops version, will display this information.

» kops version
Version 1.15.0

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

» kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-27T17:05:32Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6", GitCommit:"b1d75deca493a24a2f87eb1efde1a569e52fc8d9", GitTreeState:"clean", BuildDate:"2018-12-16T04:30:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using? AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

• kops update cluster nothing to update, everything looks good
• kops edit cluster changed kubernetesVersion from 1.11.6 to 1.12.0
• kops update cluster saw a bunch of things were going to change, didn't like that
• kops edit cluster changed kubernetesVersion from 1.12.0 back to 1.11.6
• kops update cluster still seeing a bunch of things that are going to change, most of them having to do with going from 1.11.6 to 1.12.0 (which is no longer in the cluster config)

5. What happened after the commands executed? The change from 1.11.6 to 1.12.0 seems to have staged changes that are sticking around even after changing the version back to 1.11.6. Note that I never passed the --yes flag. I just ran update to validate what was going to change and then reverted back after I saw how much it was going to do.

6. What did you expect to happen? After I changed from 1.12.0 back to 1.11.6 I expected the changes seen by kops update cluster to go away, because no changes should have been expected

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2019-02-01T20:05:17Z"
  generation: 1
  name: REDACTED
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: REDACTED
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
    name: main
    version: 3.2.24
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
    name: events
    version: 3.2.24
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeDNS:
    provider: CoreDNS
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.11.6
  masterInternalName: REDACTED
  masterPublicName: REDACTED
  networkCIDR: 10.40.0.0/16
  networkID: REDACTED
  networking:
    calico:
      majorVersion: v3
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 10.40.100.0/24
    id: REDACTED
    name: us-east-1a
    type: Private
    zone: us-east-1a
  - cidr: 10.40.101.0/24
    id: REDACTED
    name: utility-us-east-1a
    type: Utility
    zone: us-east-1a
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

*********************************************************************************

A new kubernetes version is available: 1.11.10
Upgrading is recommended (try kops upgrade cluster)

More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_k8s.md#1.11.10

*********************************************************************************

Will create resources:
  ManagedFile/prod.kubes.cognius.net-addons-coredns.addons.k8s.io-k8s-1.12
    Location                addons/coredns.addons.k8s.io/k8s-1.12.yaml

  ManagedFile/prod.kubes.cognius.net-addons-dns-controller.addons.k8s.io-k8s-1.12
    Location                addons/dns-controller.addons.k8s.io/k8s-1.12.yaml

  ManagedFile/prod.kubes.cognius.net-addons-kubelet-api.rbac.addons.k8s.io-k8s-1.9
    Location                addons/kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml

  ManagedFile/prod.kubes.cognius.net-addons-networking.projectcalico.org-k8s-1.12
    Location                addons/networking.projectcalico.org/k8s-1.12.yaml

  ManagedFile/prod.kubes.cognius.net-addons-storage-aws.addons.k8s.io-v1.15.0
    Location                addons/storage-aws.addons.k8s.io/v1.15.0.yaml

  SecurityGroupRule/icmp-pmtu-api-elb-0.0.0.0/0
    SecurityGroup           name:api-elb.prod.kubes.cognius.net id:sg-0e7365f6fbad2d66a
    CIDR                    0.0.0.0/0
    Protocol                icmp
    FromPort                3
    ToPort                  4

Will modify resources:
  AutoscalingGroup/composer-nodes.prod.kubes.cognius.net
    Tags                     {KubernetesCluster: prod.kubes.cognius.net, Name: composer-nodes.prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: composer-nodes, k8s.io/cluster-autoscaler/node-template/taint/dedicated: composers:NoSchedule, k8s.io/role/node: 1} -> {k8s.io/cluster-autoscaler/node-template/taint/dedicated: composers:NoSchedule, k8s.io/role/node: 1, kops.k8s.io/instancegroup: composer-nodes, Name: composer-nodes.prod.kubes.cognius.net, KubernetesCluster: prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: composer-nodes}

  AutoscalingGroup/master-us-east-1a.masters.prod.kubes.cognius.net
    Tags                     {KubernetesCluster: prod.kubes.cognius.net, Name: master-us-east-1a.masters.prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: master-us-east-1a, k8s.io/role/master: 1} -> {kops.k8s.io/instancegroup: master-us-east-1a, Name: master-us-east-1a.masters.prod.kubes.cognius.net, KubernetesCluster: prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: master-us-east-1a, k8s.io/role/master: 1}

  AutoscalingGroup/nodes.prod.kubes.cognius.net
    Tags                     {k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: nodes, k8s.io/role/node: 1, KubernetesCluster: prod.kubes.cognius.net, Name: nodes.prod.kubes.cognius.net} -> {k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: nodes, k8s.io/role/node: 1, kops.k8s.io/instancegroup: nodes, Name: nodes.prod.kubes.cognius.net, KubernetesCluster: prod.kubes.cognius.net}

  AutoscalingGroup/pony-nodes.prod.kubes.cognius.net
    Tags                     {KubernetesCluster: prod.kubes.cognius.net, Name: pony-nodes.prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: pony-nodes, k8s.io/cluster-autoscaler/node-template/taint/dedicated: pony:NoSchedule, k8s.io/role/node: 1} -> {k8s.io/cluster-autoscaler/node-template/taint/dedicated: pony:NoSchedule, k8s.io/role/node: 1, kops.k8s.io/instancegroup: pony-nodes, Name: pony-nodes.prod.kubes.cognius.net, KubernetesCluster: prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: pony-nodes}

  AutoscalingGroup/spot-nodes.prod.kubes.cognius.net
    Tags                     {Name: spot-nodes.prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: spot-nodes, k8s.io/role/node: 1, KubernetesCluster: prod.kubes.cognius.net} -> {kops.k8s.io/instancegroup: spot-nodes, Name: spot-nodes.prod.kubes.cognius.net, KubernetesCluster: prod.kubes.cognius.net, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: spot-nodes, k8s.io/role/node: 1}

  IAMRolePolicy/masters.prod.kubes.cognius.net
    PolicyDocument      
                            ...
                                    "Effect": "Allow",
                                    "Action": [
                            +         "ec2:DescribeAccountAttributes",
                            +         "ec2:DescribeInstances",
                            +         "ec2:DescribeInternetGateways",
                            -         "ec2:DescribeInstances",
                                      "ec2:DescribeRegions",
                                      "ec2:DescribeRouteTables",
                            ...
                                      "autoscaling:DescribeAutoScalingGroups",
                                      "autoscaling:DescribeLaunchConfigurations",
                            +         "autoscaling:DescribeTags",
                            +         "ec2:DescribeLaunchTemplateVersions"
                            -         "autoscaling:DescribeTags"
                                    ],
                                    "Resource": [
                            ...

  LaunchConfiguration/composer-nodes.prod.kubes.cognius.net
    UserData            
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/nodeup,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-nodeup,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/nodeup
                            + NODEUP_HASH=9604ef18267ad7b5cf4cebbf7ab64423cf5bb0342d169c608ac6376e6af26d81
                            - NODEUP_HASH=6ee282d77600c47ed7744435400e163fa34ee17e

                              export AWS_REGION=us-east-1
                            ...
                              }

                            - # Retry a download until we get it. Takes a hash and a set of URLs.
                            - #
                            - # $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
                            + # Retry a download until we get it. args: name, sha, url1, url2...
                            - # $2+ are the URLs to download.
                              download-or-bust() {
                            +   local -r file="$1"
                            +   local -r hash="$2"
                            -   local -r hash="$1"
                            +   shift 2
                            -   shift 1

                                urls=( $* )
                                while true; do
                                  for url in "${urls[@]}"; do
                            -       local file="${url##*/}"
                            - 
                                    if [[ -e "${file}" ]]; then
                                      echo "== File exists for ${url} =="
                            ...
                                      if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 "${url}"; then
                                        echo "== Failed to curl ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    else
                                      if ! wget --inet4-only -O "${file}" --connect-timeout=20 --tries=6 --wait=10 "${url}"; then
                                        echo "== Failed to wget ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    fi
                            ...
                                local actual

                            +   actual=$(sha256sum ${file} | awk '{ print $1 }') || true
                            -   actual=$(sha1sum ${file} | awk '{ print $1 }') || true
                                if [[ "${actual}" != "${expected}" ]]; then
                            +     echo "== ${file} corrupted, hash ${actual} doesn't match expected ${expected} =="
                            -     echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
                                  return 1
                                fi
                            ...

                                local -r nodeup_urls=( $(split-commas "${NODEUP_URL}") )
                            -   local -r nodeup_filename="${nodeup_urls[0]##*/}"
                                if [[ -n "${NODEUP_HASH:-}" ]]; then
                                  local -r nodeup_hash="${NODEUP_HASH}"
                                else
                                # TODO: Remove?
                            +     echo "Downloading sha256 (not found in env)"
                            -     echo "Downloading sha1 (not found in env)"
                            +     download-or-bust nodeup.sha256 "" "${nodeup_urls[@]/%/.sha256}"
                            -     download-or-bust "" "${nodeup_urls[@]/%/.sha1}"
                            +     local -r nodeup_hash=$(cat nodeup.sha256)
                            -     local -r nodeup_hash=$(cat "${nodeup_filename}.sha1")
                                fi

                                echo "Downloading nodeup (${nodeup_urls[@]})"
                            +   download-or-bust nodeup "${nodeup_hash}" "${nodeup_urls[@]}"
                            -   download-or-bust "${nodeup_hash}" "${nodeup_urls[@]}"

                                chmod +x nodeup
                            ...
                              - a006b4680640e5c88742e22b904623a77257f416@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubelet
                              - c3f7fbab5ba39e3ec20b32f0e7bcad6cc0704792@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubectl
                            + - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
                            - - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - 71b7bc444ba0a5f7cd7a36e91b594c1c3d13890e160d85e0dfde38c46a24e416@https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/utils.tar.gz,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-utils.tar.gz,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/utils.tar.gz
                            - - b16b5367e05bad082f416f786c7f8813f7794630@https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/utils.tar.gz
                              ClusterName: prod.kubes.cognius.net
                              ConfigBase: s3://hopjump-kops-state-store/prod.kubes.cognius.net
                            ...
                              - s3://hopjump-kops-state-store/prod.kubes.cognius.net/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 42a9c4324fe26d63ce11f3dd7836371bc93fa06ca8f479807728f3746e27061b
                            -   hash: 725c2de47755544a9aa349e27ed9900d195f0ceb
                            +   name: protokube:1.15.0
                            -   name: protokube:1.11.0
                            +   sources:
                            +   - https://artifacts.k8s.io/binaries/kops/1.15.0/images/protokube.tar.gz
                            +   - https://github.com/kubernetes/kops/releases/download/1.15.0/images-protokube.tar.gz
                            +   - https://kubeupv2.s3.amazonaws.com/kops/1.15.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.11.0/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...

  LaunchConfiguration/master-us-east-1a.masters.prod.kubes.cognius.net
    UserData            
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/nodeup,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-nodeup,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/nodeup
                            + NODEUP_HASH=9604ef18267ad7b5cf4cebbf7ab64423cf5bb0342d169c608ac6376e6af26d81
                            - NODEUP_HASH=6ee282d77600c47ed7744435400e163fa34ee17e

                              export AWS_REGION=us-east-1
                            ...
                              }

                            - # Retry a download until we get it. Takes a hash and a set of URLs.
                            - #
                            - # $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
                            + # Retry a download until we get it. args: name, sha, url1, url2...
                            - # $2+ are the URLs to download.
                              download-or-bust() {
                            +   local -r file="$1"
                            +   local -r hash="$2"
                            -   local -r hash="$1"
                            +   shift 2
                            -   shift 1

                                urls=( $* )
                                while true; do
                                  for url in "${urls[@]}"; do
                            -       local file="${url##*/}"
                            - 
                                    if [[ -e "${file}" ]]; then
                                      echo "== File exists for ${url} =="
                            ...
                                      if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 "${url}"; then
                                        echo "== Failed to curl ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    else
                                      if ! wget --inet4-only -O "${file}" --connect-timeout=20 --tries=6 --wait=10 "${url}"; then
                                        echo "== Failed to wget ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    fi
                            ...
                                local actual

                            +   actual=$(sha256sum ${file} | awk '{ print $1 }') || true
                            -   actual=$(sha1sum ${file} | awk '{ print $1 }') || true
                                if [[ "${actual}" != "${expected}" ]]; then
                            +     echo "== ${file} corrupted, hash ${actual} doesn't match expected ${expected} =="
                            -     echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
                                  return 1
                                fi
                            ...

                                local -r nodeup_urls=( $(split-commas "${NODEUP_URL}") )
                            -   local -r nodeup_filename="${nodeup_urls[0]##*/}"
                                if [[ -n "${NODEUP_HASH:-}" ]]; then
                                  local -r nodeup_hash="${NODEUP_HASH}"
                                else
                                # TODO: Remove?
                            +     echo "Downloading sha256 (not found in env)"
                            -     echo "Downloading sha1 (not found in env)"
                            +     download-or-bust nodeup.sha256 "" "${nodeup_urls[@]/%/.sha256}"
                            -     download-or-bust "" "${nodeup_urls[@]/%/.sha1}"
                            +     local -r nodeup_hash=$(cat nodeup.sha256)
                            -     local -r nodeup_hash=$(cat "${nodeup_filename}.sha1")
                                fi

                                echo "Downloading nodeup (${nodeup_urls[@]})"
                            +   download-or-bust nodeup "${nodeup_hash}" "${nodeup_urls[@]}"
                            -   download-or-bust "${nodeup_hash}" "${nodeup_urls[@]}"

                                chmod +x nodeup
                            ...
                              - a006b4680640e5c88742e22b904623a77257f416@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubelet
                              - c3f7fbab5ba39e3ec20b32f0e7bcad6cc0704792@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubectl
                            + - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
                            - - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - 71b7bc444ba0a5f7cd7a36e91b594c1c3d13890e160d85e0dfde38c46a24e416@https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/utils.tar.gz,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-utils.tar.gz,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/utils.tar.gz
                            - - b16b5367e05bad082f416f786c7f8813f7794630@https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/utils.tar.gz
                              ClusterName: prod.kubes.cognius.net
                              ConfigBase: s3://hopjump-kops-state-store/prod.kubes.cognius.net
                            ...
                              - s3://hopjump-kops-state-store/prod.kubes.cognius.net/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 42a9c4324fe26d63ce11f3dd7836371bc93fa06ca8f479807728f3746e27061b
                            -   hash: 725c2de47755544a9aa349e27ed9900d195f0ceb
                            +   name: protokube:1.15.0
                            -   name: protokube:1.11.0
                            +   sources:
                            +   - https://artifacts.k8s.io/binaries/kops/1.15.0/images/protokube.tar.gz
                            +   - https://github.com/kubernetes/kops/releases/download/1.15.0/images-protokube.tar.gz
                            +   - https://kubeupv2.s3.amazonaws.com/kops/1.15.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.11.0/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...

  LaunchConfiguration/nodes.prod.kubes.cognius.net
    UserData            
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/nodeup,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-nodeup,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/nodeup
                            + NODEUP_HASH=9604ef18267ad7b5cf4cebbf7ab64423cf5bb0342d169c608ac6376e6af26d81
                            - NODEUP_HASH=6ee282d77600c47ed7744435400e163fa34ee17e

                              export AWS_REGION=us-east-1
                            ...
                              }

                            - # Retry a download until we get it. Takes a hash and a set of URLs.
                            - #
                            - # $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
                            + # Retry a download until we get it. args: name, sha, url1, url2...
                            - # $2+ are the URLs to download.
                              download-or-bust() {
                            +   local -r file="$1"
                            +   local -r hash="$2"
                            -   local -r hash="$1"
                            +   shift 2
                            -   shift 1

                                urls=( $* )
                                while true; do
                                  for url in "${urls[@]}"; do
                            -       local file="${url##*/}"
                            - 
                                    if [[ -e "${file}" ]]; then
                                      echo "== File exists for ${url} =="
                            ...
                                      if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 "${url}"; then
                                        echo "== Failed to curl ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    else
                                      if ! wget --inet4-only -O "${file}" --connect-timeout=20 --tries=6 --wait=10 "${url}"; then
                                        echo "== Failed to wget ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    fi
                            ...
                                local actual

                            +   actual=$(sha256sum ${file} | awk '{ print $1 }') || true
                            -   actual=$(sha1sum ${file} | awk '{ print $1 }') || true
                                if [[ "${actual}" != "${expected}" ]]; then
                            +     echo "== ${file} corrupted, hash ${actual} doesn't match expected ${expected} =="
                            -     echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
                                  return 1
                                fi
                            ...

                                local -r nodeup_urls=( $(split-commas "${NODEUP_URL}") )
                            -   local -r nodeup_filename="${nodeup_urls[0]##*/}"
                                if [[ -n "${NODEUP_HASH:-}" ]]; then
                                  local -r nodeup_hash="${NODEUP_HASH}"
                                else
                                # TODO: Remove?
                            +     echo "Downloading sha256 (not found in env)"
                            -     echo "Downloading sha1 (not found in env)"
                            +     download-or-bust nodeup.sha256 "" "${nodeup_urls[@]/%/.sha256}"
                            -     download-or-bust "" "${nodeup_urls[@]/%/.sha1}"
                            +     local -r nodeup_hash=$(cat nodeup.sha256)
                            -     local -r nodeup_hash=$(cat "${nodeup_filename}.sha1")
                                fi

                                echo "Downloading nodeup (${nodeup_urls[@]})"
                            +   download-or-bust nodeup "${nodeup_hash}" "${nodeup_urls[@]}"
                            -   download-or-bust "${nodeup_hash}" "${nodeup_urls[@]}"

                                chmod +x nodeup
                            ...
                              - a006b4680640e5c88742e22b904623a77257f416@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubelet
                              - c3f7fbab5ba39e3ec20b32f0e7bcad6cc0704792@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubectl
                            + - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
                            - - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - 71b7bc444ba0a5f7cd7a36e91b594c1c3d13890e160d85e0dfde38c46a24e416@https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/utils.tar.gz,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-utils.tar.gz,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/utils.tar.gz
                            - - b16b5367e05bad082f416f786c7f8813f7794630@https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/utils.tar.gz
                              ClusterName: prod.kubes.cognius.net
                              ConfigBase: s3://hopjump-kops-state-store/prod.kubes.cognius.net
                            ...
                              - s3://hopjump-kops-state-store/prod.kubes.cognius.net/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 42a9c4324fe26d63ce11f3dd7836371bc93fa06ca8f479807728f3746e27061b
                            -   hash: 725c2de47755544a9aa349e27ed9900d195f0ceb
                            +   name: protokube:1.15.0
                            -   name: protokube:1.11.0
                            +   sources:
                            +   - https://artifacts.k8s.io/binaries/kops/1.15.0/images/protokube.tar.gz
                            +   - https://github.com/kubernetes/kops/releases/download/1.15.0/images-protokube.tar.gz
                            +   - https://kubeupv2.s3.amazonaws.com/kops/1.15.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.11.0/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...

  LaunchConfiguration/pony-nodes.prod.kubes.cognius.net
    UserData            
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/nodeup,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-nodeup,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/nodeup
                            + NODEUP_HASH=9604ef18267ad7b5cf4cebbf7ab64423cf5bb0342d169c608ac6376e6af26d81
                            - NODEUP_HASH=6ee282d77600c47ed7744435400e163fa34ee17e

                              export AWS_REGION=us-east-1
                            ...
                              }

                            - # Retry a download until we get it. Takes a hash and a set of URLs.
                            - #
                            - # $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
                            + # Retry a download until we get it. args: name, sha, url1, url2...
                            - # $2+ are the URLs to download.
                              download-or-bust() {
                            +   local -r file="$1"
                            +   local -r hash="$2"
                            -   local -r hash="$1"
                            +   shift 2
                            -   shift 1

                                urls=( $* )
                                while true; do
                                  for url in "${urls[@]}"; do
                            -       local file="${url##*/}"
                            - 
                                    if [[ -e "${file}" ]]; then
                                      echo "== File exists for ${url} =="
                            ...
                                      if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 "${url}"; then
                                        echo "== Failed to curl ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    else
                                      if ! wget --inet4-only -O "${file}" --connect-timeout=20 --tries=6 --wait=10 "${url}"; then
                                        echo "== Failed to wget ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    fi
                            ...
                                local actual

                            +   actual=$(sha256sum ${file} | awk '{ print $1 }') || true
                            -   actual=$(sha1sum ${file} | awk '{ print $1 }') || true
                                if [[ "${actual}" != "${expected}" ]]; then
                            +     echo "== ${file} corrupted, hash ${actual} doesn't match expected ${expected} =="
                            -     echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
                                  return 1
                                fi
                            ...

                                local -r nodeup_urls=( $(split-commas "${NODEUP_URL}") )
                            -   local -r nodeup_filename="${nodeup_urls[0]##*/}"
                                if [[ -n "${NODEUP_HASH:-}" ]]; then
                                  local -r nodeup_hash="${NODEUP_HASH}"
                                else
                                # TODO: Remove?
                            +     echo "Downloading sha256 (not found in env)"
                            -     echo "Downloading sha1 (not found in env)"
                            +     download-or-bust nodeup.sha256 "" "${nodeup_urls[@]/%/.sha256}"
                            -     download-or-bust "" "${nodeup_urls[@]/%/.sha1}"
                            +     local -r nodeup_hash=$(cat nodeup.sha256)
                            -     local -r nodeup_hash=$(cat "${nodeup_filename}.sha1")
                                fi

                                echo "Downloading nodeup (${nodeup_urls[@]})"
                            +   download-or-bust nodeup "${nodeup_hash}" "${nodeup_urls[@]}"
                            -   download-or-bust "${nodeup_hash}" "${nodeup_urls[@]}"

                                chmod +x nodeup
                            ...
                              - a006b4680640e5c88742e22b904623a77257f416@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubelet
                              - c3f7fbab5ba39e3ec20b32f0e7bcad6cc0704792@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubectl
                            + - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
                            - - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - 71b7bc444ba0a5f7cd7a36e91b594c1c3d13890e160d85e0dfde38c46a24e416@https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/utils.tar.gz,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-utils.tar.gz,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/utils.tar.gz
                            - - b16b5367e05bad082f416f786c7f8813f7794630@https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/utils.tar.gz
                              ClusterName: prod.kubes.cognius.net
                              ConfigBase: s3://hopjump-kops-state-store/prod.kubes.cognius.net
                            ...
                              - s3://hopjump-kops-state-store/prod.kubes.cognius.net/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 42a9c4324fe26d63ce11f3dd7836371bc93fa06ca8f479807728f3746e27061b
                            -   hash: 725c2de47755544a9aa349e27ed9900d195f0ceb
                            +   name: protokube:1.15.0
                            -   name: protokube:1.11.0
                            +   sources:
                            +   - https://artifacts.k8s.io/binaries/kops/1.15.0/images/protokube.tar.gz
                            +   - https://github.com/kubernetes/kops/releases/download/1.15.0/images-protokube.tar.gz
                            +   - https://kubeupv2.s3.amazonaws.com/kops/1.15.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.11.0/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...

  LaunchConfiguration/spot-nodes.prod.kubes.cognius.net
    UserData            
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/nodeup,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-nodeup,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/nodeup
                            + NODEUP_HASH=9604ef18267ad7b5cf4cebbf7ab64423cf5bb0342d169c608ac6376e6af26d81
                            - NODEUP_HASH=6ee282d77600c47ed7744435400e163fa34ee17e

                              export AWS_REGION=us-east-1
                            ...
                              }

                            - # Retry a download until we get it. Takes a hash and a set of URLs.
                            - #
                            - # $1 is the sha1 of the URL. Can be "" if the sha1 is unknown.
                            + # Retry a download until we get it. args: name, sha, url1, url2...
                            - # $2+ are the URLs to download.
                              download-or-bust() {
                            +   local -r file="$1"
                            +   local -r hash="$2"
                            -   local -r hash="$1"
                            +   shift 2
                            -   shift 1

                                urls=( $* )
                                while true; do
                                  for url in "${urls[@]}"; do
                            -       local file="${url##*/}"
                            - 
                                    if [[ -e "${file}" ]]; then
                                      echo "== File exists for ${url} =="
                            ...
                                      if ! curl -f --ipv4 -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 "${url}"; then
                                        echo "== Failed to curl ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    else
                                      if ! wget --inet4-only -O "${file}" --connect-timeout=20 --tries=6 --wait=10 "${url}"; then
                                        echo "== Failed to wget ${url}. Retrying. =="
                            +           continue
                            -           break
                                      fi
                                    fi
                            ...
                                local actual

                            +   actual=$(sha256sum ${file} | awk '{ print $1 }') || true
                            -   actual=$(sha1sum ${file} | awk '{ print $1 }') || true
                                if [[ "${actual}" != "${expected}" ]]; then
                            +     echo "== ${file} corrupted, hash ${actual} doesn't match expected ${expected} =="
                            -     echo "== ${file} corrupted, sha1 ${actual} doesn't match expected ${expected} =="
                                  return 1
                                fi
                            ...

                                local -r nodeup_urls=( $(split-commas "${NODEUP_URL}") )
                            -   local -r nodeup_filename="${nodeup_urls[0]##*/}"
                                if [[ -n "${NODEUP_HASH:-}" ]]; then
                                  local -r nodeup_hash="${NODEUP_HASH}"
                                else
                                # TODO: Remove?
                            +     echo "Downloading sha256 (not found in env)"
                            -     echo "Downloading sha1 (not found in env)"
                            +     download-or-bust nodeup.sha256 "" "${nodeup_urls[@]/%/.sha256}"
                            -     download-or-bust "" "${nodeup_urls[@]/%/.sha1}"
                            +     local -r nodeup_hash=$(cat nodeup.sha256)
                            -     local -r nodeup_hash=$(cat "${nodeup_filename}.sha1")
                                fi

                                echo "Downloading nodeup (${nodeup_urls[@]})"
                            +   download-or-bust nodeup "${nodeup_hash}" "${nodeup_urls[@]}"
                            -   download-or-bust "${nodeup_hash}" "${nodeup_urls[@]}"

                                chmod +x nodeup
                            ...
                              - a006b4680640e5c88742e22b904623a77257f416@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubelet
                              - c3f7fbab5ba39e3ec20b32f0e7bcad6cc0704792@https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubectl
                            + - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
                            - - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - 71b7bc444ba0a5f7cd7a36e91b594c1c3d13890e160d85e0dfde38c46a24e416@https://artifacts.k8s.io/binaries/kops/1.15.0/linux/amd64/utils.tar.gz,https://github.com/kubernetes/kops/releases/download/1.15.0/linux-amd64-utils.tar.gz,https://kubeupv2.s3.amazonaws.com/kops/1.15.0/linux/amd64/utils.tar.gz
                            - - b16b5367e05bad082f416f786c7f8813f7794630@https://kubeupv2.s3.amazonaws.com/kops/1.11.0/linux/amd64/utils.tar.gz
                              ClusterName: prod.kubes.cognius.net
                              ConfigBase: s3://hopjump-kops-state-store/prod.kubes.cognius.net
                            ...
                              - s3://hopjump-kops-state-store/prod.kubes.cognius.net/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 42a9c4324fe26d63ce11f3dd7836371bc93fa06ca8f479807728f3746e27061b
                            -   hash: 725c2de47755544a9aa349e27ed9900d195f0ceb
                            +   name: protokube:1.15.0
                            -   name: protokube:1.11.0
                            +   sources:
                            +   - https://artifacts.k8s.io/binaries/kops/1.15.0/images/protokube.tar.gz
                            +   - https://github.com/kubernetes/kops/releases/download/1.15.0/images-protokube.tar.gz
                            +   - https://kubeupv2.s3.amazonaws.com/kops/1.15.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.11.0/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...

  LoadBalancer/api.prod.kubes.cognius.net
    Tags                     {KubernetesCluster: prod.kubes.cognius.net, Name: api.prod.kubes.cognius.net} -> {Name: api.prod.kubes.cognius.net, KubernetesCluster: prod.kubes.cognius.net, kubernetes.io/cluster/prod.kubes.cognius.net: owned}

  ManagedFile/prod.kubes.cognius.net-addons-bootstrap
    Contents            
                            ...
                                addons:
                                - manifest: core.addons.k8s.io/v1.4.0.yaml
                            +     manifestHash: 3ffe9ac576f9eec72e2bdfbd2ea17d56d9b17b90
                                  name: core.addons.k8s.io
                                  selector:
                            ...
                                  version: 1.4.0
                                - id: k8s-1.6
                            +     kubernetesVersion: '>=1.6.0 <1.12.0'
                            -     kubernetesVersion: '>=1.6.0'
                                  manifest: coredns.addons.k8s.io/k8s-1.6.yaml
                            +     manifestHash: e6ce4dab8fe82e3577d0925b60d825aa629f9ad3
                                  name: coredns.addons.k8s.io
                                  selector:
                                    k8s-addon: coredns.addons.k8s.io
                            +     version: 1.3.1-kops.5
                            +   - id: k8s-1.12
                            +     kubernetesVersion: '>=1.12.0'
                            +     manifest: coredns.addons.k8s.io/k8s-1.12.yaml
                            +     manifestHash: 15936bac001198f0b7f5851f109b1127e93574f3
                            +     name: coredns.addons.k8s.io
                            +     selector:
                            +       k8s-addon: coredns.addons.k8s.io
                            +     version: 1.3.1-kops.5
                            -     version: 1.2.6-kops.1
                                - id: k8s-1.8
                                  kubernetesVersion: '>=1.8.0'
                                  manifest: rbac.addons.k8s.io/k8s-1.8.yaml
                            +     manifestHash: 5d53ce7b920cd1e8d65d2306d80a041420711914
                                  name: rbac.addons.k8s.io
                                  selector:
                                    k8s-addon: rbac.addons.k8s.io
                                  version: 1.8.0
                            +   - id: k8s-1.9
                            +     kubernetesVersion: '>=1.9.0'
                            +     manifest: kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml
                            +     manifestHash: e1508d77cb4e527d7a2939babe36dc350dd83745
                            +     name: kubelet-api.rbac.addons.k8s.io
                            +     selector:
                            +       k8s-addon: kubelet-api.rbac.addons.k8s.io
                            +     version: v0.0.1
                            +   - manifest: limit-range.addons.k8s.io/v1.5.0.yaml
                            +     manifestHash: 2ea50e23f1a5aa41df3724630ac25173738cc90c
                            -   - manifest: limit-range.addons.k8s.io/v1.5.0.yaml
                                  name: limit-range.addons.k8s.io
                                  selector:
                            ...
                                  kubernetesVersion: <1.6.0
                                  manifest: dns-controller.addons.k8s.io/pre-k8s-1.6.yaml
                            +     manifestHash: e0177ad5f8ea6665ff9c4101a69d853e849819c1
                                  name: dns-controller.addons.k8s.io
                                  selector:
                                    k8s-addon: dns-controller.addons.k8s.io
                            +     version: 1.15.0
                            -     version: 1.11.0
                                - id: k8s-1.6
                            +     kubernetesVersion: '>=1.6.0 <1.12.0'
                            -     kubernetesVersion: '>=1.6.0'
                                  manifest: dns-controller.addons.k8s.io/k8s-1.6.yaml
                            +     manifestHash: 89bd49c128aa22699570578d7cdf95126a3512e6
                                  name: dns-controller.addons.k8s.io
                                  selector:
                                    k8s-addon: dns-controller.addons.k8s.io
                            +     version: 1.15.0
                            +   - id: k8s-1.12
                            +     kubernetesVersion: '>=1.12.0'
                            +     manifest: dns-controller.addons.k8s.io/k8s-1.12.yaml
                            +     manifestHash: f5f689416f69cdff546c14ef82cdf63f8b053bc3
                            +     name: dns-controller.addons.k8s.io
                            +     selector:
                            +       k8s-addon: dns-controller.addons.k8s.io
                            +     version: 1.15.0
                            +   - id: v1.15.0
                            +     kubernetesVersion: '>=1.15.0'
                            +     manifest: storage-aws.addons.k8s.io/v1.15.0.yaml
                            +     manifestHash: 23459f7be52d7c818dc060a8bcf5e3565bd87a7b
                            +     name: storage-aws.addons.k8s.io
                            +     selector:
                            +       k8s-addon: storage-aws.addons.k8s.io
                            +     version: 1.15.0
                            -     version: 1.11.0
                                - id: v1.7.0
                            +     kubernetesVersion: '>=1.7.0 <1.15.0'
                            -     kubernetesVersion: '>=1.7.0'
                                  manifest: storage-aws.addons.k8s.io/v1.7.0.yaml
                            +     manifestHash: 62705a596142e6cc283280e8aa973e51536994c5
                                  name: storage-aws.addons.k8s.io
                                  selector:
                                    k8s-addon: storage-aws.addons.k8s.io
                            +     version: 1.15.0
                            -     version: 1.7.0
                                - id: v1.6.0
                                  kubernetesVersion: <1.7.0
                                  manifest: storage-aws.addons.k8s.io/v1.6.0.yaml
                            +     manifestHash: 7de4b2eb0521d669172038759c521418711d8266
                                  name: storage-aws.addons.k8s.io
                                  selector:
                                    k8s-addon: storage-aws.addons.k8s.io
                            +     version: 1.15.0
                            -     version: 1.7.0
                            +   - id: k8s-1.12
                            +     kubernetesVersion: '>=1.12.0'
                            +     manifest: networking.projectcalico.org/k8s-1.12.yaml
                            +     manifestHash: 2ff04777f60a843a15b021e7b94a63dff3645a0b
                            +     name: networking.projectcalico.org
                            +     selector:
                            +       role.kubernetes.io/networking: "1"
                            +     version: 3.9.1-kops.2
                                - id: k8s-1.7-v3
                            +     kubernetesVersion: '>=1.7.0 <1.12.0'
                            -     kubernetesVersion: '>=1.7.0'
                                  manifest: networking.projectcalico.org/k8s-1.7-v3.yaml
                            +     manifestHash: 296f867a05c1d4ee7eb048a822a3f0a12976e7a4
                                  name: networking.projectcalico.org
                                  selector:
                                    role.kubernetes.io/networking: "1"
                            +     version: 3.8.0-kops.1
                            -     version: 3.3.1-kops.3

  ManagedFile/prod.kubes.cognius.net-addons-core.addons.k8s.io
    Contents            
                            ...
                              kind: Namespace
                              metadata:
                            -   name: kube-system
                            +   name: kube-system

  ManagedFile/prod.kubes.cognius.net-addons-coredns.addons.k8s.io-k8s-1.6
    Contents            
                            ...
                              ---

                            + apiVersion: rbac.authorization.k8s.io/v1
                            - apiVersion: rbac.authorization.k8s.io/v1beta1
                              kind: ClusterRole
                              metadata:
                            ...
                              ---

                            + apiVersion: rbac.authorization.k8s.io/v1
                            - apiVersion: rbac.authorization.k8s.io/v1beta1
                              kind: ClusterRoleBinding
                              metadata:
                            ...
                                      }
                                      prometheus :9153
                            +         forward . /etc/resolv.conf
                            -         proxy . /etc/resolv.conf
                                      loop
                                      cache 30
                            ...
                              ---

                            + apiVersion: apps/v1
                            - apiVersion: extensions/v1beta1
                              kind: Deployment
                              metadata:
                            ...
                                      - -conf
                                      - /etc/coredns/Corefile
                            +         image: k8s.gcr.io/coredns:1.3.1
                            -         image: k8s.gcr.io/coredns:1.2.6
                                      imagePullPolicy: IfNotPresent
                                      livenessProbe:
                            ...
                                        name: metrics
                                        protocol: TCP
                            +         readinessProbe:
                            +           httpGet:
                            +             path: /health
                            +             port: 8080
                            +             scheme: HTTP
                                      resources:
                                        limits:
                            ...
                                    nodeSelector:
                                      beta.kubernetes.io/os: linux
                            +       priorityClassName: system-cluster-critical
                                    serviceAccountName: coredns
                                    tolerations:
                            ...
                                  port: 53
                                  protocol: TCP
                            +   - name: metrics
                            +     port: 9153
                            +     protocol: TCP
                                selector:
                            -     k8s-app: kube-dns
                            +     k8s-app: kube-dns

  ManagedFile/prod.kubes.cognius.net-addons-dns-controller.addons.k8s.io-k8s-1.6
    Contents            
                            ...
                                  k8s-addon: dns-controller.addons.k8s.io
                                  k8s-app: dns-controller
                            +     version: v1.15.0
                            -     version: v1.11.0
                                name: dns-controller
                                namespace: kube-system
                            ...
                                      k8s-addon: dns-controller.addons.k8s.io
                                      k8s-app: dns-controller
                            +         version: v1.15.0
                            -         version: v1.11.0
                                  spec:
                                    containers:
                            ...
                                      - --zone=*/*
                                      - -v=2
                            +         image: kope/dns-controller:1.15.0
                            -         image: kope/dns-controller:1.11.0
                                      name: dns-controller
                                      resources:
                            ...
                              - apiGroup: rbac.authorization.k8s.io
                                kind: User
                            -   name: system:serviceaccount:kube-system:dns-controller
                            +   name: system:serviceaccount:kube-system:dns-controller

  ManagedFile/prod.kubes.cognius.net-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
    Contents            
                            ...
                                  k8s-addon: dns-controller.addons.k8s.io
                                  k8s-app: dns-controller
                            +     version: v1.15.0
                            -     version: v1.11.0
                                name: dns-controller
                                namespace: kube-system
                            ...
                                      k8s-addon: dns-controller.addons.k8s.io
                                      k8s-app: dns-controller
                            +         version: v1.15.0
                            -         version: v1.11.0
                                  spec:
                                    containers:
                            ...
                                      - --zone=*/*
                                      - -v=2
                            +         image: kope/dns-controller:1.15.0
                            -         image: kope/dns-controller:1.11.0
                                      name: dns-controller
                                      resources:
                            ...
                                    hostNetwork: true
                                    nodeSelector:
                            -         kubernetes.io/role: master
                            +         kubernetes.io/role: master

  ManagedFile/prod.kubes.cognius.net-addons-limit-range.addons.k8s.io
    Contents            
                            ...
                                - defaultRequest:
                                    cpu: 100m
                            -     type: Container
                            +     type: Container

  ManagedFile/prod.kubes.cognius.net-addons-networking.projectcalico.org-k8s-1.7-v3
    Contents            
                            ...
                                  {
                                    "name": "k8s-pod-network",
                            +       "cniVersion": "0.3.1",
                            -       "cniVersion": "0.3.0",
                                    "plugins": [
                                      {
                            ...
                                      - name: FELIX_HEALTHENABLED
                                        value: "true"
                            +         image: calico/node:v3.8.0
                            -         image: quay.io/calico/node:v3.3.1
                                      livenessProbe:
                                        failureThreshold: 6
                            ...
                                            key: cni_network_config
                                            name: calico-config
                            +         image: calico/cni:v3.8.0
                            -         image: quay.io/calico/cni:v3.3.1
                                      name: install-cni
                                      resources:
                            ...
                                      - name: ENABLED_CONTROLLERS
                                        value: policy,profile,workloadendpoint,node
                            +         image: calico/kube-controllers:v3.8.0
                            -         image: quay.io/calico/kube-controllers:v3.3.1
                                      name: calico-kube-controllers
                                      readinessProbe:
                            ...
                                      env:
                                      - name: EXPECTED_NODE_IMAGE
                            +           value: quay.io/calico/node:v3.7.4
                            -           value: quay.io/calico/node:v3.3.1
                                      - name: CALICO_ETCD_ENDPOINTS
                                        valueFrom:
                            ...
                                    - hostPath:
                                        path: /etc/hosts
                            -         name: etc-hosts
                            +         name: etc-hosts

  ManagedFile/prod.kubes.cognius.net-addons-rbac.addons.k8s.io-k8s-1.8
    Contents            
                            ...
                              - apiGroup: rbac.authorization.k8s.io
                                kind: User
                            -   name: kubelet
                            +   name: kubelet

  ManagedFile/prod.kubes.cognius.net-addons-storage-aws.addons.k8s.io-v1.6.0
    Contents            
                            ...
                              parameters:
                                type: gp2
                            - provisioner: kubernetes.io/aws-ebs
                            + provisioner: kubernetes.io/aws-ebs

  ManagedFile/prod.kubes.cognius.net-addons-storage-aws.addons.k8s.io-v1.7.0
    Contents            
                            ...
                              parameters:
                                type: gp2
                            - provisioner: kubernetes.io/aws-ebs
                            + provisioner: kubernetes.io/aws-ebs

Must specify --yes to apply changes

9. Anything else do we need to know? No

[pradeepnnv]

@cnw004 , we've observed this behavior when kops itself is upgraded. Even without any changes to Kube version, it'll try to upgrade various internal components like protokube etc.

[cnw004]

@pradeepnnv thank you so much, this is exactly what the issue was. Do you have any experience doing this? Is updating the kops version dangerous to the cluster itself? It seems to be wanting to make a ton of changes and that makes me nervous. I know that the matrix here says that it shouldn't be a problem

[pradeepnnv]

@cnw004 , we didn't run into any issues when did similar updates. We had kops at a higher version than Kube itself. But i think it's best not to use a very latest kops for managing an older version of kube cluster. Best to upgrade both of them in tandem.

[cnw004]

@pradeepnnv sounds good, thanks for the tips. I really appreciate you helping me out with this, I was totally lost. Closing out the issue because this is resolved on my end!