search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.99k stars 4.65k forks source link

CLB to NLB migration not working as expected with gossip clusters. #11736

Closed dskatz closed 3 years ago

dskatz commented 3 years ago

/kind bug

1. What kops version are you running? The command kops version, will display this information.

Version 1.20.0 (git-8ea83c6d233a15dacfcc769d4d82bea3f530cf72)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

v1.18.18

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

The commands were executed as described in the ACM-NLB documentation to prep a kops cluster for k8s 1.19 removal of basic auth credentials. (https://github.com/kubernetes/kops/blob/master/permalinks/acm_nlb.md)

For brevity, after the cluster spec was updated with spec.api.loadBalancer.class: Network..

  1. kops update cluster --yes
  2. Since running a gossip cluster, attempted to delete the master secret kops delete secret master. This does not exist, after failing to find documentation on kops secret, I deleted the master keypair with a kops delete keypair master
  3. kops rolling-update cluster --instance-group-roles Master --cloudonly

5. What happened after the commands executed?

The control plane came back, however since the master keypair was deleted which essentially destroys the certificate authority, all service-account tokens need to be re-created. Deleting all the tokens and rolling pods recovered the cluster.

6. What did you expect to happen?

I did not expect that I would need to delete the CA in order to get kops to re-issue the K8s API Sever certificates to simply update the SAN with the new NLB domain name.

I expected this to be somewhat transparent to a kops administrator and simply just require a normal rolling-update of the control-plane.

I also inspected the K8s API Server certificate and discovered all SANs included which as follows:

  1. kubernetes.default
  2. kubernetes.default.svc
  3. kubernetes.default.svc.cluster.local
  4. api.sandbox..com ( this is our external DNS CNAME to the K8s API Server)
  5. api.internal.sandbox.k8s.local (the internal DNS name that dns-control uses for gossip)
  6. .elb.us-east-1.amazonaws.com - ELB address
  7. IP: 100.x.x.1
  8. IP 127.0.0.1

As far as I can tell, in our cluster spec nothing ever references the api server by the ELB name. Kops kubecfg uses the DNS CNAME.

What is the purpose of including the ELB name as a SAN?

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2021-06-01T18:43:53Z"
  name: sandbox.k8s.local
spec:
  api:
    loadBalancer:
      class: Network
      type: Public
      sslCertificate: <redacted>
  authorization:
    rbac: {}
  channel: stable
  cloudLabels:
    owner: katz
  containerRuntime: docker
  docker:
    version: 19.03.11
  cloudProvider: aws
  configBase: s3://sandbox/sandbox.k8s.local
  containerRuntime: docker
  etcdClusters:
    - cpuRequest: 200m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-us-east-1a
          name: a
        - encryptedVolume: true
          instanceGroup: master-us-east-1b
          name: b
        - encryptedVolume: true
          instanceGroup: master-us-east-1c
          name: c
      memoryRequest: 100Mi
      name: main
    - cpuRequest: 100m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-us-east-1a
          name: a
        - encryptedVolume: true
          instanceGroup: master-us-east-1b
          name: b
        - encryptedVolume: true
          instanceGroup: master-us-east-1c
          name: c
      memoryRequest: 100Mi
      name: events
  gossipConfig:
    protocol: memberlist
    listen: "0.0.0.0:4000"
    secondary:
      protocol: ""
  dnsControllerGossipConfig:
    protocol: memberlist
    listen: "0.0.0.0:3993"
    seed: "127.0.0.1:4000"
    secondary:
      protocol: ""
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubeAPIServer:
    disableBasicAuth: false
    enableAdmissionPlugins:
    - NamespaceLifecycle
    - LimitRanger
    - ServiceAccount
    - PersistentVolumeLabel
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - MutatingAdmissionWebhook
    - ValidatingAdmissionWebhook
    - NodeRestriction
    - ResourceQuota
    - PodNodeSelector
    - PodTolerationRestriction
    - PersistentVolumeClaimResize
    - PodSecurityPolicy
  kubernetesApiAccess:
    - <redacted>
  kubernetesVersion: 1.18.18
  masterInternalName: api.internal.sandbox.k8s.local
  masterPublicName: api.sandbox.<redacted>.com
  networkCIDR: 172.41.0.0/16
  networking:
    canal: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
    - <redacted>
  subnets:
    - cidr: 172.41.32.0/19
      name: us-east-1a
      type: Private
      zone: us-east-1a
    - cidr: 172.41.64.0/19
      name: us-east-1b
      type: Private
      zone: us-east-1b
    - cidr: 172.41.96.0/19
      name: us-east-1c
      type: Private
      zone: us-east-1c
    - cidr: 172.41.0.0/22
      name: utility-us-east-1a
      type: Utility
      zone: us-east-1a
    - cidr: 172.41.4.0/22
      name: utility-us-east-1b
      type: Utility
      zone: us-east-1b
    - cidr: 172.41.8.0/22
      name: utility-us-east-1c
      type: Utility
      zone: us-east-1c
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

N/A - logs can be provided if the discussion warrants it. There were no command failures.

9. Anything else do we need to know?

The purpose of this issue is to seek clarification on how to migrate to NLB with gossip clusters that have the least amount of impact to cluster operations.

Based on the above information I re-ran migration steps using a modified approach that more closely aligned with what I expected to happen given my assumption that kops does not actually connect to the ELB name, actually rotating the K8s API server certificate would not be necessary.

  1. Updated the cluster spec with spec.api.loadBalancer.class: Network..
  2. kops update cluster --yes
  3. This caused about 15 minutes of downtime as expected while the new NLB provisioned with a DNS change.
  4. kops export kubecfg --admin
  5. At this point, I have full access to the control-plane running via the nlb
  6. Validated that kubecontext is using the DNS CNAME using mTLS on port 8443.
  7. Validated via ssh that the K8s API Server cert still references the OLD ELB as a SAN.

On a hunch, I ran kops rolling-update cluster and discovered that the control-plane instance groups were all tagged as NeedsUpdate. I decided to roll them and after inspected the K8s API certificate and the SAN was replaced with the NLB amazon DNS name.

Questions

  1. is this procedure documented? It seems to contradict portions of the upstream docs.
  2. Are there any issues I am not aware of by doing this migration in this way?

Based on the feedback this issue gets, I am willing to contribute updated documentation.

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 3 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/kops/issues/11736#issuecomment-962692739): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.