search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.89k stars 4.65k forks source link

AWS LaunchTemplates with no userdata rebuild on each kops update cluster #13320

Closed kuldazbraslav closed 2 years ago

kuldazbraslav commented 2 years ago

/kind bug

1. What kops version are you running? The command kops version, will display this information.

Version 1.21.4 (git-53e6bf3e5b0a77c78df2a7c60baca6e926fd5105)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

v1.19.16

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

This spec is in S3:

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2022-02-21T15:24:49Z"
  labels:
    kops.k8s.io/cluster: <REDACTED>
  name: bastions
spec:
  associatePublicIp: false
  cloudLabels:
    Owner: Core
    Service: Bastion
    env: dev
  image: <REDACTED>
  machineType: t3.micro
  maxSize: 2
  minSize: 1
  nodeLabels:
    ami/id: <REDACTED>
    ami/name: <REDACTED>
    ami/version: <REDACTED>
    environment: development
  role: Bastion
  rootVolumeOptimization: false
  rootVolumeSize: 10
  rootVolumeType: gp3
  subnets:
  - eu-west-1a
  - eu-west-1b
  - eu-west-1c

Then I run kops --state s3://<REDACTED> update cluster <REDACTED>

5. What happened after the commands executed?

Bastion instance group is reported with changes, even though in both cases there are no userdata either before or after.

Will modify resources:
  LaunchTemplate/bastions.<REDACTED>
    UserData                 <nil> -> <resource>

If I run kops update cluster --yes, new launch template is generated and bastion IG requires rolling update.

6. What did you expect to happen?

No changes reported, no rolling update required.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null
  generation: 26
  name: <REDACTED>
spec:
  api:
    loadBalancer:
      class: Classic
      idleTimeoutSeconds: 300
      type: Public
  authorization:
    rbac: {}
  awsLoadBalancerController:
    enabled: true
  certManager:
    enabled: true
  channel: stable
  cloudConfig:
    awsEBSCSIDriver:
      enabled: true
    disableSecurityGroupIngress: false
  cloudProvider: aws
  clusterAutoscaler:
    balanceSimilarNodeGroups: true
    cpuRequest: 100m
    enabled: true
    expander: most-pods
    memoryRequest: 600Mi
    scaleDownDelayAfterAdd: 10m0s
    scaleDownUtilizationThreshold: "0.6"
    skipNodesWithLocalStorage: false
    skipNodesWithSystemPods: true
  clusterDNSDomain: cluster.local
  configBase: <REDACTED>
  configStore: <REDACTED>
  dnsZone: <REDACTED>
  etcdClusters:
  - backups:
      backupStore: <REDACTED>
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-eu-west-1a
      name: eu-west-1a
    - encryptedVolume: true
      instanceGroup: master-eu-west-1b
      name: eu-west-1b
    - encryptedVolume: true
      instanceGroup: master-eu-west-1c
      name: eu-west-1c
    name: main
    provider: Manager
  - backups:
      backupStore: <REDACTED>
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-eu-west-1a
      name: eu-west-1a
    - encryptedVolume: true
      instanceGroup: master-eu-west-1b
      name: eu-west-1b
    - encryptedVolume: true
      instanceGroup: master-eu-west-1c
      name: eu-west-1c
    name: events
    provider: Manager
  externalPolicies:
    bastion:
    - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    master:
    - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    node:
    - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
  iam:
    allowContainerRegistry: true
    legacy: false
  keyStore: <REDACTED>
  kubeAPIServer:
    anonymousAuth: false
    apiServerCount: 3
    authorizationMode: Node,RBAC
    cloudProvider: aws
    enableAdmissionPlugins:
    - DefaultStorageClass
    - DefaultTolerationSeconds
    - DenyEscalatingExec
    - LimitRanger
    - MutatingAdmissionWebhook
    - NamespaceLifecycle
    - NodeRestriction
    - PersistentVolumeClaimResize
    - PersistentVolumeLabel
    - PodPreset
    - PodSecurityPolicy
    - Priority
    - ResourceQuota
    - ServiceAccount
    - ValidatingAdmissionWebhook
    insecureBindAddress: 127.0.0.1
    insecurePort: 8080
    kubeletPreferredAddressTypes:
    - InternalIP
    - Hostname
    - ExternalIP
    oidcClientID: <REDACTED>
    oidcGroupsClaim: groups
    oidcIssuerURL: https://accounts.google.com
    oidcUsernameClaim: email
    runtimeConfig:
      settings.k8s.io/v1alpha1: "true"
    securePort: 443
    serviceClusterIPRange: 100.64.0.0/13
  kubeControllerManager:
    allocateNodeCIDRs: true
    attachDetachReconcileSyncPeriod: 1m0s
    cloudProvider: aws
    clusterCIDR: 100.96.0.0/11
    clusterName: <REDACTED>
    configureCloudRoutes: false
    horizontalPodAutoscalerDownscaleStabilization: 5m0s
    horizontalPodAutoscalerSyncPeriod: 30s
    kubeAPIBurst: 150
    kubeAPIQPS: "100"
    leaderElection:
      leaderElect: true
    useServiceAccountCredentials: true
  kubeDNS:
    domain: cluster.local
    externalCoreFile: <REDACTED>
    nodeLocalDNS:
      cpuRequest: 25m
      enabled: true
      memoryRequest: 5Mi
    provider: CoreDNS
    serverIP: 100.64.0.10
  kubeProxy:
    clusterCIDR: 100.96.0.0/11
    cpuRequest: 100m
    enabled: true
    hostnameOverride: '@aws'
  kubeScheduler:
    leaderElection:
      leaderElect: true
  kubelet:
    allowedUnsafeSysctls:
    - net.*
    anonymousAuth: true
    cgroupRoot: /
    cloudProvider: aws
    clusterDNS: 100.64.0.10
    clusterDomain: cluster.local
    cpuCFSQuota: true
    cpuCFSQuotaPeriod: 5ms
    enableDebuggingHandlers: true
    enforceNodeAllocatable: pods
    evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    hostnameOverride: '@aws'
    imagePullProgressDeadline: 1m0s
    kubeReserved:
      cpu: 50m
      memory: 256Mi
    kubeconfigPath: /var/lib/kubelet/kubeconfig
    logLevel: 0
    networkPluginName: cni
    nonMasqueradeCIDR: 100.64.0.0/10
    podManifestPath: /etc/kubernetes/manifests
    serializeImagePulls: true
    systemReserved:
      cpu: 50m
      memory: 256Mi
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.19.16
  masterInternalName: <REDACTED>
  masterKubelet:
    cgroupRoot: /
    cloudProvider: aws
    clusterDNS: 100.64.0.10
    clusterDomain: cluster.local
    enableDebuggingHandlers: true
    evictionHard: memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    hostnameOverride: '@aws'
    kubeconfigPath: /var/lib/kubelet/kubeconfig
    logLevel: 0
    networkPluginName: cni
    nonMasqueradeCIDR: 100.64.0.0/10
    podManifestPath: /etc/kubernetes/manifests
    registerSchedulable: false
  masterPublicName: <REDACTED>
  metricsServer:
    enabled: true
    insecure: true
  networkCIDR: <REDACTED>
  networkID: <REDACTED>
  networking:
    calico:
      bpfExternalServiceMode: Tunnel
      bpfLogLevel: "Off"
      crossSubnet: true
      encapsulationMode: ipip
      mtu: 8981
      prometheusGoMetricsEnabled: true
      prometheusMetricsEnabled: true
      prometheusMetricsPort: 9091
      prometheusProcessMetricsEnabled: true
      typhaPrometheusMetricsEnabled: true
      typhaPrometheusMetricsPort: 9093
      typhaReplicas: 3
  nonMasqueradeCIDR: 100.64.0.0/10
  secretStore: <REDACTED>
  serviceClusterIPRange: 100.64.0.0/13
  sshAccess: <REDACTED>
  subnets:
  - cidr: <REDACTED>
    egress: <REDACTED>
    id: <REDACTED>
    name: eu-west-1a
    type: Private
    zone: eu-west-1a
  - cidr: <REDACTED>
    id: <REDACTED>
    name: <REDACTED>
    type: Utility
    zone: eu-west-1a
  - cidr: <REDACTED>
    egress: <REDACTED>
    id: <REDACTED>
    name: eu-west-1b
    type: Private
    zone: eu-west-1b
  - cidr: <REDACTED>
    id: <REDACTED>
    name: utility-eu-west-1b
    type: Utility
    zone: eu-west-1b
  - cidr: <REDACTED>
    egress: <REDACTED>
    id: <REDACTED>
    name: eu-west-1c
    type: Private
    zone: eu-west-1c
  - cidr: <REDACTED>
    id: <REDACTED>
    name: utility-eu-west-1c
    type: Utility
    zone: eu-west-1c
  topology:
    bastion:
      bastionPublicName: <REDACTED>
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2022-02-21T15:24:49Z"
  labels:
    kops.k8s.io/cluster: <REDACTED>
  name: bastions
spec:
  associatePublicIp: false
  cloudLabels:
    Owner: Core
    Service: Bastion
    env: dev
  image: <REDACTED>
  machineType: t3.micro
  maxSize: 2
  minSize: 1
  nodeLabels:
    ami/id: <REDACTED>
    ami/name: <REDACTED>
    ami/version: <REDACTED>
    environment: development
  role: Bastion
  rootVolumeOptimization: false
  rootVolumeSize: 10
  rootVolumeType: gp3
  subnets:
  - eu-west-1a
  - eu-west-1b
  - eu-west-1c

---

<OMITTING OTHER INSTANCE GROUPS>

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

Sorry, I'm not able to redact sensitive values from that large amount of logs.

9. Anything else do we need to know?

I had look at the code briefly and found out the following:

kuldazbraslav commented 2 years ago

This likely affects only bastions as other node roles contain some userdata.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

olemarkus commented 2 years ago

I wonder if this comes from missing this:

nodeLabels:
    kops.k8s.io/instancegroup: bastions

Without this, kOps is not able to map instances to the IG.

So the bug here is that it's allowed to have an IG without that node label, I think.

Can you try adding the above?

/remove-lifecycle stale

kuldazbraslav commented 2 years ago

@olemarkus I tried adding the label you suggested and nothing changed. According to the code it seems that the label is added automatically - not to the spec, but to the launch template.

However, I come up with a new observation:

olemarkus commented 2 years ago

I tried to reproduce this and the issue only surfaced once I removed those labels. I'll try some more later.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

seh commented 2 years ago

This came up in the "kops-users" channel of the "Kubernetes" Slack team.