Deploy Kubernetes 1.24.10 with kOps 1.25.3 without CCM

[ialejandro]

Hi,

I open this issue, but I'm sure it's my fault (not a bug). I've a problem when I upgrade from kOps 1.24.3 to 1.25.3 and upgrade my cluster from 1.23.13 to 1.24.10.

Currently:

• Kubernetes version: 1.24.10
• kOps version: 1.25.3
• Provider: AWS
• Image base: amzn2-ami-kernel-5.10-hvm-2.0.20230119.1-x86_64-gp2

So, @olemarkus in this issue (#14035) you say:

IP based naming without using CCM was resolved with https://github.com/kubernetes/kops/pull/14024

However, when you upgrade to use CCM on a supported k8s version, resource based naming will indeed be forced. Your arguments seem based around a cluster running in multiple regions, but I don't see how that is possible. Are you saying you have such a setup?

In another issue (#13934) you say:

For kOps, the more flexible resource-based naming alternative will be used by used if you are using kops 1.24 and kubernetes version is 1.24. You can opt-in with earlier k8s versions by enabling CCM:
spec:
  cloudControllerManager: {}

So, If don't put cloudControllerManager: {} in spec section I can't upgrade. This param is required.

More info, I checked the hostname on my nodes:

[root@ip-10-3-4-112 /]# hostname
ip-10-3-4-112.eu-west-1.compute.internal

And node info (from Lens):

Addresses
InternalIP: 10.3.4.112
InternalDNS: ip-10-3-4-112.eu-west-1.compute.internal
Hostname: ip-10-3-4-112.eu-west-1.compute.internal

From my AWS console, I review my subnet config and IP name is selected. (https://cloud-provider-aws.sigs.k8s.io/prerequisites/)

But, is possible upgrade to 1.24.10 (k8s) and 1.25.3 (kOps) without CCM? Because I'll try some configs and I can't evict CCM deployed. So, any doc about that? I checked kOps CRD kops.k8s.io_clusters.yaml#L546-L651 about cloudControllerManager but nothing. I would like keep the old name (aws ip private dns name) and don't use the AWS ID nomenclature.

My applied kOps cluster spec config:

# kOps Cluster definition
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null
  ## Cluster name
  name: cluster.k8s.XXXXXXXXXXXX
spec:
  api:
    loadBalancer:
      type: Public
      class: Network
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  ## Bucket name and state file
  configBase: s3://XXXXXXXXXXXX
  dnsZone: k8s.XXXXXXXXXXXX
  etcdClusters:
  - name: main
    cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-1
      name: etcd-1
    - instanceGroup: master-2
      name: etcd-2
    - instanceGroup: master-3
      name: etcd-3
    memoryRequest: 100Mi
  - name: events
    cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-1
      name: etcd-1
    - instanceGroup: master-2
      name: etcd-2
    - instanceGroup: master-3
      name: etcd-3
    memoryRequest: 100Mi
  iam:
    legacy: false
  ## Kubelet config
  kubelet:
    ### Enable to use metrics-server
    anonymousAuth: false
    ### Allow serviceaccount tokens to communicate with kubelet
    authorizationMode: Webhook
    authenticationTokenWebhook: true
  ## VPC CIDR access to Kubernetes API Server
  kubernetesApiAccess:
  - XXXXXXXXXXXX
  kubernetesVersion: 1.24.10
  masterPublicName: api.cluster.k8s.XXXXXXXXXXXX
  ## VPC CIDR
  networkCIDR: 10.3.0.0/16
  # VPC additional CIDR
  additionalNetworkCIDRs:
  - 100.112.176.0/25
  ## VPC ID
  networkID: XXXXXXXXXXXX
  networking:
    ## CNI
    calico: {}
  ## Internal kubernetes CIDR
  nonMasqueradeCIDR: 110.64.0.0/10
  sshAccess: null
  subnets:
  - cidr: 10.3.8.0/24
    id: XXXXXXXXXXXX
    name: utility-eu-west-1a
    type: Utility
    zone: eu-west-1a
  - cidr: 10.3.9.0/24
    id: XXXXXXXXXXXX
    name: utility-eu-west-1b
    type: Utility
    zone: eu-west-1b
  - cidr: 10.3.10.0/24
    id: XXXXXXXXXXXX
    name: utility-eu-west-1c
    type: Utility
    zone: eu-west-1c
  - cidr: 10.3.2.0/23
    id: XXXXXXXXXXXX
    name: eu-west-1a
    type: Private
    zone: eu-west-1a
  - cidr: 10.3.4.0/23
    id: XXXXXXXXXXXX
    name: eu-west-1b
    type: Private
    zone: eu-west-1b
  - cidr: 10.3.6.0/23
    id: XXXXXXXXXXXX
    name: eu-west-1c
    type: Private
    zone: eu-west-1c
  topology:
    dns:
      type: Private
    masters: private
    nodes: private
  ## Addons
  awsLoadBalancerController:
    enabled: false
  nodeTerminationHandler:
    enabled: true
  clusterAutoscaler:
    enabled: true
    awsUseStaticInstanceList: false
    balanceSimilarNodeGroups: false
    expander: random
    image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0
    maxNodeProvisionTime: 15m0s
    newPodScaleUpDelay: 0s
    scaleDownDelayAfterAdd: 10m0s
    scaleDownUtilizationThreshold: "0.5"
    skipNodesWithLocalStorage: true
    skipNodesWithSystemPods: true
  cloudControllerManager: {}
  cloudConfig:
    awsEBSCSIDriver:
      enabled: true
      managed: false
    manageStorageClasses: false
  nodeProblemDetector:
    enabled: true

I only added this new lines:

  kubernetesVersion: 1.24.10
  cloudControllerManager: {}
  cloudConfig:
    awsEBSCSIDriver:
      enabled: true    # force CCM (false = can't deploy)
      managed: false
    manageStorageClasses: false
  clusterAutoscaler:
    ...
    image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.24.0

[olemarkus]

A bit tricky to determine exactly what you are asking here, but:

• When you run K8s 1.24 and above, CCM is mandatory.
• When you use CCM, resource-based naming is mandatory.

[ialejandro]

A bit tricky to determine exactly what you are asking here, but:

* When you run K8s 1.24 and above, CCM is mandatory.

* When you use CCM, resource-based naming is mandatory.

That's it. That's what I needed to know. Regards! :)