search

kubernetes / cloud-provider-aws

Cloud provider for AWS
https://cloud-provider-aws.sigs.k8s.io/
Apache License 2.0
397 stars 305 forks source link

aws cloud controller manager is unable to manage the nodes in cluster #916

Closed karty-s closed 1 month ago

karty-s commented 6 months ago

What happened: We are running k8s cluster of version 1.26 using kubeadm with resources from aws. We wanted to upgrade our clusters to 1.28 (1.26->1.27->1.28) as per update notes we tried to move from in-tree aws cloud provider to external aws cloud provider. As per the upgrade process we deployed the new 1.27 nodes along with aws cloud controller manager in the cluster, post which we scaled down the 1.26 nodes.

What you expected to happen: The issue we face is that the etcd and worker nodes of 1.26 version which is scaled down gets removed from the cluster, but the control plane nodes still shows up in the cluster even after its ec2 instance is removed. eg -

NAME                            STATUS                     ROLES                  AGE     VERSION
ip-.ec2.internal   Ready,SchedulingDisabled   control-plane,master   96m     v1.26.7
ip-.ec2.internal   Ready                      etcd                   11m     v1.27.13
ip-.ec2.internal   Ready                      etcd                   9m10s   v1.27.13
ip-.ec2.internal   Ready                      control-plane,master   5m59s   v1.27.13
ip-.ec2.internal   Ready,SchedulingDisabled   control-plane,master   95m     v1.26.7
ip-.ec2.internal    Ready                      node                   6m12s   v1.27.13
ip-.ec2.internal    Ready                      etcd                   14m     v1.27.13
ip-.ec2.internal    Ready                      control-plane,master   6m1s    v1.27.13
ip-.ec2.internal    Ready                      node                   6m9s    v1.27.13
ip-.ec2.internal    Ready                      node                   6m14s   v1.27.13
ip-.ec2.internal    Ready                      node                   6m15s   v1.27.13
ip-.ec2.internal    Ready,SchedulingDisabled   control-plane,master   96m     v1.26.7
ip-.ec2.internal    Ready                      node                   6m15s   v1.27.13
ip-.ec2.internal    Ready                      node                   6m15s   v1.27.13
ip-.ec2.internal    Ready                      control-plane,master   5m43s   v1.27.13

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: we are seeing this error in the cloud controller manager pod logs -

I0516 08:13:24.811572       1 node_lifecycle_controller.go:164] deleting node since it is no longer present in cloud provider: ip-10-230-13-35.ec2.internal
I0516 08:13:24.812083       1 event.go:307] "Event occurred" object="ip-10-230-13-35.ec2.internal" fieldPath="" kind="Node" apiVersion="" type="Normal" reason="DeletingNode" message="Deleting node ip-10-230-13-35.ec2.internal because it does not exist in the cloud provider"

we have set the hostname according to the pre req but still we get this

Environment: kubeadm

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:23:27Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7

/kind bug

k8s-ci-robot commented 6 months ago

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
cartermckinnon commented 6 months ago

This:

I0516 08:13:24.811572 1 node_lifecycle_controller.go:164] deleting node since it is no longer present in cloud provider: ip-10-230-13-35.ec2.internal

Isn't an error, it's expected behavior when a Node becomes NotReady and the corresponding EC2 instance is terminated (or doesn't exist). Are you sure the EC2 instances for your old 1.26 control plane nodes have been terminated? They wouldn't have a Ready status if the kubelet stopped heartbeating.

shnigam2 commented 6 months ago

@cartermckinnon We have followed below steps on existing 1.26 cluster to make it ready for 1.27 upgrade

On existing version 1.26
Add tag to each node [kubernetes.io/cluster/cluster-name: owned
k edit cm kubeadm-config -n kube-system to update cloud-provider=external
Update existing master kube-controller and kube-apiserver manifest to use cloud-provider=external
Made aws-controller-manager running

Now when upgrading cluster to 1.27, below are the issues which we are facing:-

cartermckinnon commented 6 months ago

Are you passing --cloud-provider=external to kubelet as well?

CCM should fill in the provider ID if it's missing, but it's generally preferable to just pass it to kubelet to avoid extra API calls in CCM. The EKS AMI uses this helper script to set it: https://github.com/awslabs/amazon-eks-ami/blob/f5111dd100ebd94d9fbfbb1fe2f43b75fd1a6703/templates/al2/runtime/bin/provider-id

shnigam2 commented 6 months ago

@cartermckinnon Let me share you 10-kubeadm-conf and kubeadm-config which we currently have in 1.26 where in tree support is there :-

10-kubeam-conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=aws --node-labels=node.kubernetes.io/role=${kind},instance-group=${group_name},${extra_labels} --register-with-taints=${taints} --cert-dir=/etc/kubernetes/pki --cgroup-driver=systemd"
# Environment="KUBELET_KUBEADM_ARGS=--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true  --rotate-certificates"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
kubeadm-config
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  certSANs:
  - "api-int.${cluster_fqdn}"
  - "api.${cluster_fqdn}"
  extraArgs:
    anonymous-auth: "true"
    audit-log-maxage: "7"
    audit-log-maxbackup: "50"
    audit-log-maxsize: "100"
    audit-log-path: /var/log/kube-apiserver-audit.log
    audit-policy-file: /etc/kubernetes/files/audit-log-policy.yaml
    authorization-mode: Node,RBAC
    cloud-provider: aws
    max-mutating-requests-inflight: "400"
    max-requests-inflight: "800"
    oidc-client-id: "${dex_oidc_client_id}"
    oidc-groups-claim: "${dex_oidc_groups_claim}"
    oidc-issuer-url: "${dex_oidc_issuer_url}"
    oidc-username-claim: "${dex_oidc_username_claim}"
    profiling: "false"
    request-timeout: 30m0s
    service-account-lookup: "true"
    tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
  extraVolumes:
  - hostPath: /etc/kubernetes/files
    mountPath: /etc/kubernetes/files
    name: cloud-config
    readOnly: true
  - hostPath: /var/log
    mountPath: /var/log
    name: var-log
    readOnly: false
  timeoutForControlPlane: 10m0s
certificatesDir: /etc/kubernetes/pki
clusterName: "${cluster_fqdn}"
controlPlaneEndpoint: "${api_endpoint}:${api_port}"
controllerManager:
  extraArgs:
    cluster-signing-cert-file: /etc/kubernetes/pki/ca.crt
    cluster-signing-key-file: /etc/kubernetes/pki/ca.key
    feature-gates: RotateKubeletServerCertificate=true
    profiling: "false"
    terminated-pod-gc-threshold: "12500"
    configure-cloud-routes: "false"
    cluster-name: "${cluster_fqdn}"
    attach-detach-reconcile-sync-period: "1m0s"
    cloud-provider: "aws"
{{- if contains "1.15" .Kubernetes.Version | not }}
    flex-volume-plugin-dir: "/var/lib/kubelet/volumeplugins/"
{{- end }}
dns:
  type: CoreDNS
etcd:
  ${etcd_type}:
    endpoints:
    ${endpoints}
    caFile: ${etcd_cafile}
    certFile: "/etc/kubernetes/pki/apiserver-etcd-client.crt"
    keyFile: "/etc/kubernetes/pki/apiserver-etcd-client.key"
imageRepository: registry.k8s.io
kubernetesVersion: "${k8s_version}"
networking:
  dnsDomain: cluster.local
  podSubnet: "${pod_subnet}"
  serviceSubnet: "100.64.0.0/13"

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
- token: "${kubeadm_token}"
  description: "kubeadm bootstrap token"
  ttl: "43800h"
nodeRegistration:
  criSocket: "unix:///var/run/containerd/containerd.sock"
  kubeletExtraArgs:
    container-runtime: remote
    container-runtime-endpoint: unix:///run/containerd/containerd.sock
  ignorePreflightErrors:
  - IsPrivilegedUser
localAPIEndpoint:
  bindPort: 443
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

Now we are planning to opt out off tree aws cloud controller manager, Could you please guide us what changes we need to make to migrate from in-tree to out-tree . Currently we have deployed aws-cloud-controllermanager daemonset and those are running. But kube-controller-manager also running with above configurations.

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 month ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/cloud-provider-aws/issues/916#issuecomment-2410303201): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.