kubernetes / cloud-provider-aws

Cloud provider for AWS
https://cloud-provider-aws.sigs.k8s.io/
Apache License 2.0
387 stars 301 forks source link

Deprecate and remove AWS cloud-controller-manager Helm Chart #933

Open dims opened 4 months ago

dims commented 4 months ago

We have a helm chart which currently defaults to support for k8s 1.27. We do not have a CI job that tests this chart though.

Looking through usages in the wild: https://github.com/search?type=code&q=%2Fhelm+repo+add+aws-cloud-controller-manager%2F&p=1

kops et al do use manifests directly: https://github.com/search?q=repo%3Akubernetes%2Fkops+aws-cloud-controller-manager&type=code&p=4

Given community k8s 1.27 is EOL in June 2024 we should just drop the chart entirely by the time k8s 1.31 is released.

k8s-ci-robot commented 4 months ago

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
dims commented 4 months ago

cc @cartermckinnon @kmala

kmala commented 4 months ago

/triage accepted

k8s-ci-robot commented 4 months ago

@kmala: The label triage/accepted cannot be applied. Only GitHub organization members can add the label.

In response to [this](https://github.com/kubernetes/cloud-provider-aws/issues/933#issuecomment-2142638250): >/triage accepted Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
cartermckinnon commented 4 months ago

I think it's worth keeping around, it has a decent number of users (including Rancher: https://github.com/rancher/rke1-docs). Can we switch the values.yaml to use a latest tag/something low maintenance instead?

dims commented 4 months ago

@cartermckinnon we don't have a CI job to verify it does work .. so can't guarantee the functionality

atsai1220 commented 4 months ago

What would be the recommended way to install this cloud-provider?

dims commented 4 months ago

What would be the recommended way to install this cloud-provider?

https://github.com/kubernetes/cloud-provider-aws/blob/master/docs/getting_started.md#upgrading-an-existing-cluster

atsai1220 commented 4 months ago

It sounds like you’re saying to use kustomize since the Helm chart will be deprecated. Is that correct?On Jun 4, 2024, at 1:37 PM, Davanum Srinivas @.***> wrote:

What would be the recommended way to install this cloud-provider?

https://github.com/kubernetes/cloud-provider-aws/blob/master/docs/getting_started.md#upgrading-an-existing-cluster

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

et304383 commented 3 months ago

What would be the recommended way to install this cloud-provider?

https://github.com/kubernetes/cloud-provider-aws/blob/master/docs/getting_started.md#upgrading-an-existing-cluster

We are expected to apply a static example bunch of yaml files? They're still referencing version 1.27.1. What is the correct way to install the latest? Use this directory and do a bunch of sed commands for the latest?

Why not support the helm chart as the installation mechanism like the EBS CSI driver is doing? https://github.com/kubernetes-sigs/aws-ebs-csi-driver/releases

I hate to be blunt, but it's bad enough that the documentation for this CRITICAL project is severely lacking, but now a stable installation mechanism isn't even provided?

I REALLY wish AWS would chime in and take over this project so it has proper support instead of just expecting everyone to use EKS.

dims commented 3 months ago

@et304383 can you please share where / how exactly you are using this chart?

et304383 commented 3 months ago

@et304383 can you please share where / how exactly you are using this chart?

We are managing our own K8s control plane for over 40 clusters, and obviously have to install this controller to support 1.27+

So far, this is how script to install:

helm repo add aws-cloud-controller-manager https://kubernetes.github.io/cloud-provider-aws
helm repo update
helm pull aws-cloud-controller-manager/aws-cloud-controller-manager --version 0.0.8 --untar

cd aws-cloud-controller-manager
cluster_cidr=$(grep cluster-cidr /etc/kubernetes/manifests/kube-controller-manager.yaml | cut -d = -f2)
cluster_name=$(grep cluster-name /etc/kubernetes/manifests/kube-controller-manager.yaml | cut -d = -f2)
yq -i '.args += ["--allocate-node-cidrs=false", "--configure-cloud-routes=false", "--cluster-cidr='$cluster_cidr'", "--cluster-name='$cluster_name'"]' values.yaml

export KUBECONFIG=/etc/kubernetes/admin.conf
helm upgrade --install -n kube-system aws-cloud-controller-manager .

You can see we already have to edit the values file to specify cluster cidr and cluster name (which there is zero documentation for, mind you), so adding another update to change the image version is just another line of code.

atsai1220 commented 3 months ago

We have been managing k8s cluster on ec2 instances with cloud-provider=external then installing this aws cloud-provider. We leverage the helm-controller inside k3s/rke2 to bootstrap the cluster with this cloud-provider.

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: cloud-provider-aws
  namespace: kube-system
spec:
  chart: https://github.com/kubernetes/cloud-provider-aws/releases/download/helm-chart-aws-cloud-controller-manager-0.0.7/aws-cloud-controller-manager-0.0.7.tgz
  targetNamespace: kube-system
  bootstrap: true
  valuesContent: |-
    args:
      - --v=2
      - --cloud-provider=aws
      - --controllers=cloud-node,cloud-node-lifecycle,service,-route
      - '--cloud-config=/tmp/cloud-provider-config'
    nodeSelector:
      node-role.kubernetes.io/control-plane: "true"
    tolerations:
    - effect: NoSchedule
      key: ""
      operator: Exists
    - effect: NoExecute
      key: ""
      operator: Exists
    extraVolumes:
    - hostPath:
        path: /var/lib/rancher/rke2/etc/config-files/cloud-provider-config
        type: File
      name: cloud-provider-config
    extraVolumeMounts:
    - mountPath: /tmp/cloud-provider-config
      name: cloud-provider-config
apt-x4869 commented 3 months ago

We are managing multiple cluster. We need cloud provider helm chart to install for kubernetes cluster to work. We have automated process for upgrading and testing clusters so it is easier to use helm chart values to update , rather that manually updating in a yaml file. if something goes wrong in testing , it could be rolled back easily using helm.

I think its better to keep the helm chart.

954

jobcespedes commented 3 months ago

Currently using helm chart for self managed clusters

sdlarsen commented 2 months ago

We're using it for self-managed clusters as well, specifying the tag in a values.yaml.