search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.05k stars 3.97k forks source link

Add support for other kinds of Daemonset Kube Resource (CRD) #3873

Closed clamoriniere closed 2 years ago

clamoriniere commented 3 years ago

component: cluster-autoscaler

Issue description

The cluster-autoscaler uses the Daemonsets spec to simulate the resources needed for the Daemonset’s pods on a new node. We are looking to extend the capability of the cluster-autoscaler in terms of watching for other resources that implement an equivalent of a Daemonset (one Pod deployed on each Node).

We have the following established open-source project github.com/DataDog/ExtendedDaemonset that defines a CRD (ExtendedDaemonset) that replaces the Daemonset object for some of our applications that need to be deployed on all nodes. One of the reasons that we have worked on this project is to allow a canary deployment on a subset of nodes,as well as have more control over the rollout of an application (velocity, pausing…).

We made this previous contribution that allows the cluster-autoscaler controller to recognize the Pods created by the ExtendedDaemonset as Daemonset’s Pod.

Now the idea is to provide a way for the controller to consider other kinds of Daemonset resources when it estimates the Daemonsets’ pod resources usage in the GetDaemonSetPodsForNode() function

Proposed solutions

To make it possible, we identified two ways to do it:

  1. Add the possibility for the cluster-autoscaler controller to look at PodTemplates resources in addition to the Daemonsets. To avoid querying PodTemplates that are not linked to our use case, we can limit the query with a LabelSelector on a dedicated label like cluster-autoscaler.kubernetes.io/daemonset-pod. Then we can simulate a pod creation like for the Daemonset.

  2. Add the possibility to list any kind of resource that contains a PodTemplate in its spec section.

    We can imagine providing a list of resource metadata (kind, apiVersion) as controller parameters. Attach to each resource, also the path to the PodTemplate in the spec section.

    daemonsetResources:
- kind: CustomeDaemonset
 apiVersion: foo.io/v1
 podTemplate:
   path: .spec.template
   kind: PodTemplate
   apiVersion: v1

    Then, using the dynamic goclient, we can query the api-server to get the list of resources, and then decoding the object, access the PodTemplate section.

Questions

elmiko commented 3 years ago

What do you think about this feature?

this feature seems reasonable to me

Which implementation solution makes more sense for you? Do you see any other alternatives?

i like that option 1 uses labels to help further refine the search, i would be ok with something that could look at any resource with a PodTemplate in its spec, provided that the "anything" has a label or annotation which marks it as such.

i guess that means i like parts of both implementations presented. maybe this is another alternative, eg any object with cluster-autoscaler.kubernetes.io/daemonset-pod label, provided it contains a PodTemplate in the spec.

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/autoscaler/issues/3873#issuecomment-968292867): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.