search

kubernetes-sigs / karpenter

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
Apache License 2.0
577 stars 193 forks source link

Allow registration TTL to be user configurable #357

Open garvinp-stripe opened 1 year ago

garvinp-stripe commented 1 year ago

Tell us about your request

At this time registration TTL is hard code to 15 minutes. It would be greatly appreciated if we can make this configurable by the user.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

At this time registration TTL is hard code, https://github.com/aws/karpenter-core/blob/main/pkg/controllers/machine/lifecycle/liveness.go#L38, however some of our nodes takes close to and some times over the 15 minute time window to run -> bootstrap -> join the cluster. This means that these nodes will be terminated after 15 minutes even though they would have been able to join the cluster.

Are you currently working around this issue?

Since this isn't user configurable, we have the fork the code and build our own image for Karpenter to bypass this limitation.

Additional Context

https://kubernetes.slack.com/archives/C02SFFZSA2K/p1685123613509889 Conversation from slack

Attachments

No response

Community Note

sadath-12 commented 1 year ago

Were you able to find a way to reduce the time for boostraping the cluster at your end itself , since if you try to keep a higher registeration TTL you might encounter additional latency in provision of other nodes here (You will notice this when there are many of them you require) , When we think in terms of our reconcillation time and performance trade offs handling. Generally there might be a good chance to improve the boostraping process over configuring the timeouts here since there are higher possibilities you might end up compounding your latencies

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

garvinp-stripe commented 7 months ago

To some degree we were able to address our bootstrapping times however we continue to run into issues with GPU based nodes. This is because without significantly fragmenting our nodepools because essentially we would have to bake an AMI per driver variants and version, we can't actually drive down the bootstrap time.

Instances like x1.32xlarge takes 20 minutes for us to start up due to downloading + installing drivers. I suspect this is common for those who use GPUs. Chatted with @Bryce-Soghigian who also have some data points

Bryce-Soghigian commented 7 months ago

/remove-lifecycle stale

Bryce-Soghigian commented 7 months ago

AKS has some failures where the E2E tests we have for GPU bootstrapping will fail due to taking to long to initialize some of the GPU instances. (Would have to go digging its not common since we use small GPU skus for the e2es)

One could argue, use Prebaked GPU images. Bootstrapping taking longer than 15 minutes is ridiculous and we should insist on the highest standard. This is a pattern AKS is moving away from see dedicated GPU VHD and the vhds for Ubuntu2204 will not be supporting this pattern. Even if they did, it doesn't fully solve the problem. If I want to use GPU operator to leverage some drivers AKS doesn't have, or some other pattern to install the GPU drivers myself and manage that lifecycle, then it can't be baked in. (Note azure/karpenter-provider-azure, does not support GPU operator currently, but is an interesting direction)

Cluster Autoscaler exposes MaxProvisioningTime which serves a similar purpose(CAS GCs nodes/vms that don't register with the api by MaxProvisioningTime)

tallaxes commented 7 months ago

Allow registration TTL to be user configurable

Other alternatives to consider may be provider-configurable (a more conservative step forward, likely a low-hanging fruit), or internally configurable by instance type (assuming all known use cases are directly related to using particular instance types).

tallaxes commented 7 months ago

Also, here is the updated code link: https://github.com/kubernetes-sigs/karpenter/blob/0fea7cee34594515d5b4e53dbd0f12ab721c482e/pkg/controllers/nodeclaim/lifecycle/liveness.go#L38-L40

jonathan-innis commented 7 months ago

I'm open to make this a var and provider-configurable. I'm not 100% sold yet that this should be user-configurable since ideally we should just enforce best practices rather than surfacing a bunch of options that allow users to get away with the wrong thing.

garvinp-stripe commented 7 months ago

I guess rather than a raw number, providing users with 2 different TTLs. One for normal instances (15minutes) and second for slow start up (30 minutes?). That way we provide some flexibility without completely allowing users to make a really long duration? We can expose this via EC2NodeClass or the config map instance type override config map https://github.com/kubernetes-sigs/karpenter/issues/751#issuecomment-1996074401

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 2 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/karpenter/issues/357#issuecomment-2282914388): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
leoryu commented 3 weeks ago

Is ther any plan for this issue?

k8s-ci-robot commented 1 week ago

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.