Closed christophercorn closed 1 week ago
karpenter-6b785d6cb7-qvlmx controller {"level":"INFO","time":"2024-07-24T00:24:21.340Z","logger":"controller.nodeclaim.lifecycle","message":"initialized nodeclaim","commit":"6b868db","nodeclaim":"test-amd64-mkvrb","provider-id":"aws:///us-west-2c/i-0bdfa9539de03d1e2","node":"ip-10-129-89-193.us-west-2.compute.internal","allocatable":{"cpu":"1930m","ephemeral-storage":"18242267924","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7179184Ki","pods":"29"}}
karpenter-6b785d6cb7-qvlmx controller {"level":"INFO","time":"2024-07-24T00:24:21.987Z","logger":"controller.disruption.queue","message":"command succeeded","commit":"6b868db","command-id":"02393ae8-52ee-4bf0-9ce3-ce081a934367"}
karpenter-6b785d6cb7-qvlmx controller {"level":"INFO","time":"2024-07-24T00:24:22.014Z","logger":"controller.node.termination","message":"tainted node","commit":"6b868db","node":"ip-10-129-90-75.us-west-2.compute.internal"}
karpenter-6b785d6cb7-qvlmx controller {"level":"INFO","time":"2024-07-24T00:24:31.072Z","logger":"controller.node.termination","message":"deleted node","commit":"6b868db","node":"ip-10-129-90-75.us-west-2.compute.internal"}
karpenter-6b785d6cb7-qvlmx controller {"level":"INFO","time":"2024-07-24T00:24:31.438Z","logger":"controller.nodeclaim.termination","message":"deleted nodeclaim","commit":"6b868db","nodeclaim":"test-amd64-568w6","node":"ip-10-129-90-75.us-west-2.compute.internal","provider-id":"aws:///us-west-2c/i-062a83255a79deefe"}
From the logs you have provide, it Seems that Karpenter waited until the node was ready and initialized. Can you show maybe k8s events that show the nodes being prevented from being able to run on the new node?
This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.
Description
Observed Behavior: when changing instance-category, karpenter will spin up a node but delete the old new before the new node is ready. at times i've even seen it completely delete the old node before the new node has even begun to spin up.
this seems like the same issue as: https://github.com/aws/karpenter-provider-aws/issues/3979
Expected Behavior: disruption controller states that the before spinning down a node, a new node must come up first
"Pre-spin any replacement nodes needed as calculated in Step (2), and wait for them to become ready. If a replacement node fails to initialize, un-taint the node(s), and restart from Step (1), starting at the first disruption method again."
https://karpenter.sh/v0.32/concepts/disruption/#disruption-controller
Reproduction Steps (Please include YAML):
karpenter related log entries
Versions:
Chart Version: v0.37.0
Kubernetes Version (
kubectl version
):Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment