[Bug]: Karpenter failng to provision due to invalid node claim

[mschnee]

Prior Search

• [X] I have already searched this project's issues to determine if a bug report has already been made.

What happened?

Karpenter is throwing errors attempting to create Node claims.:

creating node claim, NodeClaim.karpenter.sh \"spot-8c7mc\" is invalid: [spec.requirements: Too many: 32: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]; creating node claim, NodeClaim.karpenter.sh \"burstable-2l6cf\" is invalid: [spec.requirements: Too many: 33: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]

Steps to Reproduce

Working through the bootstrap documentation, when attempting to scale up a service to test Karpenter getting provisioining nodes.

Version

main (development branch)

Relevant log output

{
  "level": "ERROR",
  "time": "2024-05-29T22:07:31.187Z",
  "logger": "controller.provisioner",
  "message": "creating node claim, NodeClaim.karpenter.sh \"burstable-9ns7l\" is invalid: [spec.requirements: Too many: 33: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]; creating node claim, NodeClaim.karpenter.sh \"burstable-k4q92\" is invalid: [spec.requirements: Too many: 33: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]; creating node claim, NodeClaim.karpenter.sh \"spot-qvhz8\" is invalid: [spec.requirements: Too many: 32: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]",
  "commit": "8b2d1d7"
}

{
  "level": "ERROR",
  "time": "2024-05-29T22:07:35.510Z",
  "logger": "controller.provisioner",
  "message": "creating node claim, NodeClaim.karpenter.sh \"spot-8c7mc\" is invalid: [spec.requirements: Too many: 32: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]; creating node claim, NodeClaim.karpenter.sh \"burstable-2l6cf\" is invalid: [spec.requirements: Too many: 33: must have at most 30 items, <nil>: Invalid value: \"null\": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]",
  "commit": "8b2d1d7"
}

[fullykubed]

I believe you are referencing this guide section where it asks you to scale cert-manager to 25 replicas to validate that Karpenter is working properly.

That guide section has unfortunately become out-of-date which I believe caused confusion:

• the jetstack-cert-manager deployment no longer exists (is now called cert-manager)
• the cert-manager deployment now does not contain any podAntiAffinity rules because it is now designed to run as a single replica only. Scaling it up will not trigger Karpenter as multiple pods can be scheduled on the same node.
• if you did test with the cert-manager-webhook (which is designed to run HA), I can confirm that 25 is too high of a number. While Karpenter can provision arbitrarily high number of nodes, the podAntiAffinity rules for our core controllers ensures that each pod in a set runs on a separate instance type to prevent issues that might affect a particular instance type (e.g., spot instance scale-in). Each new pod adds a new scheduling constraint, and as you can see from the logs, Karpenter can only handle 30 constraints. However, if you use 10 instead of 25, provisioning will proceed as expected.

I will update the edge documentation to correct these errors.

[fullykubed]

The documentation has been updated.