aws / karpenter-provider-aws

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
https://karpenter.sh
Apache License 2.0
6.6k stars 919 forks source link

CRD configuration leads to v1 resources being served by default on v0.37.2 #6897

Open msaah-benchling opened 2 weeks ago

msaah-benchling commented 2 weeks ago

Description

Observed Behavior: While running 0.37.2, I expect to be able to use kubectl edit nodepool foo. However, this returns:

#·nodepools.karpenter.sh·"foo"·was·not·valid:$
#·*·spec.template.spec.nodeClassRef.group:·Required·value$
#·*·spec.template.spec.nodeClassRef.kind:·Required·value$
#·*·spec.disruption.consolidationPolicy:·Unsupported·value:·"WhenUnderutilized":·supported·values:·"WhenEmpty",·"WhenEmptyOrUnderutilized"$
#·*·spec.disruption.consolidateAfter:·Required·value$
#·*·<nil>:·Invalid·value:·"null":·some·validation·rules·were·not·checked·because·the·object·was·invalid;·correct·the·existing·errors·to·complete·validation$

kubectl edit nodepools.v1beta1.karpenter.sh foo works as expected.

I believe this is ultimately because v1 is being served by default:

kubectl get nodepool foo -oyaml | grep apiVersion
apiVersion: karpenter.sh/v1

~this could potentially be mitigated by enabling the conversion webhook, which is disabled by default.~ this did not work.

Expected Behavior:

kubectl edit works without specifying the fully qualified version. more broadly, v1beta1 should be served by default since it's still to my knowledge it's still the officially used version for 0.37.2.

Reproduction Steps (Please include YAML): install karpenter crds from helm chart 0.37.2 with default values

Versions:

dblackdblack commented 2 weeks ago

Simple repro case:

$ kubectl --kubeconfig kubeconfig.yaml --context my-cluster-context annotate nodepool/graviton foo=bar

The NodePool "graviton" is invalid: 
* spec.template.spec.nodeClassRef.group: Required value
* spec.template.spec.nodeClassRef.kind: Required value
* spec.disruption.consolidationPolicy: Unsupported value: "WhenUnderutilized": supported values: "WhenEmpty", "WhenEmptyOrUnderutilized"
* spec.disruption.consolidateAfter: Required value
* <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation
engedaam commented 1 week ago

Unfortunately, this is how Kubernetes default versioning works https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-priority. If there are more then one version supported by CRDs kubectl will need users specify the version to be served if it's not the latest. for example kubectl get nodepool.v1beta1.karpenter.sh

msaah-benchling commented 1 week ago

@engedaam understood thanks. perhaps it could be noted in the docs (maybe in the upgrade guides) for any pre-1.0 versions that enable both v1 and v1beta.