PSP Rollout is broken

[dschunack]

Hi,

the last Chart update breaks the rollout of the PSP. API Version v1 is not available on AWS EKS 1.22, 1.23 and 1.24.

Error: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1"

https://github.com/kubernetes/autoscaler/blob/a4847138d6264c124578abecaf4b3dd7c0d5b680/charts/cluster-autoscaler/templates/_helpers.tpl#L73-L74

EKS 1.22

kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION       NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1        true         PodDisruptionBudget
podsecuritypolicies    psp          policy/v1beta1   false        PodSecurityPolicy

kubectl version 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.16-eks-ffeb93d", GitCommit:"52e500d139bdef42fbc4540c357f0565c7867a81", GitTreeState:"clean", BuildDate:"2022-11-29T18:41:42Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

EKS 1.24

kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION       NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1        true         PodDisruptionBudget
podsecuritypolicies    psp          policy/v1beta1   false        PodSecurityPolicy

kubectl version                       
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

v1 doesn't exist in the API Documentation for PSP [API ref 1.24] (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podsecuritypolicy-v1beta1-policy) and PSP are removed in 1.25. It make sense to revert #5357 and #5500 .

https://github.com/kubernetes/autoscaler/blob/b57d917632e048bbe15e47d8a6084b8fd8a5919c/charts/cluster-autoscaler/templates/_helpers.tpl#L66-L76

I will create a PR to fix this in the next minutes.

[gjtempleton]

5480 should resolve this once merged.

[gjtempleton]

5480 has now been merged, if you can confirm that's fixed this, we can close this off. Thanks for raising it.

[llamahunter]

tried to update to 1.25 today, and got this:

Error: resource mapping not found for name: "aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" ensure CRDs are installed first

Using helm chart 9.28.0 and image tag v1.25.1

[josecsotomorales]

Upgraded to k8s 1.25 ... getting this issue with helm chart 9.28.0 as well

[josecsotomorales]

This is the error I'm getting on my side: no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

[josecsotomorales]

From my research it's a Helm issue actually, tested helm template and it produces the expected policy, more details here: https://github.com/helm/helm/issues/7219

[4sudiptodas]

Upgraded to aws k8s 1.27 and getting this issue with helm chart 9.28.0, unable to build kubernetes objects from current release manifest: resource mapping not found for name: "aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first

[JCBSLMN]

i'm having the same issue as @4sudiptodas

[xiaodong-xie]

Got inspirations from this comment: https://github.com/helm/helm/issues/7219#issuecomment-1488061676

This works for me:

helm plugin install https://github.com/helm/helm-mapkubeapis
helm mapkubeapis -n kube-system cluster-autoscaler

Then upgrading cluster-autoscaler using helm upgrade --install succeeded.

[jd-sandk]

Same issue with Helm Chart: 9.29.1 when using Terraform to apply - we're were getting Error:

╷
│ Error: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
│ ensure CRDs are installed first
│ 
│   on modules/cluster/helm.tf line 45, in resource "helm_release" "cluster-autoscaler":
│   45: resource "helm_release" "cluster-autoscaler" {
│ 
╵

Exited with code exit status 1
CircleCI received exit code 1

We've even tried to explicitly set the following in the Helm Chart:

  set {
    name  = "kubeTargetVersionOverride"
    value = "1.27.0"
    type  = "string"
  }

As we are on v1.27 and due to seeing:

{{- define "podDisruptionBudget.apiVersion" -}}
{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
{{- if semverCompare "<1.21-0" $kubeTargetVersion -}}
{{- print "policy/v1beta1" -}}
{{- else -}}
{{- print "policy/v1" -}}
{{- end -}}
{{- end -}}

https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/templates/_helpers.tpl#L78-L88

Also later based upon our looking of the Template code (https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/templates/podsecuritypolicy.yaml#L1C3-L1C3) a member of our team thought the following might help trying to set the following on the Helm Chart:

  set {
    name  = "rbac.create"
    value = "true"
    type  = "string"
  }

  set {
    name  = "rbac.pspEnabled"
    value = "true"
    type  = "string"
  }

but then that also only leads to another similiar issue:

╷
│ Error: resource mapping not found for name: "cluster-autoscaler-aws-cluster-autoscaler" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
│ ensure CRDs are installed first
│ 
│   on modules/cluster/helm.tf line 45, in resource "helm_release" "cluster-autoscaler":
│   45: resource "helm_release" "cluster-autoscaler" {
│ 
╵

Exited with code exit status 1
CircleCI received exit code 1

Why is the Helm chart trying to force us to usev1beta1, when we're on EKS Cluster/Node Group (Kubernetes) Version: 1.27, when it should be trying to use v1 instead.

We have those Kinds available in our Cluster:

# kubectl api-resources --api-group='policy'   
NAME                   SHORTNAMES   APIVERSION   NAMESPACED   KIND
poddisruptionbudgets   pdb          policy/v1    true         PodDisruptionBudget

😞

[Idan-Lazar]

@jd-sandk Do you find any solutions?

[jd-sandk]

@Idan-Lazar We don't remember fully, since it was awhile ago, but we tried to recall this morning, and we want to say: that once we upgraded the Kubernetes version, then semverCompare started to play nice, and everything just worked from that point on (I think after removing and readding the HELM Chart).

[Idan-Lazar]

@jd-sandk What is your Helm Chart version now? And cluster-autoscaler tag?

[vtrenton]

Considering <1.21 has been out of support for years now. I dont really understand the reasoning of tempting the api-version for the pdb. How likely is it that it will change soon/ever from policy/v1? I'm looking at the chart and see apiVersion: {{ template "podDisruptionBudget.apiVersion" . }} per https://github.com/kubernetes/autoscaler/blob/8f75e9cf2bad02d8bd89ef6b4dd71d4994933715/charts/cluster-autoscaler/templates/pdb.yaml#L2 yet the values.yaml doesn't specify the version: https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/values.yaml#L258-L262

I'm of the opinion we just hardcode policy/v1 and simply state that this chart requires 1.21+. I dont think that's a huge ask.

[marinator86]

Hi, does anyone have a workaround for this? Would be great to share. Thanks

[gxpd-jjh]

FWIW, here is what i had to override as a parameter (using the Chart) to get autoscaler deployed on 1.25.

        - name: podDisruptionBudget
          value: 'false'
        - name: rbac.pspEnabled
          value: 'false'

The first one required because https://github.com/kubernetes/autoscaler/blob/a2f4caca4b988ea167f2e9c10070b547d5c522eb/charts/cluster-autoscaler/templates/pdb.yaml#L1 returns True always due to chart's Values.yaml.

The second one is required because https://github.com/kubernetes/autoscaler/blob/a2f4caca4b988ea167f2e9c10070b547d5c522eb/charts/cluster-autoscaler/templates/podsecuritypolicy.yaml#L1 and then there is still an issue from https://github.com/kubernetes/autoscaler/issues/5364 where they just gave up, so it somehow returns "policy/v1beta1" still which is gone in 1.25.

cc @marinator86

[sappusaketh]

is this function even working I m on 9.34.1 and even after setting kubeTargetVersionOverride: 1.28.6 still its using "policy/v1beta1" tried kubeTargetVersionOverride: v1.28.6 same error no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

I m using TF deleting chart and recreating helped in my case