Unable to upgrade helm chart from 4.3.0 to 4.7.0 after upgrading cluster from v1.23 to v1.27

[armujahid]

What happened:

I recently upgraded an old EKS cluster from v1.23 to v1.27. Now I am unable to upgrade ingress-nginx chart 4.3.0 to 4.7.0 I am getting this error after terraform apply

│ Error: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "ingress-nginx-controller" namespace: "ingress-nginx" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"
│ ensure CRDs are installed first

What you expected to happen:

Helm should upgrade ingress-nginx without any issue. Note that I know that the autoscaling/v2beta2 API version of HorizontalPodAutoscaler is no longer served as of v1.26 but latest chart has autoscaling/v2 so helm should still upgrade it as per my understanding

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
  Release:       v1.4.0
  Build:         50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

Kubernetes version (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.2-eks-c12679a", GitCommit:"002c6bc4e142b1f48b9405140e559194a094dcff", GitTreeState:"clean", BuildDate:"2023-05-22T20:23:13Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS EKS

• OS (e.g. from /etc/os-release): Bottlerocket OS 1.14.1 (aws-k8s-1.27)

• Kernel (e.g. uname -a): 5.15.108

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc. Cluster was installed using terraform eks blueprints https://github.com/aws-ia/terraform-aws-eks-blueprints (Currently at v4.32.1)

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

    
    NAME                                         STATUS   ROLES    AGE    VERSION               INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                KERNEL-VERSION   CONTAINER-RUNTIME
    a.something.compute.internal   Ready    <none>   23h    v1.27.1-eks-61789d8   10.3.11.158   <none>        Bottlerocket OS 1.14.1 (aws-k8s-1.27)   5.15.108         containerd://1.6.20+bottlerocket
    ----

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

    ingress-nginx                     ingress-nginx   6           2023-01-13 16:12:46.72856264 +0500 PKT  deployed    ingress-nginx-4.3.0                     1.4.0      

  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

USER-SUPPLIED VALUES:
commonLabels: {}
controller:
  addHeaders: {}
  admissionWebhooks:
    annotations: {}
    certificate: /usr/local/certificates/cert
    createSecretJob:
      resources: {}
    enabled: true
    existingPsp: ""
    extraEnvs: []
    failurePolicy: Fail
    key: /usr/local/certificates/key
    labels: {}
    namespaceSelector: {}
    networkPolicyEnabled: false
    objectSelector: {}
    patch:
      enabled: true
      labels: {}
      nodeSelector:
        kubernetes.io/os: linux
      podAnnotations: {}
      priorityClassName: ""
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
      tolerations: []
    patchWebhookJob:
      resources: {}
    port: 8443
    service:
      annotations: {}
      externalIPs: []
      loadBalancerSourceRanges: []
      servicePort: 443
      type: ClusterIP
  affinity: {}
  allowSnippetAnnotations: true
  annotations: {}
  autoscaling:
    behavior:
      scaleDown:
        policies:
        - periodSeconds: 180
          type: Pods
          value: 1
        stabilizationWindowSeconds: 300
    enabled: true
    maxReplicas: 12
    minReplicas: 3
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 75
  autoscalingTemplate: []
  config:
    enable-real-ip: true
    forwarded-for-header: proxy_protocol
    proxy-real-ip-cidr: 10.3.0.0/16
    real-ip-header: proxy_protocol
    use-forwarded-headers: "true"
    use-proxy-protocol: "true"
  configAnnotations: {}
  configMapNamespace: ""
  containerName: controller
  containerPort:
    http: 80
    https: 443
  dnsConfig: {}
  dnsPolicy: ClusterFirst
  electionID: ingress-controller-leader
  enableMimalloc: true
  existingPsp: ""
  extraArgs: {}
  extraContainers: []
  extraEnvs: []
  extraInitContainers: []
  extraModules: []
  extraVolumeMounts: []
  extraVolumes: []
  healthCheckHost: ""
  healthCheckPath: /healthz
  hostNetwork: false
  hostPort:
    enabled: false
    ports:
      http: 80
      https: 443
  hostname: {}
  ingressClass: nginx
  ingressClassByName: false
  ingressClassResource:
    controllerValue: k8s.io/ingress-nginx
    default: false
    enabled: true
    name: nginx
    parameters: {}
  keda:
    apiVersion: keda.sh/v1alpha1
    behavior: {}
    cooldownPeriod: 300
    enabled: false
    maxReplicas: 11
    minReplicas: 1
    pollingInterval: 30
    restoreToOriginalReplicaCount: false
    scaledObject:
      annotations: {}
    triggers: []
  kind: Deployment
  labels: {}
  lifecycle:
    preStop:
      exec:
        command:
        - /wait-shutdown
  livenessProbe:
    failureThreshold: 5
    httpGet:
      path: /healthz
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
  maxmindLicenseKey: ""
  metrics:
    enabled: true
    port: 10254
    portName: metrics
    prometheusRule:
      additionalLabels: {}
      enabled: false
      rules: []
    service:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      externalIPs: []
      loadBalancerSourceRanges: []
      servicePort: 10254
      type: ClusterIP
    serviceMonitor:
      additionalLabels: {}
      enabled: true
      metricRelabelings: []
      namespace: ""
      namespaceSelector: {}
      relabelings: []
      scrapeInterval: 30s
      targetLabels: []
  minAvailable: 50%
  minReadySeconds: 0
  name: controller
  nodeSelector:
    kubernetes.io/os: linux
  podAnnotations: {}
  podLabels: {}
  podSecurityContext: {}
  priorityClassName: ""
  proxySetHeaders: {}
  publishService:
    enabled: true
    pathOverride: ""
  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
  replicaCount: 3
  reportNodeInternalIp: false
  resources:
    requests:
      cpu: 100m
      memory: 210Mi
  scope:
    enabled: false
    namespace: ""
    namespaceSelector: ""
  service:
    annotations:
      labels: {}
      service.beta.kubernetes.io/aws-load-balancer-alpn-policy: None
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-south-1:684485736903:certificate/7e4fda03-a2d3-4a6d-be66-5d4eae530984
      service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: proxy_protocol_v2.enabled=true
      service.beta.kubernetes.io/aws-load-balancer-type: external
    appProtocol: true
    enableHttp: true
    enableHttps: true
    enabled: true
    external:
      enabled: true
    externalIPs: []
    internal:
      annotations: {}
      enabled: false
      loadBalancerSourceRanges: []
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    nodePorts:
      http: ""
      https: ""
      tcp: {}
      udp: {}
    ports:
      http: 80
      https: 443
    targetPorts:
      http: http
      https: http
    type: LoadBalancer
  shareProcessNamespace: false
  sysctls: {}
  tcp:
    annotations: {}
    configMapNamespace: ""
  terminationGracePeriodSeconds: 300
  tolerations: []
  topologySpreadConstraints:
  - labelSelector:
      matchLabels:
        app.kubernetes.io/instance: ingress-nginx
    maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
  udp:
    annotations: {}
    configMapNamespace: ""
  updateStrategy: {}
  watchIngressWithoutClass: false
defaultBackend:
  affinity: {}
  autoscaling:
    annotations: {}
    enabled: false
    maxReplicas: 2
    minReplicas: 1
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 50
  containerSecurityContext: {}
  enabled: false
  existingPsp: ""
  extraArgs: {}
  extraEnvs: []
  extraVolumeMounts: []
  extraVolumes: []
  labels: {}
  livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 30
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
  minAvailable: 1
  name: defaultbackend
  nodeSelector:
    kubernetes.io/os: linux
  podAnnotations: {}
  podLabels: {}
  podSecurityContext: {}
  port: 8080
  priorityClassName: ""
  readinessProbe:
    failureThreshold: 6
    initialDelaySeconds: 0
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5
  replicaCount: 1
  resources: {}
  service:
    annotations: {}
    externalIPs: []
    loadBalancerSourceRanges: []
    servicePort: 80
    type: ClusterIP
  serviceAccount:
    automountServiceAccountToken: true
    create: true
    name: ""
  tolerations: []
dhParam: null
imagePullSecrets: []
podSecurityPolicy:
  enabled: false
portNamePrefix: ""
rbac:
  create: true
  scope: false
revisionHistoryLimit: 10
serviceAccount:
  annotations: {}
  automountServiceAccountToken: true
  create: true
  name: ""
tcp: {}
udp: {}

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances N/A

• Current State of the controller:
  • kubectl describe ingressclasses

Name:         alb
Labels:       app.kubernetes.io/instance=aws-load-balancer-controller
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=aws-load-balancer-controller
              app.kubernetes.io/version=v2.4.7
              helm.sh/chart=aws-load-balancer-controller-1.4.8
Annotations:  meta.helm.sh/release-name: aws-load-balancer-controller
              meta.helm.sh/release-namespace: kube-system
Controller:   ingress.k8s.aws/alb
Events:       <none>

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.4.0
              helm.sh/chart=ingress-nginx-4.3.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide
• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
  • copy/paste of the snippet (if applicable)
  • kubectl describe ... of any custom configmap(s) created and in use
  • Any other related information that may help

How to reproduce this issue:

1. Provision a v1.23 eks cluster
2. install ingress-nginx chart 4.3.0 https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx/4.3.0
3. Incrementally upgrade cluster to 1.27
4. Now try to upgrade ingress-nginx to latest version

Anything else we need to know:

I should have upgraded it before upgrading the EKS version but I think this case should also be supported. I don't want to reinstall it since that will recreate the AWS NLB and I will have to update all the CNAME records.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

I think there is too much shortage of resources to test all the upgrade paths from all older versions to all later versions, during CI.

The CI only tests a fresh install so that KEPs are complied with.

Such breaking changes do disrupt uptime but I feel the resources required to check all upgrade paths is currently not available.

/remove-kind bug

[strongjz]

Helm upgrades do not upgrade CRDs; they have to be done separately.

It looks like this was fixed here https://github.com/kubernetes/ingress-nginx/commit/06612e6ffd044171e33b5e8315b9605eb8730c55

Does the helm chart render the right template for hpa?

Is the hpa CRD's upgraded in your cluster? That is not something I think the ingress-nginx controller helm chart will do.

[armujahid]

@strongjz

Does the helm chart render the right template for hpa?

Yes, it's rendering autoscaling/v2

Is the hpa CRD's upgraded in your cluster?

Yes of course EKS ensures that, Re-installation of chart worked perfectly fine with exact version and helm values on that same cluster.

[strongjz]

So it works? I'm confused about the issue then.

[armujahid]

@strongjz Helm upgrade doesn't work. That's what the issue is about. Re-installation (Uninstall => Install) works but that recreates cloud components (e.g. AWS NLB in my case) instead of reusing existing components.

[sirkubax]

I can confirm, that the upgrade to 4.7.1 fails

Well, I think ingress-nginx is/should-be mature enough, NOT to have braking changes :/

Or at least it should be clearly mentioned, that the upgrades are likely to fail, and the solutions is X where X is to deinstall old version and install new chart...

As previously stated, helm template or helm install dry-run produce proper template...

 Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "nginx-ingress-controller" namespace: "nginx-ingress" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"
ensure CRDs are installed first

[longwuyuan]

Its originating from the deprecation of HPA I think

[abh1kg]

Before executing helm upgrade, you would need to upgrade the API versions of the current release manifest:

1. create a file (e.g. named deprecated-apis.yaml) with the content:

mappings:
  - deprecatedAPI: "apiVersion: extensions/v1beta1\nkind: Deployment\n"
    newAPI: "apiVersion: apps/v1\nkind: Deployment\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta1\nkind: Deployment\n"
    newAPI: "apiVersion: apps/v1\nkind: Deployment\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta2\nkind: Deployment\n"
    newAPI: "apiVersion: apps/v1\nkind: Deployment\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta1\nkind: StatefulSet\n"
    newAPI: "apiVersion: apps/v1\nkind: StatefulSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta2\nkind: StatefulSet\n"
    newAPI: "apiVersion: apps/v1\nkind: StatefulSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: extensions/v1beta1\nkind: DaemonSet\n"
    newAPI: "apiVersion: apps/v1\nkind: DaemonSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta2\nkind: DaemonSet\n"
    newAPI: "apiVersion: apps/v1\nkind: DaemonSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: extensions/v1beta1\nkind: ReplicaSet\n"
    newAPI: "apiVersion: apps/v1\nkind: ReplicaSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta1\nkind: ReplicaSet\n"
    newAPI: "apiVersion: apps/v1\nkind: ReplicaSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: apps/v1beta2\nkind: ReplicaSet\n"
    newAPI: "apiVersion: apps/v1\nkind: ReplicaSet\n"
    deprecatedInVersion: "v1.9"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: extensions/v1beta1\nkind: NetworkPolicy\n"
    newAPI: "apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\n"
    deprecatedInVersion: "v1.8"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: extensions/v1beta1\nkind: PodSecurityPolicy\n"
    newAPI: "apiVersion: policy/v1beta1\nkind: PodSecurityPolicy\n"
    deprecatedInVersion: "v1.10"
    removedInVersion: "v1.16"
  - deprecatedAPI: "apiVersion: admissionregistration.k8s.io/v1beta1\nkind: MutatingWebhookConfiguration\n"
    newAPI: "apiVersion: admissionregistration.k8s.io/v1\nkind: MutatingWebhookConfiguration\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: admissionregistration.k8s.io/v1beta1\nkind: ValidatingWebhookConfiguration\n"
    newAPI: "apiVersion: admissionregistration.k8s.io/v1\nkind: ValidatingWebhookConfiguration\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: apiextensions.k8s.io/v1beta1\nkind: CustomResourceDefinition\n"
    newAPI: "apiVersion: apiextensions.k8s.io/v1\nkind: CustomResourceDefinition\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: apiregistration.k8s.io/v1beta1\nkind: APIService\n"
    newAPI: "apiVersion: apiregistration.k8s.io/v1\nkind: APIService\n"
    deprecatedInVersion: "v1.19"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: apiregistration.k8s.io/v1beta1\nkind: APIServiceList\n"
    newAPI: "apiVersion: apiregistration.k8s.io/v1\nkind: APIServiceList\n"
    deprecatedInVersion: "v1.19"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: authentication.k8s.io/v1beta1\nkind: TokenReview\n"
    newAPI: "apiVersion: authentication.k8s.io/v1\nkind: TokenReview\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: authorization.k8s.io/v1beta1\nkind: LocalSubjectAccessReview\n"
    newAPI: "apiVersion: authorization.k8s.io/v1\nkind: LocalSubjectAccessReview\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: authorization.k8s.io/v1beta1\nkind: SelfSubjectAccessReview\n"
    newAPI: "apiVersion: authorization.k8s.io/v1\nkind: SelfSubjectAccessReview\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: authorization.k8s.io/v1beta1\nkind: SubjectAccessReview\n"
    newAPI: "apiVersion: authorization.k8s.io/v1\nkind: SubjectAccessReview\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: autoscaling/v2beta1\nkind: HorizontalPodAutoscaler\n"
    newAPI: "apiVersion: autoscaling/v2\nkind: HorizontalPodAutoscaler\n"
    deprecatedInVersion: "v1.23"
    removedInVersion: "v1.25"
  - deprecatedAPI: "apiVersion: autoscaling/v2beta2\nkind: HorizontalPodAutoscaler\n"
    newAPI: "apiVersion: autoscaling/v2\nkind: HorizontalPodAutoscaler\n"
    deprecatedInVersion: "v1.23"
    removedInVersion: "v1.26"
  - deprecatedAPI: "apiVersion: batch/v1beta1\nkind: CronJob\n"
    newAPI: "apiVersion: batch/v1\nkind: CronJob\n"
    deprecatedInVersion: "v1.21"
    removedInVersion: "v1.25"
  - deprecatedAPI: "apiVersion: certificates.k8s.io/v1beta1\nkind: CertificateSigningRequest\n"
    newAPI: "apiVersion: certificates.k8s.io/v1\nkind: CertificateSigningRequest\n"
    deprecatedInVersion: "v1.19"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: coordination.k8s.io/v1beta1\nkind: Lease\n"
    newAPI: "apiVersion: coordination.k8s.io/v1\nkind: Lease\n"
    deprecatedInVersion: "v1.14"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: extensions/v1beta1\nkind: Ingress\n"
    newAPI: "apiVersion: networking.k8s.io/v1beta1\nkind: Ingress\n"
    deprecatedInVersion: "v1.14"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: networking.k8s.io/v1beta1\nkind: Ingress\n"
    newAPI: "apiVersion: networking.k8s.io/v1\nkind: Ingress\n"
    deprecatedInVersion: "v1.19"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: networking.k8s.io/v1beta1\nkind: IngressClass\n"
    newAPI: "apiVersion: networking.k8s.io/v1\nkind: IngressClass\n"
    deprecatedInVersion: "v1.19"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: policy/v1beta1\nkind: PodDisruptionBudget\n"
    newAPI: "apiVersion: policy/v1\nkind: PodDisruptionBudget\n"
    deprecatedInVersion: "v1.21"
    removedInVersion: "v1.25"
  - deprecatedAPI: "apiVersion: policy/v1beta1\nkind: PodSecurityPolicy\n"
    removedInVersion: "v1.25"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: ClusterRole\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: ClusterRoleList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: ClusterRoleBinding\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: ClusterRoleBindingList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBindingList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: Role\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: Role\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: RoleList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: RoleList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: RoleBinding\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1alpha1\nkind: RoleBindingList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBindingList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRoleList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRoleBinding\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRoleBindingList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBindingList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: Role\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: Role\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: RoleList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: RoleList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: RoleBinding\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBinding\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: RoleBindingList\n"
    newAPI: "apiVersion: rbac.authorization.k8s.io/v1\nkind: RoleBindingList\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: scheduling.k8s.io/v1beta1\nkind: PriorityClass\n"
    newAPI: "apiVersion: scheduling.k8s.io/v1\nkind: PriorityClass\n"
    deprecatedInVersion: "v1.14"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: storage.k8s.io/v1beta1\nkind: CSIDriver\n"
    newAPI: "apiVersion: storage.k8s.io/v1\nkind: CSIDriver\n"
    deprecatedInVersion: "v1.19"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: storage.k8s.io/v1beta1\nkind: CSINode\n"
    newAPI: "apiVersion: storage.k8s.io/v1\nkind: CSINode\n"
    deprecatedInVersion: "v1.17"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: storage.k8s.io/v1beta1\nkind: StorageClass\n"
    newAPI: "apiVersion: storage.k8s.io/v1\nkind: StorageClass\n"
    deprecatedInVersion: "v1.16"
    removedInVersion: "v1.22"
  - deprecatedAPI: "apiVersion: storage.k8s.io/v1beta1\nkind: VolumeAttachment\n"
    newAPI: "apiVersion: storage.k8s.io/v1\nkind: VolumeAttachment\n"
    deprecatedInVersion: "v1.13"
    removedInVersion: "v1.22"

2. Execute:

helm plugin install https://github.com/helm/helm-mapkubeapis
helm mapkubeapis --mapfile "./deprecated-apis.yaml" "ingress-nginx"

3. Run the helm upgrade command

Please see https://helm.sh/docs/topics/kubernetes_apis/ for the motivation behind doing so.

[MI180388]

Facing same issue for OKE 1.26 version. Could anyone guide for resolution

[kfox1111]

I'd also recommend this tool: https://pluto.docs.fairwinds.com/quickstart/ running: pluto detect-helm -r -o wide

before every Kubernetes minor upgrade. Make sure it comes back clean before upgrading to avoid these issues in the future.

[strongjz]

Unfortunately, the CI only tests uninstall and installs but not upgrades. We will have to investigate if helm chart install we use in CI can do upgrades.

[hmi12]

so,how you guys to perform rolling upgrade with the minimum downtime?

[richardmatthewsdev]

Hey, I am running into this issue as well. Is there any update?

Thanks :D

[longwuyuan]

Both the from and to versions mentioned in the issue description are not supported anymore.

We are not able to maintain migration docs for migration paths as that is just too much for the negligible resources available.

Please check the Release Notes and Changelogs, in addition to the support matrix on the README of the project.

There is no action item for the project here so I am closing the issue as it is just adding to the tally of open issues with no action item.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/10132#issuecomment-2348237106): >Both the from and to versions mentioned in the issue description are not supported anymore. > >We are not able to maintain migration docs for migration paths as that is just too much for the negligible resources available. > >Please check the Release Notes and Changelogs, in addition to the support matrix on the README of the project. > >There is no action item for the project here so I am closing the issue as it is just adding to the tally of open issues with no action item. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.