Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "emissary-ingress" namespace: "emissary" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"

[imoisharma]

Describe the bug I am running Emissary Ingress Version: 2.5.Z via Helm inside my EKS cluster. I am trying to upgrade to the Emissary Ingress from 2.5.Z to 3.6.0 or to the latest version.

On running this command

helm upgrade -f my-values.yaml emissary-ingress datawire/emissary-ingress -n emissary --version 8.6.0

I am getting this error

Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "emissary-ingress" namespace: "emissary" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"

To Reproduce Steps to reproduce the behaviour:

1. Go to https://www.getambassador.io/docs/emissary/3.6/topics/install/upgrade/helm/emissary-2.5/emissary-3.X
2. Follow the Migration Steps
3. After removing and adding the datawire repo in Helm
4. Run the above-mentioned command for the helm upgrade

Expected behaviour Helm upgrade should be successfully able to install the newest version of Emissary Ingress inside my EKS Cluster without showing an HPA error.

Versions (please complete the following information):

• Ambassador: 2.5.Z
• Chart Info: emissary-ingress-7.4.1
• Chart App Version Info: 2.3.1
• EKS Version: 1.27

Additional context My my-values.yaml already has a new HPA syntax which is supported by v2 but still it's failed with the above mentioned error.

autoscaling: # +doc-gen:break
  enabled: false
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 100
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 300

[eevdev]

I believe this was fixed in 03a32471a3bd2652770660abe141de2f4627d8db, which is unreleased yet.

[CheyiLin]

Even with the latest chart version 8.9.1 I still got the same error when upgrading from 8.5.2 on AWS EKS 1.27 via Terraform resource helm_release.

It should update HPA from autoscaling/v2beta2 to autoscaling/v2 as expected.

Is any workaround for this?

[CheyiLin]

Just found that the error unable to build kubernetes objects from current release manifest means the problem is the current release which lives in cluster cannot be built due to its API version autoscaling/v2beta2 was deprecated in the 1.27, not the new rendered helm release.

And it is a known issue of helm https://github.com/helm/helm/issues/7219.

A workable solution is using helm plugin mapkubeapis to fix/replace deprecated API version autoscaling/v2beta2 in current release:

1. Install plugin mapkubeapis

   helm plugin install https://github.com/helm/helm-mapkubeapis

2. Edit the config/Map.yaml

   Run helm mapkubeapis --help to check the location of Map.yaml. For macOS it should be ~/Library/helm/plugins/helm-mapkubeapis/config/Map.yaml.

   Add missing autoscaling/v2beta2 config:

   - deprecatedAPI: "apiVersion: autoscaling/v2beta2\nkind: HorizontalPodAutoscaler\n"
    newAPI: "apiVersion: autoscaling/v2\nkind: HorizontalPodAutoscaler\n"
    deprecatedInVersion: "v1.23"
    removedInVersion: "v1.26"

   refs: https://github.com/helm/helm-mapkubeapis/pull/108

3. Run helm mapkubeapis --namespace <emissary-namespace> <emissary-ingress-release-name>

   $ helm mapkubeapis --namespace emissary emissary-ingress
   2023/11/22 18:17:23 Release 'emissary-ingress' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
   2023/11/22 18:17:23 Get release 'emissary-ingress' latest version.
   2023/11/22 18:17:25 Check release 'emissary-ingress' for deprecated or removed APIs...
   2023/11/22 18:17:26 Found 1 instances of deprecated or removed Kubernetes API:
   "apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   "
   Supported API equivalent:
   "apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   "
   2023/11/22 18:17:26 Finished checking release 'emissary-ingress' for deprecated or removed APIs.
   2023/11/22 18:17:26 Deprecated or removed APIs exist, updating release: emissary-ingress.
   2023/11/22 18:17:26 Set status of release version 'emissary-ingress.v10' to 'superseded'.
   2023/11/22 18:17:27 Release version 'emissary-ingress.v10' updated successfully.
   2023/11/22 18:17:27 Add release version 'emissary-ingress.v11' with updated supported APIs.
   2023/11/22 18:17:29 Release version 'emissary-ingress.v11' added successfully.
   2023/11/22 18:17:29 Release 'emissary-ingress' with deprecated or removed APIs updated successfully to new version.
   2023/11/22 18:17:29 Map of release 'emissary-ingress' deprecated or removed APIs to supported versions, completed successfully.

4. Upgrade the chart to 8.9.0+

[ramon-carrasco]

Thanks @CheyiLin for your answer. By following that same process I was able to fix the issue locally. In my case I manage this particular chart with terraform, so one workaround I found in order to fix this issue was to: 1- remove the autoscaling field from my values.yaml file 2- upgrade the emissary crds version to 3.9.0 and chart version to 8.9.0 3- run pipeline and allow it to create the new charts with the new version 4- re-add autoscaling and run the pipeline again with the new version