[Bug]: Update Lease failed on nginx-ingress update to 3.7.0

[thomas-schn]

Version

3.7.0

What Kubernetes platforms are you running on?

EKS Amazon

Steps to reproduce

I updated my nginx-ingress-controller in eks from Version 3.6.2 to 3.7.0. The eks-cluster is provisioned with terraform helm-provider. During Update process I got the following error-message:

Error: Unable to continue with update: Lease "internal-ingress-controller-nginx-ingress-leader-election" in namespace
 "ingress-internal" exists and cannot be imported into the current release: invalid ownership metadata; label validation
 error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key 
 "meta.helm.sh/release-name": must be set to "internal-ingress-controller"; annotation validation error: missing key 
 "meta.helm.sh/release-namespace": must be set to "ingress-internal"
│ 
│   with module.cluster.module.nginx_ingress_controller_internal[0].helm_release.nginx_ingress_controller_helm,
│   on ../../modules/eks/nginx-ingress-controller/main.tf line 104, in resource "helm_release" "nginx_ingress_controller_helm":
│  104: resource "helm_release" "nginx_ingress_controller_helm" {
│ 
╵ 

The describe on my lease resource in my cluster looks like this:

~ kubectl -n ingress-internal describe lease/internal-ingress-controller-nginx-ingress-leader-election
Name:         internal-ingress-controller-nginx-ingress-leader-election
Namespace:    ingress-internal
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2022-11-08T11:03:42Z
  Resource Version:    185034718
  UID:                 10111f40-61f1-41d2-894e-c47b4fb92841
Spec:
  Acquire Time:            2023-12-04T10:06:03.514514Z
  Holder Identity:         internal-ingress-controller-nginx-ingress-nginx-ingress-cont2lnm5
  Lease Duration Seconds:  30
  Lease Transitions:       25
  Renew Time:              2024-01-30T13:46:25.679644Z
Events:                    <none>

Did I miss some new configuration for my cluster? For now, I fixed it, by manually adding the missing annotations and label to my existing lease-resource befor updating to new nginx-ingress-version.

[github-actions[bot]]

Hi @thomas-schn thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

[vepatel]

hi @thomas-schn , from release 3.7.0, leases are managed by helm and will be prefixed by helm release name (unless overridden using flag) so I wonder if this is because of conflict with older lease with the same name. Can you provide us with the output of kubectl get leases to check if there's one leftover form last release?

[thomas-schn]

hi @vepatel

There are two leases. I had to edit internal-ingress-controller-nginx-ingress-leader-election manualy.

גּ kubectl -n ingress-internal get leases
NAME                                                        HOLDER                                                            AGE
internal-ingress-controller-nginx-ingress-leader-election   internal-ingress-controller-nginx-ingress-nginx-ingress-cokrqqq   539d
nginx-ingress-leader                                        internal-ingress-controller-nginx-ingress-nginx-ingress-coz7pbf   259d

Here is the output of kubectl -n ingress-internal describe lease/nginx-ingress-leader

Name:         nginx-ingress-leader
Namespace:    ingress-internal
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2024-01-30T13:47:14Z
  Resource Version:    206270450
  UID:                 b2d55696-7bb6-436c-b2e3-95c360473a96
Spec:
  Acquire Time:            2024-09-10T09:28:32.347165Z
  Holder Identity:         internal-ingress-controller-nginx-ingress-nginx-ingress-coz7pbf
  Lease Duration Seconds:  30
  Lease Transitions:       14
  Renew Time:              2024-10-15T12:44:35.272016Z
Events:                    <none>

and this is after I edited the lease manualy and the update nginx-ingress to 3.7.0

~ kubectl -n ingress-internal describe lease/internal-ingress-controller-nginx-ingress-leader-election
Name:         internal-ingress-controller-nginx-ingress-leader-election
Namespace:    ingress-internal
Labels:       app.kubernetes.io/instance=internal-ingress-controller
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=nginx-ingress
              app.kubernetes.io/version=3.7.0
              helm.sh/chart=nginx-ingress-1.4.0
Annotations:  meta.helm.sh/release-name: internal-ingress-controller
              meta.helm.sh/release-namespace: ingress-internal
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2023-04-25T11:22:43Z
  Resource Version:    206357669
  UID:                 46bbc16e-ea8a-4cc5-b6f1-e6daf96bad7b
Spec:
  Acquire Time:            2024-10-15T12:45:20.790142Z
  Holder Identity:         internal-ingress-controller-nginx-ingress-nginx-ingress-cokrqqq
  Lease Duration Seconds:  30
  Lease Transitions:       18
  Renew Time:              2024-10-15T17:55:25.890838Z
Events:                    <none>

[vepatel]

okay looks like issue has nothing to do with the orphaned lease from last installation nginx-ingress-leader which can now be deleted. Can you please check if this issue only occurs when deploying helm via terraform but not with simple helm install commands?