Changing to IP mode breaks

[olespagnon-vdm]

Bug Description

When changing from alb.ingress.kubernetes.io/target-type: instance to alb.ingress.kubernetes.io/target-type : ip Load balancer fails to change the targetgroup and I get this error: {"level":"error","ts":"2025-03-20T13:04:05Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-mediaspo-servicea-80a47aec1e","namespace":"mynamespace"},"namespace":"mynamespace","name":"k8s-mediaspo-servicea-80a47aec1e","reconcileID":"f931fc53-3c5d-4f1d-a62d-44f589c0d7f2","error":"service type must be either 'NodePort' or 'LoadBalancer': mynamespace/service-api-accessservice"}

Which is weird because some of the api were properly upudated and work as intended. While some don't. In ArgoCD which I use to deploy I get this error on the ingress resource : Failed deploy model due to timed out waiting for the condition

Steps to Reproduce

• Step-by-step guide to reproduce the bug: Change target mode from instance to ip.
• Manifests applied while reproducing the issue:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
  annotations:
  alb.ingress.kubernetes.io/actions.service-api-logservice: >-
    {"Type":"redirect","RedirectConfig":{"Path":"/log-service/api/","Port":"443","Protocol":"HTTPS","Query":"#{query}","StatusCode":"HTTP_301"}}
  alb.ingress.kubernetes.io/actions.ssl-redirect: >-
    {"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port":
    "443", "StatusCode": "HTTP_301"}}
  alb.ingress.kubernetes.io/group.name: service-alb-company
  alb.ingress.kubernetes.io/healthcheck-interval-seconds: '5'
  alb.ingress.kubernetes.io/healthcheck-path: /log-service/api/index.html
  alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '2'
  alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
  alb.ingress.kubernetes.io/scheme: internet-facing
  alb.ingress.kubernetes.io/ssl-redirect: '443'
  alb.ingress.kubernetes.io/success-codes: '200'
  alb.ingress.kubernetes.io/target-group-attributes: load_balancing.algorithm.type=least_outstanding_requests
  alb.ingress.kubernetes.io/target-type: ip
  argocd.argoproj.io/tracking-id: >-
    helm-api-logservice:networking.k8s.io/Ingress:company/ingress-api-logservice
  kubernetes.io/ingress.class: alb
  service.beta.kubernetes.io/aws-load-balancer-name: service-alb-company
  creationTimestamp: '2023-10-03T10:54:58Z'
  finalizers:
  - group.ingress.k8s.aws/service-alb-company
  generation: 2
  labels:
  app.kubernetes.io/instance: testing
  app.kubernetes.io/managed-by: Helm
  app.kubernetes.io/name: helm-company
  app.kubernetes.io/version: 1.16.0
  argocd.argoproj.io/instance: helm-api-logservice
  helm.sh/chart: helm-company-0.1.0
  name: ingress-api-logservice
  namespace: mynamespace
  resourceVersion: '526357625'
  uid: dabb1e4a-2a00-4b12-8552-8beeee6ea855
  spec:
  rules:
  - host: api-test.host.io
    http:
      paths:
        - backend:
            service:
              name: service-api-logservice
              port:
                number: 9511
          path: /log-service/api/
          pathType: Prefix

• Controller logs/error messages while reproducing the issue: {"level":"error","ts":"2025-03-20T13:04:05Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"k8s-mediaspo-servicea-80a47aec1e","namespace":"mynamespace"},"namespace":"mynamespace","name":"k8s-mediaspo-servicea-80a47aec1e","reconcileID":"f931fc53-3c5d-4f1d-a62d-44f589c0d7f2","error":"service type must be either 'NodePort' or 'LoadBalancer': mynamespace/service-api-accessservice"}

Expected Behavior

Ingress are valid and TargetGroups are updated properly Actual Behavior

Regression Was the functionality working correctly in a previous version ? [Yes / No] If yes, specify the last version where it worked as expected

Current Workarounds

Environment

• AWS Load Balancer controller version: v2.8.2
• Kubernetes version: 1.30
• Using EKS (yes/no), if so version?: eks.27
• Using Service or Ingress: Ingress
• AWS region: eu-west-3
• How was the aws-load-balancer-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i aws-load-balancer-controller
  • aws-load-balancer-controller kube-system 20 2025-03-20 12:51:44.894449318 +0100 CET deployed aws-load-balancer-controller-1.6.1 v2.6.1
  • If helm was used then please show output of helm -n <controllernamespace> get values <helmreleasename>

    Name:                   aws-load-balancer-controller
    Namespace:              kube-system
    CreationTimestamp:      Mon, 02 Oct 2023 20:53:08 +0200
    Labels:                 app.kubernetes.io/instance=aws-load-balancer-controller
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=aws-load-balancer-controller
                    app.kubernetes.io/version=v2.6.1
                    helm.sh/chart=aws-load-balancer-controller-1.6.1
    Annotations:            deployment.kubernetes.io/revision: 16
                    meta.helm.sh/release-name: aws-load-balancer-controller
                    meta.helm.sh/release-namespace: kube-system
    Selector:               app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller
    Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
    StrategyType:           RollingUpdate
    MinReadySeconds:        0
    RollingUpdateStrategy:  25% max unavailable, 25% max surge
    Pod Template:
    Labels:           app.kubernetes.io/instance=aws-load-balancer-controller
                app.kubernetes.io/name=aws-load-balancer-controller
    Annotations:      kubectl.kubernetes.io/restartedAt: 2024-10-01T10:32:36+02:00
                prometheus.io/port: 8080
                prometheus.io/scrape: true
    Service Account:  aws-load-balancer-controller
    Containers:
    aws-load-balancer-controller:
    Image:       602401143452.dkr.ecr.eu-west-3.amazonaws.com/amazon/aws-load-balancer-controller:v2.8.2
    Ports:       9443/TCP, 8080/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
    --cluster-name=EKScluster
    --ingress-class=alb
    --aws-region=eu-west-3
    --aws-vpc-id=vpc-0307f743efe855b0a
    Liveness:     http-get http://:61779/healthz delay=30s timeout=10s period=10s #success=1 #failure=2
    Environment:  <none>
    Mounts:
    /tmp/k8s-webhook-server/serving-certs from cert (ro)
    Volumes:
    cert:
    Type:               Secret (a volume populated by a Secret)
    SecretName:         aws-load-balancer-tls
    Optional:           false
    Priority Class Name:  system-cluster-critical
    Node-Selectors:       <none>
    Tolerations:          dedicated=kube-system:NoSchedule
    Conditions:
    Type           Status  Reason
    ----           ------  ------
    Progressing    True    NewReplicaSetAvailable
    Available      True    MinimumReplicasAvailable
    OldReplicaSets:  aws-load-balancer-controller-d75c5b56d (0/0 replicas created), aws-load-balancer-controller-6b56994c5 (0/0 replicas created), aws-load-balancer-controller-59598f7745 (0/0 replicas created), aws-load-balancer-controller-565cbf9bdf (0/0 replicas created), aws-load-balancer-controller-66f6c679b7 (0/0 replicas created), aws-load-balancer-controller-585c4fd77 (0/0 replicas created), aws-load-balancer-controller-756cbf5fd9 (0/0 replicas created), aws-load-balancer-controller-7c68c6d77 (0/0 replicas created), aws-load-balancer-controller-9556d967d (0/0 replicas created), aws-load-balancer-controller-676b8fb796 (0/0 replicas created)
    NewReplicaSet:   aws-load-balancer-controller-84f5c87869 (2/2 replicas created)
    Events:          <none>

• Current state of the Ingress/Service configuration:
  • kubectl describe ingressclasses

    Name:         alb
    Labels:       app.kubernetes.io/instance=aws-load-balancer-controller
          app.kubernetes.io/managed-by=Helm
          app.kubernetes.io/name=aws-load-balancer-controller
          app.kubernetes.io/version=v2.6.1
          helm.sh/chart=aws-load-balancer-controller-1.6.1
    Annotations:  meta.helm.sh/release-name: aws-load-balancer-controller
          meta.helm.sh/release-namespace: kube-system
    Controller:   ingress.k8s.aws/alb
    Events:       <none>

  • kubectl -n <appnamespace> describe ingress <ingressname>

[olespagnon-vdm]

Okay so uppon further investigating I found something weird. The new target groups are working as intended but the old target groups remains. Should'nt they be removed once the ingress is updated ? Why is the ALB LC still trying to validate the ingress with the old targets ?

[zac-nixon]

Hello, please fill out this part of the template so we can properly help you.

AWS Load Balancer controller version:
Kubernetes version:
Using EKS (yes/no), if so version?:
Using Service or Ingress:
AWS region:

specifically the AWS Load Balancer controller version and Kubernetes version.

[olespagnon-vdm]

My bad forgot about this. I have edited my original post.

[zac-nixon]

Error is coming from here, https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/pkg/backend/endpoint_resolver.go#L87

I wonder if it's because all you did was change the target type while using the same service. Instance and IP targets use different service types: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.12/guide/ingress/annotations/#target-type

[olespagnon-vdm]

Hi, thanks for the response. You might be right. I'm using a helm template, inflated with ArgoCD and deployed by ArgoCD. But upon further inspection, it seems that indeed the modification of the Service type did not recreate a new service bu changed the existing ones. What do you suggest I me doing ? Should I delete the TargetGroupBinding in the cluster and the aws resources would be deleted ? Should I delete both ? Or should I wait for a 'fix' for this case ?

[zac-nixon]

I think this scenario would fall under not supported behavior, moving to instance -> ip requires a service update and corresponding target group update. If you want to clean up the target group, I would suggest deleting the unused binding and deleting the target group

[olespagnon-vdm]

I've now deleted remaining Instance type TargetGroupBinding and their associated TargetGroups. But now I see an error on the ingresses that ALB Controller monitor for infra. Warning FailedDeployModel 4m14s (x621 over 7d1h) ingress Failed deploy model due to timed out waiting for the condition Is there a way to have more detailed info on why this is failing ? I have a log saying "level": "info", "ts": "2025-03-27T12:43:13Z", "logger": "controllers.ingress", "msg": "successfully built model", "model" :"very long json string of my model"

[olespagnon-vdm]

I think that maybe this is due to my CRDs. But since I'm using helm via Terraform I cannot reinstall them. Is there a way to install CRDs at a specified version instead of master ref ?