Service controller doesn't populate TargetGroups

What happened:

I deployed the controller and configured NLB type Service. The service was created in AWS with associated target group but the target group is empty

What you expected to happen:

I expected the service-lb-controller controller to populate the target group

How to reproduce it (as minimally and precisely as possible):

I deployed the controller using manifest generated like this:

cat << EOF | helm template --values=- aws-cloud-controller-manager aws-cloud-controller-manager/aws-cloud-controller-manager>aws-cloud-controller.yaml
args:
  - --v=2
  - --cloud-provider=aws
  - --cluster-name=mycluster
  - --controllers=service-lb-controller,cloud-node
  - --allocate-node-cidrs=false
  - --configure-cloud-routes=false

image:
    repository: registry.k8s.io/provider-aws/cloud-controller-manager
    tag: v1.30.0
EOF

And used IAM policy from the docs. I then created the Service object like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: test-service
  namespace: mynamespace
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Local
  healthCheckNodePort: 30309
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    nodePort: 31199
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: myapp
  sessionAffinity: None
  type: LoadBalancer

Once applied ELB was created together with health checks and TargetGroups. However target groups are empty. I also noticed that security group entries were not added.

Anything else we need to know?:

This used to work with in-tree controller. We disabled the in-tree and moved to external and the Service controller no longer works in the same cluster.

Logs show successful calls to retrieve node details from AWS, for example:

I0516 09:28:28.565710       1 log_handler.go:37] AWS API ValidateResponse: ec2 DescribeInstances &{DescribeInstances POST / 0xc0004fee60 <nil>} {
  InstanceIds: ["i-xyz"]
} 200 OK

I also confirmed with CloudTrail that there are no permission error with API calls.

If I add nodes manually to the Target group they are removed from the Target group.

Environment:

Kubernetes version (use kubectl version):

Client Version: v1.28.9
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS
Kernel (e.g. uname -a): 5.15.0-1058-aws #64~20.04.1-Ubuntu SMP
Install tools: kubeadm
Others:

/kind bug

kubernetes / cloud-provider-aws

Service controller doesn't populate TargetGroups #915