Error when creating Ingress: NoCredentialProviders: no valid providers in chain

Describe the bug When trying to create Kubernetes Ingress with AWS Load Balancer Controller, no Elastic Load Balancer is deployed. Error logs (retrieved by running kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller): logs.txt

Steps to reproduce I'm trying to create an Application Load Balancer on intranet subnets (subnets with transitive gateway to corporate network). These subnets do not have any internet access. Besides these, there are also private subnets within the VPC.

I have marked all intranet subnets with kubernetes.io/role/internal-elb=1 and kubernetes.io/cluster/eks-ilz=owned Private subnets have no such tags.

For Kubernetes I am using EKS, provisioned by official terraform provider resources only. For the cluster I have one t3-large node instance.

For addons on the EKS I have:

Amazon VPC CNI
kube-proxy
CoreDNS
Amazon EKS Pod Identity Agent

To my understanding, EKS Pod Identity Agent addon is responsible with linking the AWS IAM Role with the Kubernetes Service account. The policy used within the AWS Load Balancer Controller IAM Role is the same as the one found at https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.1/docs/install/iam_policy.json (as per the official documentation - https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.8/deploy/installation/)

I did not try to configure IRSA, kube2iam or EKS OIDC providers.

I have forked public.ecr.aws/eks/aws-load-balancer-controller:v2.8.1 and added to my private ecr repository so the kubernetes pods can reach it.

I install AWS Load Balancer Controller via helm install:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=eks-ilz --set serviceAccount.name=aws-load-balancer-controller --set region=eu-central-1 --set vpcId=<vpcId> --set image.repository=<accountId>.dkr.ecr.eu-central-1.amazonaws.com/aws-load-balancer-controller --set image.tag=v2.8.1

I try to setup a Deployment, LoadBalancer and Ingress with these Kubernetes manifests:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ilz-lbc
  namespace: ilz-lbc
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /q/health
spec:
  ingressClassName: alb
  rules:
    - host: incident-manager.ilz-test.aws.cloud.bmw
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ilz-lbc
                port:
                  number: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: ilz-lbc
  namespace: ilz-lbc
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: ilz-lbc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ilz-lbc
  namespace: ilz-lbc
spec:
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: ilz-lbc
  template:
    metadata:
      labels:
        app: ilz-lbc
    spec:
      containers:
      - image: <accountId>.dkr.ecr.eu-central-1.amazonaws.com/<imageName>:<imageTag>
        name: ilz-lbc
        ports:
        - containerPort: 8080

Expected outcome An Application Load Balancer should be deployed pointing to the kubernetes ingress. However, there is no such deployment available:

❯ kubectl describe ingress ilz-lbc -n ilz-lbc
Name:             ilz-lbc
Labels:           <none>
Namespace:        ilz-lbc
Address:
Ingress Class:    alb
Default backend:  <default>
Rules:
  Host                                     Path  Backends
  ----                                     ----  --------
  incident-manager.ilz-test.aws.cloud.bmw
                                           /   ilz-lbc:8080 (10.19.0.38:8080)
Annotations:                               alb.ingress.kubernetes.io/healthcheck-path: /q/health
                                           alb.ingress.kubernetes.io/scheme: internal
                                           alb.ingress.kubernetes.io/target-type: ip
Events:
  Type     Reason            Age                    From     Message
  ----     ------            ----                   ----     -------
  Warning  FailedBuildModel  2m4s (x16 over 5m16s)  ingress  Failed build model due to NoCredentialProviders: no valid providers in chain. Deprecated.
           For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Environment

AWS Load Balancer controller version: 2.8.1
Kubernetes version: 1.30
Using EKS (yes/no), if so version? yes, not sure how to find it out: I'm using the terraform aws provider hashicorp/aws version ~> 1.7

Additional Context: Helm chart used to install AWS Load Balancer Controller version: 1.8.1

kubernetes-sigs / aws-load-balancer-controller

Error when creating Ingress: NoCredentialProviders: no valid providers in chain #3779