504 Gateway Time-out error

[mart3051]

We have two VPCs in our system. There is a VPC peering in between them. We are also having two ingress controllers . Everything is ok with first ingress controller which is referencing subnets of same VPC where the EKS is placed.

For the 2nd ingress controller we are pointing to the 2nd VPC .
New Ingress is pointing to public subnets of 2nd VPC and ingress-class of second ingress controller.
Below is the the ingress controller yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: second-ingress-controller-leader-alb
  namespace: kube-system
---
# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller-s
  name: alb-ingress-controller-s
  # Namespace the ALB Ingress Controller should run in. Does not impact which
  # namespaces it's able to resolve ingress resource for. For limiting ingress
  # namespace scope, see --watch-namespace.
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: alb-ingress-controller-s
  template:
    metadata:
      labels:
        app.kubernetes.io/name: alb-ingress-controller-s
    spec:
      containers:
        - name: alb-ingress-controller
          args:
            # Limit the namespace where this ALB Ingress Controller deployment will
            # resolve ingress resources. If left commented, all namespaces are used.
            # - --watch-namespace=your-k8s-namespace

            # Setting the ingress-class flag below ensures that only ingress resources with the
            # annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
            # choose any class you'd like for this controller to respect.
            - --ingress-class=albsc
            - --election-id=second-ingress-controller-leader-alb

            # REQUIRED
            # Name of your cluster. Used when naming resources created
            # by the ALB Ingress Controller, providing distinction between
            # clusters.
            - --cluster-name=******

            # AWS VPC ID this ingress controller will use to create AWS resources.
            # If unspecified, it will be discovered from ec2metadata.
            - --aws-vpc-id=*******
            # AWS region this ingress controller will operate in.
            # If unspecified, it will be discovered from ec2metadata.
            # List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
            - --aws-region=#####
            - --target-type=ip

            # Enables logging on all outbound requests sent to the AWS API.
            # If logging is desired, set to true.
            # - --aws-api-debug
            # Maximum number of times to retry the aws calls.
            # defaults to 10.
            # - --aws-max-retries=10
          # Repository location of the ALB Ingress Controller.
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.6
      serviceAccountName: alb-ingress-controller

ingress.yml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
 annotations:
    alb.ingress.kubernetes.io/certificate-arn: *******
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/waf-acl-id: ******
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/load-balancer-attributes: deletion_protection.enabled=true
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=60
    external-dns.alpha.kubernetes.io/hostname: someHost
    alb.ingress.kubernetes.io/subnets: public-subnetsof-VPC-2-AZ1, public-subnetsof-VPC-2-AZ2, public-subnetsof-VPC-2-AZ3
    kubernetes.io/ingress.class: albsc

 name: somename
spec:
 rules:
   - host: someHost
     http:
       paths:
         - path: /*
           backend:
             serviceName: ssl-redirect
             servicePort: use-annotation
         - path: /*
           backend:
             serviceName: service-name
             servicePort: 3000

In logs we see the successful reconciliation message

I0527 12:53:41.377957       1 controller.go:236] kubebuilder/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="alb-ingress-controller" "request"={"Namespace":"namesspace","Name":"service-name"}
I0527 12:53:41.780673       1 controller.go:236] kubebuilder/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="alb-ingress-controller" "request"={"Namespace":"namesspace","Name":"service-name"}
I0527 12:53:43.163216       1 controller.go:236] kubebuilder/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="alb-ingress-controller" "request"={"Namespace":"namesspace","Name":"service-name"}
I0527 12:54:19.423774       1 targets.go:93] namesspace/service-name": Adding targets to arn:aws:elasticloadbalancing:region:xxxxxx:targetgroup/**********: 10.34.4.178:3000
I0527 12:54:19.871125       1 controller.go:236] kubebuilder/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="alb-ingress-controller" "request"={"Namespace":"namesspace","Name":"service-name"}
I0527 12:54:27.590730       1 targets.go:93]  namesspace/service-name: Adding targets to arn:aws:elasticloadbalancing:region:xxxxxx:targetgroup********: 10.34.4.29:3000
I0527 12:54:27.960339       1 controller.go:236] kubebuilder/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="alb-ingress-controller" "request"={"Namespace":"namesspace","Name":"service-name"}

The ALBs are also created but when we try to reach the application we receive 504 error.
We have tried increasing timeout but that has not helped either.
On some occasions deletion of ingress controller and ingress had helped but it seems to have no impact now.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[yoonkwon]

I had similar issue. In my case target group was all unhealthy, but health check traffic logs in pod were OK with 200. Then the problem must have been about network acl. After doing some research, I realized that I didn't add cluster security group to loadbalancer. I added cluster security group to alb.ingress.kubernetes.io/security-groups, target group became healthy, and all worked!

[albertollamaso]

I had same issue and by doing what @yoonkwon commented above solved my issue.

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1276#issuecomment-731868114): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[DanielLund]

@yoonkwon What security groups did you tag, and did you add this annotation to the ingress.yaml file? I've added them to mine deleted the pods and it still isn;t passing health checks