Open countergram opened 7 years ago
joshrosso
commented
7 years ago @countergram Thanks, we've heard this request and similar a few times now.
Seems a feature that many would like is an ability to explicitly call our a named ALB via annotation (or eventually configmap).
markbooch
commented
5 years ago Is there any updates for this issues?
marcosdiez
commented
5 years ago Hey, I might have solved your problem in this PR: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/830 . Testing and feedback is welcome :)
benderillo
commented
5 years ago @joshrosso Is there any update on this request?
tdmalone
commented
5 years ago 
fejta-bot
commented
5 years ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
npolagani
commented
5 years ago I was able to use one ALB for multiple Ingress Resources in Version 1.0.1. I did create a new cluster and installed Version 1.1.2, which is creating a new ALB for each Ingress Resource. Is there anyway that I can use same ALB in 1.1.2 ?
tdmalone
commented
5 years ago fejta-bot
commented
5 years ago Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten
fejta-bot
commented
5 years ago Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close
k8s-ci-robot
commented
5 years ago @fejta-bot: Closing this issue.
k8s-ci-robot
commented
4 years ago @leoskyrocker: You can't reopen an issue/PR unless you authored it or you are a collaborator.
gkrizek
commented
4 years ago @M00nF1sh @joshrosso Any update here? I think this issue is still relevant today and yet unsolved.
Having a way for Kubernetes to attach to existing ALBs/Target Groups is incredibly valuable for lots of reasons. Really surprised there's no way to do it right now.
M00nF1sh
commented
4 years ago @gkrizek
we'll address this issue in V2.
For attach to existing TargetGroups, we'll expose an CRD called endpointBinding to allow do that.
For attach to existing ALB, we haven't decided whether to use an annotation on Ingress(like alb-arn: xxxx) or an AWS tag on the ALB(like ownership: shared, ingress: ingress-name, cluster: cluster-name). any opinions?
gkrizek
commented
4 years ago @M00nF1sh Great to hear! I hate to be "that guy" but is there a v2 anticipated release date?
I like the CRD idea for target groups, I think that's the right direction. I think both options are valid for the ALB, however I think tags are preferred. Because with an ARN you might have to do some wacky stuff to get the ARN into a manifest/helm chart. With tags it'd be pretty easy to define without needed explicit values from AWS. Also it would allow for an ingress to attach to multiple ALBs if one chooses.
M00nF1sh
commented
4 years ago @gkrizek There is no anticipated released date yet( i cannot promise one), but I'll keep update https://github.com/kubernetes-sigs/aws-alb-ingress-controller/projects/1 whenever i got time to work on it 🤣. BTW, there is an alpha version of V2 which works just fine: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v1.2.0-alpha.1 (you can reuse an ALB by apply correct tags, however, the controller will try to delete the ALB once we delete the ingress)
rifelpet
commented
4 years ago having a single ingress: tag forces a many:1 relationship of ingresses to ALBs which contradicts the concept of ingress grouping mentioned in other issues. It'd be great if the design could allow many ALBs to be reused by many ingresses. Perhaps a "binding" CRD similar to the endpoint/targetgroup solution mentioned above?
gkrizek
commented
4 years ago @M00nF1sh I figured so 😉 . Sounds good, I'll check out the alpha for now. Thanks for the help.
M00nF1sh
commented
4 years ago @rifelpet
it's actually an ingress.k8s.aws/stack: <value> annotation on ALB in V2, where the
(However, personally i favor to require an explicit annotation of ..../alb-arn:xxxx on one-of-ingresses among group to denote the reuse, since tagging on ALB requires to plan for Ingress before hand)
What do you mean by allow many ALBs to be reused by many ingresses.? Current design is one group will only have one ALB.
It's possible to extend it to be like one group with multiple ALB(like auto-split rules), but is there really a use case for this? since i assume there are app-specific dependencies like some Ingress must be hosted by a single DNS name, so it's impossible for the controller to make the split decision if rule exceeds ALB's limits, instead it's better for the user to split there ingresses into different groups.
rifelpet
commented
4 years ago ok, my main concern was supporting a 1-1 relation between a group and ALB and it sounds like the tags can achieve that 👍
tomhaynes
commented
4 years ago Hi @M00nF1sh keen to start testing with the v1.2.0-alpha.1 version - can you point me to what tags are needed in order to reuse an existing ALB?
BTW, there is an alpha version of V2 which works just fine: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v1.2.0-alpha.1 (you can reuse an ALB by apply correct tags, however, the controller will try to delete the ALB once we delete the ingress)
jainishshah17
commented
4 years ago What is ETA for this?
M00nF1sh
commented
3 years ago /reopen This is something in our roadmap, we'll do designs around this and have a better ETA after v2.2.0 is released. in v2.2.0, we'll add support for tagging listeners and listener rules, which helps to implement this feature.
jwenz723
commented
3 years ago @M00nF1sh now that v2.2.0 has been released, can you provide any updates on this feature?
Erokos
commented
3 years ago Hi, can anyone please clarify a bit if my understanding is correct - Currently we have to create an ALB, listeners and target groups externally, e.g. Terraform, and then within the cluster, reference the target group arn in a target group binding right? The listener rules we don't have to create externally because we would manage it using ingress right? I appreciate any info on this subject because these links I found in a similar issue aren't working: https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/targetgroupbinding/targetgroupbinding/
jwenz723
commented
3 years ago It is possible to create only the ALB with terraform and then create the target groups and listeners with the aws-load-balancer-controller, but it isn't yet officially supported. I believe there is planned support for this in v2.3. Basically the aws-load-balancer-controller will automatically update any ALB which has the appropriate AWS tags. So if you set the tags on your ALB in Terraform then the aws-load-balancer-controller will take ownership of updating it.
I believe the necessary tags are:
"ingress.k8s.aws/stack" = "The value specified in the alb.ingress.kubernetes.io/group.name annotation"
"ingress.k8s.aws/resource" = "LoadBalancer"
"elbv2.k8s.aws/cluster" = "The name of your k8s cluster as specified in the aws-load-balancer-controller config"You can validate if the tags I mentioned above are correct by configuring your aws-load-balancer-controller instance to provision an ALB then inspect the AWS tags on the ALB after it has been provisioned.
Erokos
commented
3 years ago Thank you very much for the explanation, I'll try it out immediately.
christophebeling
commented
3 years ago @jwenz723 this is actually not working for me. If I set the tags you mentioned, my alb gets deleted, and then a new ALB is being created by the controller.
Erokos
commented
3 years ago "@christophebeling I didn't observe this behaviour. @jwenz723 To successfully manage it I need to use the target group binding as well as an ingress describing the rules right?"
To fill in the details, using the tags on the ALB,created by Terraform, didn't work for me. First I created the ALB and a target group with a listener using terraform and then, after the controller installation, tried to use the targetgroupbinding without an ingress created and then the target group doesn't see the instances anymore. Using an ingress just creates another load balancer and ignores the first.
I also tried to create just the ALB without any target groups and listeners and then used an ingress. This replaces the ALB created by terraform with the one created by the controller.
Sorry for the confusing comment in the beginning.
Erokos
commented
3 years ago Update: I've managed to get the ALB, created by terraform, to work with the load balancer controller. First step is to create the ALB along with a target group and a listener. Second is to deploy the controller and a targetgroup binding in which the target group arn is referenced. The only thing is I don't think any ingress is working in this case and everything should be defined in terraform, but I need to experiment further on that.
kuroneko25
commented
3 years ago Would a similar mechanism exist for NLBs?
kishorj
commented
3 years ago @kuroneko25, you can use targetgroupbindings for NLB as well.
bgaillard
commented
3 years ago Hi, thanks for all the details in your comments @jwenz723 @Erokos @christophebeling, they helped me making my setup work correctly.
However I struggled a little so I think complete explanations can help others, here is how I managed to make it work in my case (after this if i'm right about the setup IMO it could be great to create a PR to add more details in the documentation).
Here are the steps I followed to make the setup work with an ALB (in Terraform but can be adapted to other provisioning tool I believe).
resource "aws_alb" "this" {
name = "sample"
internal = false
load_balancer_type = "application"
subnets = module.vpc.public_subnets
enable_deletion_protection = true
security_groups = [aws_security_group.alb.id]
}
resource "aws_alb_listener" "this" {
load_balancer_arn = aws_alb.this.arn
port = "443"
protocol = "HTTPS"
certificate_arn = aws_acm_certificate.sample_com.arn
default_action {
type = "fixed-response"
fixed_response {
content_type = "text/plain"
message_body = "Sample"
status_code = "200"
}
}
}serviceRef. In this sample ${local.kubectl} contains a specific value which allows me to apply any Kubernetes file.locals {
# Trick to perform 'kubectl' calls for operations not supported by the Terraform kubernetes provider
kubectl = <<EOH
cat >/tmp/ca.crt <<EOF
${base64decode(aws_eks_cluster.this.certificate_authority.0.data)}
EOF
curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.17.9/2020-08-04/bin/linux/amd64/aws-iam-authenticator && chmod +x ./aws-iam-authenticator && \
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && \
mkdir -p /tmp/bin && mv ./aws-iam-authenticator /tmp/bin/ && export PATH=$PATH:/tmp/bin && \
./kubectl \
--server="${aws_eks_cluster.this.endpoint}" \
--certificate-authority=/tmp/ca.crt \
--token="${aws_eks_cluster_auth.this.token}" \
EOH
}
resource "aws_alb_target_group" "this" {
name = "sample"
port = 80
protocol = "HTTP"
target_type = "ip"
vpc_id = module.vpc.vpc_id
provisioner "local-exec" {
command = <<CMD
cat <<TGB >> /tmp/target-group-binding-sample.yaml
apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
name: ${self.name}
spec:
serviceRef:
name: ${self.name}-sample
port: 80
targetGroupARN: ${self.arn}
TGB
CMD
when = create
}
provisioner "local-exec" {
command = "${local.kubectl} apply -f /tmp/target-group-binding-sample.yaml"
when = create
}
}serviceRef before. In my case I used Helm to install Wordpress. Here note that I used no specific Ingress configuration, we do not want any Ingress configuration here because everything is configured through standard Terraform AWS resources + the TargetGroupBinding. After apply of this resource a new target should be automatically registered in the previously created ALB Target Group.resource "helm_release" "this" {
max_history = 3
name = "sample"
repository = "https://charts.bitnami.com/bitnami"
chart = "wordpress"
version = "12.1.16"
values = [
<<-EOT
wordpressPassword: "${data.aws_ssm_parameter.wordpress_password.value}"
service:
type: NodePort
readinessProbe:
enabled: true
mariadb:
enabled: false
primary:
persistence:
enabled: false
externalDatabase:
host: ${aws_rds_cluster.this.endpoint}
port: ${aws_rds_cluster.this.port}
user: ${aws_rds_cluster.this.master_username}
password: "${data.aws_ssm_parameter.wordpress_password.value}"
database: "${aws_rds_cluster.this.database_name}"
persistence:
enabled: false
extraEnvVars:
- name: LOG_LEVEL
value: DEBUG
- name: BITNAMI_DEBUG
value: "true"
EOT
]
}data "aws_security_group" "eks_node" {
tags = {
"aws:eks:cluster-name" = "sample"
}
}
resource "aws_security_group_rule" "eks_node_alb_ingress" {
type = "ingress"
from_port = 0
to_port = 65535
protocol = "tcp"
security_group_id = data.aws_security_group.eks_node.id
source_security_group_id = aws_security_group.alb.id
depends_on = [
aws_eks_node_group.this
]
}
resource "aws_lb_listener_rule" "this" {
listener_arn = aws_alb_listener.this.arn
priority = 100
action {
type = "forward"
target_group_arn = aws_alb_target_group.this.arn
}
condition {
host_header {
values = [
"www.sample.com"
]
}
}
}resource "aws_route53_record" "this" {
zone_id = aws_route53_zone.sample_com.zone_id
name = "www.sample.com"
type = "A"
alias {
name = aws_alb.this.dns_name
zone_id = aws_alb.this.zone_id
evaluate_target_health = false
}
}This configuration should make the application available at https://www.sample.com.
Hope this helps.
levya4
commented
2 years ago Hey, can you provide any updates on this feature? I struggled with the same issue while trying to automate the ALB controller together with AWS Global Accelerator. BTW, support for AWS Global Accelerator will be much better in my case :)
Bhashit
commented
2 years ago I believe there is planned support for this in v2.3.
Looks like version 2.3 is out.
blevine
commented
2 years ago Congratulations @bgaillard on figuring this out. IMO, if that level of complexity is required to make use of this feature, I'll forgo managing the ALB in Terraform and continue to create the ALB from my Helm charts. Once you have the Terraform boilerplate in place to setup for using the load-balancer controller, the rest is relatively easy.
Erokos
commented
2 years ago To me it comes down to a separation of concerns. I don't want my load balancer terminated as I terminate my cluster, because I could be having target groups pointing to servers not managed by kubernetes or to services like lambda functions. I don't think that creating a load balancer through kubernetes manifests is a good practice. Infrastructure components should be created in the same way to maintain them easily and everything that happens on the app lvl inside a cluster, like routing requests to services, should be done by kubernetes, i.e. ingress rules. That's one (among a few other unrelated requirements) of the reasons I decided to go with a NLB and nginx ingress controller that was much easier to set up and could use it in the way I described above.
aramkarapetian
commented
2 years ago hi @bgaillard, thank you for the note with instructions, it does work but with a small problem:
it does not create target group binding when target group port does not match service's internal port (e.g. 31045) which cannot actually be predicted but maybe hard coded when service is created.
Am I doing something wrong?
AntonAleksandrov13
commented
2 years ago Huge thanks for your guidance here, @bgaillard. I appreciate you posting code snippets! If I may add my five cents, here's what I did:
Similarly I have created an ALB with a listener:
resource "aws_alb" "this" {
name = "sample"
internal = false
load_balancer_type = "application"
subnets = module.vpc.public_subnets
enable_deletion_protection = true
security_groups = [aws_security_group.alb.id]
}
resource "aws_alb_listener" "this" {
load_balancer_arn = aws_alb.this.arn
port = "443"
protocol = "HTTPS"
certificate_arn = aws_acm_certificate.sample_com.arn
default_action {
type = "fixed-response"
fixed_response {
content_type = "text/plain"
message_body = "Sample"
status_code = "200"
}
}
}
resource "aws_lb_listener_rule" "this" {
listener_arn = aws_alb_listener.this.arn
priority = 100
action {
type = "forward"
target_group_arn = aws_alb_target_group.this.arn
}
condition {
host_header {
values = [
"www.sample.com"
]
}
}
}Instead of using local-provisioner(it gave me a lot of trouble during arn changes and destroys), I have used two resources:
resource "aws_alb_target_group" "this" {
name_prefix = "sample"
port = var.svc_port
protocol = "HTTP"
target_type = "ip"
vpc_id = module.vpc.vpc_id
health_check {
path = "/"
}
}
resource "kubernetes_manifest" "this" {
manifest = {
"apiVersion" = "elbv2.k8s.aws/v1beta1"
"kind" = "TargetGroupBinding"
"metadata" = {
"name" = aws_alb_target_group.this.name
"namespace" = var.ns
}
"spec" = {
"serviceRef" = {
"name" = "sample-svc" // you have to rely on locals/vars/hardcoded values, because it's hard to extract svc name
port = 5000
}
"targetGroupARN" = aws_alb_target_group.this.arn
"targetType" = "ip"
}
}
}
resource "kubernetes_service" "this" {
metadata {
name = "sample-svc"
namespace = local.ns
}
spec {
selector = {
app = kubernetes_deployment.this.metadata.0.labels.app
}
port {
port = var.svc_port
target_port = 80
}
type = "NodePort"
}
}And finally, the deployment:
resource "kubernetes_deployment" "this" {
metadata {
name = "sample"
namespace = var.ns
labels = {
app = "sample"
}
}
spec {
replicas = 1
selector {
match_labels = {
app = "sample"
}
}
template {
metadata {
labels = {
app = "sample"
}
}
spec {
container {
image = "nginx"
name = "webserver"
liveness_probe {
http_get {
path = "/"
port = 80
}
initial_delay_seconds = 3
period_seconds = 3
}
}
}
}
}
}Terraform version:
Terraform v0.15.3
on darwin_amd64
obvionaoe
commented
2 years ago Hi, any news on this? @M00nF1sh
aramkarapetian
commented
2 years ago @obvionaoe I discovered nice way solving this issue.
blevine
commented
2 years ago @aramkarapetian could you give more specifics about how you use an Ingress group annotation to accomplish this?
sysadmin-exe
commented
2 years ago @obvionaoe I discovered nice way solving this issue.
- you create certificate, load balancer and dns record using terraform.
- you use ingress annotation to get your ingress resource linked to one created by terraform https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/annotations/#group.name.
@aramkarapetian Please can you share how you achieved this with group.name annotation. The link shared does not give so much insight. Thanks.
blakepettersson
commented
2 years ago @jwenz723 this is actually not working for me. If I set the tags you mentioned, my alb gets deleted, and then a new ALB is being created by the controller.
This kind of works, once the elasticloadbalancing:CreateLoadBalancer and elasticloadbalancing:DeleteLoadBalancer permissions are removed from the service account role.
Creating an ingress works fine using the pattern above. What doesn't work is removal of the ingress which has been created in this way.
This is due to fact that the associated target group(s) and security group(s) does not get detached from the load balancer before the controller attempts to remove them (due to the fact that the logic assumes that the load balancer will be deleted). If the controller is modified to first detach the target group(s) from the load balancer, once its targets have been detached, before deleting them, and then in turn force detaching the security group(s) before attempting to delete them, then we'd at least be able to manage the LB itself with Terraform, and the controller can then manage the security groups and target groups.
blakepettersson
commented
2 years ago Example TF config:
// Dummy security group for initial config of the ALB. In practice the
// aws-load-balancer-controller will manage the config of the security groups
// for us.
resource "aws_security_group" "lb-test" {
name_prefix = "${local.iam_role_name_prefix}-lb"
vpc_id = local.vpc_id
}
resource "aws_lb" "test" {
name = "${local.cluster_name}-lb"
internal = true
load_balancer_type = "application"
subnets = var.subnets
security_groups = [aws_security_group.lb-test.id]
tags = {
"ingress.k8s.aws/stack" = "cluster"
"ingress.k8s.aws/resource" = "LoadBalancer"
"elbv2.k8s.aws/cluster" = local.cluster_name
}
lifecycle {
ignore_changes = [
// Let the aws-load-balancer-controller manage the security groups for us
security_groups
]
}
}
resource "aws_iam_policy" "lb-policy" {
name = "${local.iam_role_name_prefix}-aws-lb-controller"
path = "/"
description = "policy for AWS LB controller"
// This should probably be made more restrictive
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"iam:CreateServiceLinkedRole",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeInstances",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeTags",
"ec2:GetCoipPoolUsage",
"ec2:DescribeCoipPools",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeTags"
],
"Resource" : "*"
},
{
"Effect" : "Allow",
"Action" : [
"cognito-idp:DescribeUserPoolClient",
"acm:ListCertificates",
"acm:DescribeCertificate",
"iam:ListServerCertificates",
"iam:GetServerCertificate",
"waf-regional:GetWebACL",
"waf-regional:GetWebACLForResource",
"waf-regional:AssociateWebACL",
"waf-regional:DisassociateWebACL",
"wafv2:GetWebACL",
"wafv2:GetWebACLForResource",
"wafv2:AssociateWebACL",
"wafv2:DisassociateWebACL",
"shield:GetSubscriptionState",
"shield:DescribeProtection",
"shield:CreateProtection",
"shield:DeleteProtection"
],
"Resource" : "*"
},
{
"Effect" : "Allow",
"Action" : [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource" : "*"
},
{
"Effect" : "Allow",
"Action" : [
"ec2:CreateSecurityGroup"
],
"Resource" : "*"
},
{
"Effect" : "Allow",
"Action" : [
"ec2:CreateTags"
],
"Resource" : "arn:aws:ec2:*:*:security-group/*",
"Condition" : {
"StringEquals" : {
"ec2:CreateAction" : "CreateSecurityGroup"
},
"Null" : {
"aws:RequestTag/elbv2.k8s.aws/cluster" : "false"
}
}
},
{
"Effect" : "Allow",
"Action" : [
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource" : "arn:aws:ec2:*:*:security-group/*",
"Condition" : {
"Null" : {
"aws:RequestTag/elbv2.k8s.aws/cluster" : "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster" : "false"
}
}
},
{
"Effect" : "Allow",
"Action" : [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup"
],
"Resource" : "*",
"Condition" : {
"Null" : {
"aws:ResourceTag/elbv2.k8s.aws/cluster" : "false"
}
}
},
{
"Effect" : "Allow",
"Action" : [
//"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateTargetGroup"
],
"Resource" : "*",
"Condition" : {
"Null" : {
"aws:RequestTag/elbv2.k8s.aws/cluster" : "false"
}
}
},
{
"Effect" : "Allow",
"Action" : [
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:DeleteRule"
],
"Resource" : "*"
},
{
"Effect" : "Allow",
"Action" : [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:RemoveTags"
],
"Resource" : [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
// Ideally we should be able to prevent these resources from having their tags managed by the aws-lb-controller.
// Commenting these out prevents the ingress from coming up
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
],
"Condition" : {
"Null" : {
"aws:RequestTag/elbv2.k8s.aws/cluster" : "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster" : "false"
}
}
},
{
"Effect" : "Allow",
"Action" : [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:RemoveTags"
],
"Resource" : [
"arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
]
},
{
"Effect" : "Allow",
"Action" : [
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
//"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:DeleteTargetGroup"
],
"Resource" : "*",
"Condition" : {
"Null" : {
"aws:ResourceTag/elbv2.k8s.aws/cluster" : "false"
}
}
},
{
"Effect" : "Allow",
"Action" : [
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets"
],
"Resource" : "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
},
{
"Effect" : "Allow",
"Action" : [
"elasticloadbalancing:SetWebAcl",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates",
"elasticloadbalancing:ModifyRule"
],
"Resource" : "*"
}
]
})
}
module "load_balancer_controller" {
source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
version = "4.2.0"
create_role = true
role_name = "${local.iam_role_name_prefix}-aws-lb-controller"
provider_url = module.cluster.oidc_issuer_url
role_policy_arns = [
aws_iam_policy.lb-policy.arn,
]
oidc_fully_qualified_subjects = ["system:serviceaccount:kube-system:aws-load-balancer-controller"]
}
greenlaw
commented
2 years ago I was getting hung up with the TargetGroupBinding approach as well (defining the load balancer/listener outside of Kubernetes and linking to the k8s service via TargetGroupBinding), but @bgaillard 's note helped me identify that the missing piece was an aws_security_group_rule allowing ingress from the ALB to the cluster nodes.
I think this was mostly confusing because I am using ip target type to connect to the AWS CNI-provided VPC IP for the pod, rather than via NodePort, but I suppose the reason is because the Pod IP is coming from the ENI attached to the node, which is coupled to the security group.
Anyway, for anyone else who finds this, with EKS 1.2.2 and managed node groups, the security group tags might be different, so you might have to modify @bgaillard 's security group selector to something like this:
data "aws_security_group" "eks_node" {
tags = {
"kubernetes.io/cluster/<cluster_name>" = "owned"
}
}
adabuleanu
commented
2 years ago How are the workaround presented here
It is possible to create only the ALB with terraform and then create the target groups and listeners with the aws-load-balancer-controller, but it isn't yet officially supported. I believe there is planned support for this in v2.3. Basically the aws-load-balancer-controller will automatically update any ALB which has the appropriate AWS tags. So if you set the tags on your ALB in Terraform then the aws-load-balancer-controller will take ownership of updating it.
I believe the necessary tags are:
"ingress.k8s.aws/stack" = "The value specified in the alb.ingress.kubernetes.io/group.name annotation" "ingress.k8s.aws/resource" = "LoadBalancer" "elbv2.k8s.aws/cluster" = "The name of your k8s cluster as specified in the aws-load-balancer-controller config"You can validate if the tags I mentioned above are correct by configuring your aws-load-balancer-controller instance to provision an ALB then inspect the AWS tags on the ALB after it has been provisioned.
I got the solution working on version 2.4.0. So actual steps were to:
ingress.k8s.aws/stack, ingress.k8s.aws/resource, elbv2.k8s.aws/clusteralb.ingress.kubernetes.io/group.name in the Ingress resource to match the ingress.k8s.aws/stack tag in the ALB).My requirement was to use the ALB to balance traffic between two deployments.
The aws-lb-controller actually updated the existing ALB Listener with two rules for the same path ( / in my case ) pointing to ALB TargetGroups it created. This was not the desired result for me because all traffic went to the second TargetGroup (rule was set first in the Listener). I had to use the feature to set custom action on listeners and add an annotation in the second Ingress resource like :
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.<second-service-name>: >
{"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"<second-service-name>","servicePort":"http","weight":50},{"targetGroupARN":"<first-TargetGroupARN>","weight":50}],"targetGroupStickinessConfig":{"enabled":true,"durationSeconds":200}}}Note: first-TargetGroupARN is the ARN of the TargetGroup created by the aws-lb-controller for the first deployment
So, now I have two rules in the ALB Listener pointing to the same path, but the one that has precedence balances the traffic 50/50 between the two deployments.
My end goal is to use an existing ALB to balance traffic between deployments from multiple clusters (example above is working for deployments in the same cluster). I have noticed the ALB tag elbv2.k8s.aws/cluster pointd to the cluster name in the aws-lb-controller config. Will this work for multi-cluster setup? Is there another approach on achieving the same behavior?
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle stale/lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
marcosdiez
commented
1 year ago /remove-lifecycle stale
mfinkelstine
commented
1 year ago Hi,
Just wonder if this future is already available in the code or still needs to be merged
Thanks
As a user of Terraform (or, substitute CloudFormation), I would like to use an existing ALB with the ingress controller so that I can keep my infrastructure automation centralized rather than in several different places. This also externalizes the various current and future associations between ALB and other parts of the infrastructure that may already be defined in TF/CFN (certs, Route53, WAF, CloudFront, other config).