alb.ingress.kubernetes.io/tags not updating properly

t0rr3sp3dr0 commented 5 years ago

If you update alb.ingress.kubernetes.io/tags, tags are not created, deleted, or updated on ELB. For it to update you need to change alb.ingress.kubernetes.io/tags again, and it will show the previous tags. Then you change it again and the tags from the second application will appear, but not the ones from the last application.

	Tags on Kubernetes YAML	Tags on AWS ELB
Initial Apply	A=a	A=a
2nd Apply	B=b	A=a
3rd Apply	C=c	B=b
4th Apply		C=c
5th Apply	D=d
6th Apply	E=e	D=d
7th Apply	E=e,F=f	E=e
8th Apply	F=f	E=e,F=f
9th Apply	G=g	F=f
10th Apply	G=g	G=g

The AWS tags are always one state behind from the ones specified on Kubernetes YAML.

It is even stranger because AddTags and DeleteTags events are logged on CloudTrail and on AWS search bar I can see the tags and its values, but if I filter no ELBs are listed. The same happens on AWS CLI and Boto3 if you describe the ELBs, its tags do not appear or appear outdated.

I've waited more than 10min to verify if there was some delay on tag application but nothing happened. The tags just update if I change Ingress YAML again.

The changes on YAML doesn't necessarily need to be on alb.ingress.kubernetes.io/tags, any change the forces Kubernetes to reconfigure Ingress will make the tags from the previous configuration appear.

Initial Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: A=a
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

2nd Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: B=b
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

3rd Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: C=c
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

4th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

5th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: D=d
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

6th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: E=e
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

7th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: E=e,F=f
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

8th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: F=f
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

9th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: G=g
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

10th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: G=g
    alb.ingress.kubernetes.io/target-type: ip
    H: h
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

fimbulvetr commented 5 years ago

I can confirm this behavior. Tags are properly applied to TargetGroups, however.

rifelpet commented 5 years ago

waf-acl-id is also experiencing similar behavior. Adding the annotation to an existing ingress doesnt do anything. removing the annotation causes the ingress controller to add the waf acl to the alb. I believe that the ingress controller's sync period will eventually reconcile the proper behavior, but this defaults to 1 hour.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/945#issuecomment-573428169): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

salemgolemugoo commented 4 years ago

I have the same issue and it's not being fixed

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/945#issuecomment-602265315): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

swarmer commented 4 years ago

/reopen

Well, I tried

k8s-ci-robot commented 4 years ago

@swarmer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/945#issuecomment-616654818): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

t0rr3sp3dr0 commented 4 years ago

/reopen

k8s-ci-robot commented 4 years ago

@t0rr3sp3dr0: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/945#issuecomment-616666262): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/945#issuecomment-631608126): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

tibin-mfl commented 4 years ago

Any update on this issue ? Even I am facing same issue while enabling logs on alb.

k8s-ci-robot commented 4 years ago

@tibin-mfl: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/945#issuecomment-656027913): >/reopen > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

msonnabaum commented 4 years ago

Just ran into the exact same issue trying to update alb.ingress.kubernetes.io/load-balancer-attributes.

MaxBango commented 3 years ago

Also seeing the problem with alb.ingress.kubernetes.io/load-balancer-attributes.

ALB Ingress Controller v1.1.8, Kubernetes v1.17.

kubernetes-sigs / aws-load-balancer-controller

alb.ingress.kubernetes.io/tags not updating properly #945