search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.95k stars 1.47k forks source link

attributes modification failed: InvalidParameter: 1 validation error(s) found. #655

Closed natefox closed 5 years ago

natefox commented 6 years ago

Running beta 7, but was happening on beta 6 too. Deleting the pod doesnt make it go away.

I1001 22:34:08.200030       1 albingress.go:230] XXXXXXXX/YYYYYY-ing: Ingress rebuilt from existing ALB in AWS
I1001 22:34:08.200055       1 albingresses.go:101] Assembled 1 ingresses from existing AWS resources in 2.110852856s
I1001 22:34:08.506569       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
I1001 22:34:08.506750       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"af3e6175-8a10-11e8-af37-0aa470bd96fc", APIVersion:"extensions/v1beta1", ResourceVersion:"22993899", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 22:34:08.516211       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1001 22:34:08.516225       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1001 22:34:08.516227       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 22:34:08.516230       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1001 22:34:08.516237       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 500.746211ms
I1001 22:34:09.132739       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
I1001 22:34:09.132891       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"af3e6175-8a10-11e8-af37-0aa470bd96fc", APIVersion:"extensions/v1beta1", ResourceVersion:"22993899", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 22:34:09.142637       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1001 22:34:09.142654       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1001 22:34:09.142656       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 22:34:09.142659       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1001 22:34:09.142666       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 1.001226524s

[SNIP]

I1001 22:52:39.161133       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
I1001 22:52:39.161333       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"af3e6175-8a10-11e8-af37-0aa470bd96fc", APIVersion:"extensions/v1beta1", ResourceVersion:"22993899", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 22:52:39.169773       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1001 22:52:39.169791       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1001 22:52:39.169794       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 22:52:39.169796       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1001 22:52:39.169804       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 17m7.719867911s
I1001 23:10:05.780305       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
I1001 23:10:05.780461       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"af3e6175-8a10-11e8-af37-0aa470bd96fc", APIVersion:"extensions/v1beta1", ResourceVersion:"22993899", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 23:10:05.789430       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1001 23:10:05.789447       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1001 23:10:05.789449       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1001 23:10:05.789452       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1001 23:10:05.789466       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 33m53.934022039s

(XXXX and YYY are obfuscation from internal names)

I'm running the following on the server, so I think thats all the info I'm going to get.

        - name: AWS_DEBUG
          value: "true"
        - name: LOG_LEVEL
          value: DEBUG

Any help debugging this and understanding what is causing this to be in this loop is appreciated.

M00nF1sh commented 6 years ago

Hi, the AWS_DEBUG environment variable is deprecated and no longer work. Instead, you should specify --aws-api-debug as container args in the alb ingress controller yaml. similar to --ingress-class=alb.

After you specified it, it will print out AWS API call request&response in the controller logs, then you can obfuscate it and paste here :D, and i'll help to diagnose and fix it 😃

natefox commented 6 years ago

Here's a wall-o-text, mostly obfuscated. I didnt see anything but I'm hoping I'm just blind :)

-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    1.0-beta.6
  Build:      git-f740c293
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
-------------------------------------------------------------------------------

I1003 17:53:47.662130       1 flags.go:132] Watching for Ingress class: alb
W1003 17:53:47.662238       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1003 17:53:47.662361       1 main.go:159] Creating API client for https://172.20.0.1:443
I1003 17:53:47.675406       1 main.go:203] Running in Kubernetes cluster version v1.10 (v1.10.3) - git (clean) commit 2bba0127d85d5a46ab4b778548be28623b32d0b0 - platform linux/amd64
I1003 17:53:47.676020       1 alb.go:85] ALB resource names will be prefixed with 45f8e3d7
I1003 17:53:47.683640       1 alb.go:158] Starting AWS ALB Ingress controller
I1003 17:53:48.893056       1 rgt.go:128] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["ec2"],  TagFilters: [{      Key: "kubernetes.io/role/internal-elb",      Values: ["","1"]    },{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:48.984043       1 leaderelection.go:185] attempting to acquire leader lease  default/ingress-controller-leader-alb...
I1003 17:53:48.984196       1 rgt.go:128] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["ec2"],  TagFilters: [{      Key: "kubernetes.io/role/internal-elb",      Values: ["","1"]    },{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.000196       1 status.go:152] new leader elected: YYYYYY-alb-ingress-controller-controller-574b588bf5-72qdg
I1003 17:53:49.091200       1 rgt.go:128] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["ec2"],  TagFilters: [{      Key: "kubernetes.io/role/elb",      Values: ["","1"]    },{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.100870       1 rgt.go:128] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["ec2"],  TagFilters: [{      Key: "kubernetes.io/role/elb",      Values: ["","1"]    },{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.104207       1 rgt.go:128] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["elasticloadbalancing"],  TagFilters: [{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.117623       1 rgt.go:128] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["elasticloadbalancing"],  TagFilters: [{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.216030       1 rgt.go:128] Request: tagging/GetResources, Payload: {  PaginationToken: "1!1ND/ez3wSxg+pjWBauZAmKTprmfLxiLlBsjHRRdZts2k7VUXPW4skyqRroNKvlh6UE/nz5qsNJyAPmWyvlCpzJ1v+RrAdn7HYQTWMPOFfQ5voPJt6AbJOfUVMGAG/qGML39fjf1u0S8cCYIrBY119PjDCK7cqSmXa6RMPQHB8HnDGsJUY5YxGIOdRiAzVvCNHuEt5qg0vdaccytUzmpUbtGNkxhyzy857xi/DdpLMd/nodabF029aloFu1j0qZ22PecsprQpIHGGBABT4ELHP6zYoi/fO7IQo0mJLGUvOt0P+xYEWvMDJYET7UavXVpldTY2KO+zg28ghBgd87aH5Lk1PQ5RmHcHBn0iS6VJ42HyIJoV6QC5/3rNgEHZTg6GTrCOt9kNKvj/Ohy8kI0+Bw==",  ResourceTypeFilters: ["elasticloadbalancing"],  TagFilters: [{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.223558       1 rgt.go:128] Request: tagging/GetResources, Payload: {  PaginationToken: "1!1ND/ez3wSxg+pjWBauZAmKTprmfLxiLlBsjHRRdZts2k7VUXPW4skyqRroNKvlh6UE/nz5qsNJyAPmWyvlCpzJ1v+RrAdn7HYQTWMPOFfQ5voPJt6AbJOfUVMGAG/qGML39fjf1u0S8cCYIrBY119PjDCK7cqSmXa6RMPQHB8HnDGsJUY5YxGIOdRiAzVvCNHuEt5qg0vdaccytUzmpUbtGNkxhyzy857xi/DdpLMd/nodabF029aloFu1j0qZ22PecsprQpIHGGBABT4ELHP6zYoi/fO7IQo0mJLGUvOt0P+xYEWvMDJYET7UavXVpldTY2KO+zg28ghBgd87aH5AoX+mHB5KJOGR2O8YyGje/yIJoV6QC5/3rNgEHZTg6GTrCOt9kNKvj/Ohy8kI0+Bw==",  ResourceTypeFilters: ["elasticloadbalancing"],  TagFilters: [{      Key: "kubernetes.io/cluster/XXXXXXXX-dev-east",      Values: ["owned","shared"]    }]}
I1003 17:53:49.290274       1 ec2.go:856] Request: ec2/DescribeSubnets, Payload: {  Filters: [{      Name: "subnet-id",      Values: ["subnet-9506a8c9","subnet-3a354970","subnet-ecd972c2"]    }]}
I1003 17:53:49.295036       1 albingresses.go:77] Building list of existing ALBs
I1003 17:53:49.295144       1 elbv2.go:342] Request: elasticloadbalancing/DescribeLoadBalancers, Payload: {}
I1003 17:53:49.343389       1 albingresses.go:85] Fetching information on 1 ALBs
I1003 17:53:49.343523       1 elbv2.go:380] Request: elasticloadbalancing/DescribeTargetGroups, Payload: {}
I1003 17:53:49.433233       1 api.go:72] Request: ec2metadata/GetDynamicData, Payload: <invalid value>
I1003 17:53:49.434436       1 ec2.go:734] Request: ec2/DescribeInstances, Payload: {  InstanceIds: ["i-087ccbcdc2f6b6ec6"]}
I1003 17:53:49.445174       1 elbv2.go:400] Request: elasticloadbalancing/DescribeLoadBalancerAttributes, Payload: {  LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:53:49.456581       1 ec2.go:182] Request: ec2/DescribeTags, Payload: {  Filters: [{      Name: "resource-id",      Values: ["sg-0acc8a6c95abdc2c4"]    }]}
I1003 17:53:49.568492       1 wafregional.go:62] Request: waf-regional/GetWebACLForResource, Payload: {  ResourceArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:53:49.592679       1 ec2.go:286] Request: ec2/DescribeSecurityGroups, Payload: {  Filters: [{      Name: "tag:Name",      Values: ["XXXXXXXX-dev-east-Alb-Ingress"]    },{      Name: "vpc-id",      Values: ["vpc-1772806d"]    }]}
I1003 17:53:50.649631       1 wafregional.go:62] Request: waf-regional/GetWebACLForResource, Payload: {  ResourceArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:53:52.389087       1 wafregional.go:62] Request: waf-regional/GetWebACLForResource, Payload: {  ResourceArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:53:54.602862       1 wafregional.go:62] Request: waf-regional/GetWebACLForResource, Payload: {  ResourceArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:54:01.956054       1 wafregional.go:62] Request: waf-regional/GetWebACLForResource, Payload: {  ResourceArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:54:01.997870       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-170cb8c82d9b825fc36/448a6090e0a46756"}
I1003 17:54:02.284483       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-170cb8c82d9b825fc36/448a6090e0a46756
I1003 17:54:02.284631       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-170cb8c82d9b825fc36/448a6090e0a46756"}
I1003 17:54:02.329769       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-3555f5c80e528c4a040/39b93498dd8b98c8"}
I1003 17:54:02.348842       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-3555f5c80e528c4a040/39b93498dd8b98c8
I1003 17:54:02.348961       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-3555f5c80e528c4a040/39b93498dd8b98c8"}
I1003 17:54:02.385260       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-4859fc0b28a3290192c/5d1c37cdc56a8d6d"}
I1003 17:54:02.403654       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-4859fc0b28a3290192c/5d1c37cdc56a8d6d
I1003 17:54:02.403770       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-4859fc0b28a3290192c/5d1c37cdc56a8d6d"}
I1003 17:54:02.456072       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-63852ff858d0ca72924/05d0cbb0aa44ab5c"}
I1003 17:54:02.471662       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-63852ff858d0ca72924/05d0cbb0aa44ab5c
I1003 17:54:02.471794       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-63852ff858d0ca72924/05d0cbb0aa44ab5c"}
I1003 17:54:02.509281       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-69c0c521ce3fbdc4216/5945094385f820d8"}
I1003 17:54:02.524654       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-69c0c521ce3fbdc4216/5945094385f820d8
I1003 17:54:02.524789       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-69c0c521ce3fbdc4216/5945094385f820d8"}
I1003 17:54:02.573520       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-6af4c75866403501406/3f2a37d676f71b38"}
I1003 17:54:02.589240       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-6af4c75866403501406/3f2a37d676f71b38
I1003 17:54:02.589358       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-6af4c75866403501406/3f2a37d676f71b38"}
I1003 17:54:02.627412       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7d3bdb0744ac24b9f26/c3287c165ed3de4a"}
I1003 17:54:02.641072       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7d3bdb0744ac24b9f26/c3287c165ed3de4a
I1003 17:54:02.641193       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7d3bdb0744ac24b9f26/c3287c165ed3de4a"}
I1003 17:54:02.675440       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7f17e498d2e43fa36b6/591977f76643560f"}
I1003 17:54:02.691578       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7f17e498d2e43fa36b6/591977f76643560f
I1003 17:54:02.691728       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7f17e498d2e43fa36b6/591977f76643560f"}
I1003 17:54:02.751846       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-890615733aaba15d8f3/b329dbb932e49996"}
I1003 17:54:02.766718       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-890615733aaba15d8f3/b329dbb932e49996
I1003 17:54:02.766831       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-890615733aaba15d8f3/b329dbb932e49996"}
I1003 17:54:02.798498       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-8d9500bd62694e563ce/c010b33bd9bcc9d0"}
I1003 17:54:02.814049       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-8d9500bd62694e563ce/c010b33bd9bcc9d0
I1003 17:54:02.814155       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-8d9500bd62694e563ce/c010b33bd9bcc9d0"}
I1003 17:54:02.848545       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9d9337148ba46cfe852/7511d6d7a6ee9b9c"}
I1003 17:54:02.875148       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9d9337148ba46cfe852/7511d6d7a6ee9b9c
I1003 17:54:02.875274       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9d9337148ba46cfe852/7511d6d7a6ee9b9c"}
I1003 17:54:02.908722       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9fcc8775ee0b49c84dd/6e6f4e1752698edd"}
I1003 17:54:02.923454       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9fcc8775ee0b49c84dd/6e6f4e1752698edd
I1003 17:54:02.923592       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9fcc8775ee0b49c84dd/6e6f4e1752698edd"}
I1003 17:54:02.958643       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-a0a8295f0ef3f3b4020/a16ea67367ae2d36"}
I1003 17:54:02.977195       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-a0a8295f0ef3f3b4020/a16ea67367ae2d36
I1003 17:54:02.977321       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-a0a8295f0ef3f3b4020/a16ea67367ae2d36"}
I1003 17:54:03.009833       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-acfdd4a0299afd27111/29e643b05fabe757"}
I1003 17:54:03.025806       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-acfdd4a0299afd27111/29e643b05fabe757
I1003 17:54:03.025919       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-acfdd4a0299afd27111/29e643b05fabe757"}
I1003 17:54:03.077144       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-b219b7ee87885f1a15d/65e5b218c6e30ff7"}
I1003 17:54:03.094754       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-b219b7ee87885f1a15d/65e5b218c6e30ff7
I1003 17:54:03.094871       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-b219b7ee87885f1a15d/65e5b218c6e30ff7"}
I1003 17:54:03.128731       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c2154a10e8d7c038283/1ebb80feb52552f7"}
I1003 17:54:03.144119       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c2154a10e8d7c038283/1ebb80feb52552f7
I1003 17:54:03.144238       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c2154a10e8d7c038283/1ebb80feb52552f7"}
I1003 17:54:03.180261       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c521eaf10d11bddebbb/fdbf2d2b0d51fc7d"}
I1003 17:54:03.246903       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c521eaf10d11bddebbb/fdbf2d2b0d51fc7d
I1003 17:54:03.247024       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c521eaf10d11bddebbb/fdbf2d2b0d51fc7d"}
I1003 17:54:03.285464       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-ce9a200e90a867b99a7/ece35d3f2802a9fc"}
I1003 17:54:03.298614       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-ce9a200e90a867b99a7/ece35d3f2802a9fc
I1003 17:54:03.298727       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-ce9a200e90a867b99a7/ece35d3f2802a9fc"}
I1003 17:54:03.341619       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-e56291c330627b5e5e3/aac942f46948142a"}
I1003 17:54:03.354844       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-e56291c330627b5e5e3/aac942f46948142a
I1003 17:54:03.354949       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-e56291c330627b5e5e3/aac942f46948142a"}
I1003 17:54:03.400665       1 elbv2.go:413] Request: elasticloadbalancing/DescribeTargetGroupAttributes, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-fd057f9812a7f6ed65e/2b071a2f3ac98b61"}
I1003 17:54:03.415120       1 targetgroup.go:176] XXXXXXXX/YYYYYY-ing: Fetching Targets for Target Group arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-fd057f9812a7f6ed65e/2b071a2f3ac98b61
I1003 17:54:03.415229       1 elbv2.go:465] Request: elasticloadbalancing/DescribeTargetHealth, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-fd057f9812a7f6ed65e/2b071a2f3ac98b61"}
I1003 17:54:03.448222       1 api.go:1425] Request: elasticloadbalancing/DescribeListeners, Payload: {  LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}
I1003 17:54:03.467424       1 rules.go:29] XXXXXXXX/YYYYYY-ing: Fetching Rules for Listener arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4/6dab3520aa81363f
I1003 17:54:03.467531       1 rules.go:30] Request: elasticloadbalancing/DescribeRules, Payload: {  ListenerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4/6dab3520aa81363f"}
I1003 17:54:03.570851       1 albingress.go:230] XXXXXXXX/YYYYYY-ing: Ingress rebuilt from existing ALB in AWS
I1003 17:54:03.570879       1 albingresses.go:101] Assembled 1 ingresses from existing AWS resources in 14.275829865s
I1003 17:54:03.571071       1 acm.go:30] Request: acm/ListCertificates, Payload: {  MaxItems: 1}
I1003 17:54:03.571189       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-0ba345d6a2d0bac0c"]}
I1003 17:54:03.619514       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-0c8358f14848d4ee3"]}
I1003 17:54:03.630552       1 ec2.go:801] Request: ec2/DescribeTags, Payload: {  MaxResults: 6}
I1003 17:54:03.715240       1 elbv2.go:530] Request: elasticloadbalancing/DescribeLoadBalancers, Payload: {  PageSize: 1}
I1003 17:54:03.738722       1 iam.go:30] Request: iam/ListServerCertificates, Payload: {  MaxItems: 1}
I1003 17:54:03.798095       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-05ed8331dcc654a8d"]}
I1003 17:54:04.034011       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-0fcb95e0a2ece85c5"]}
I1003 17:54:04.169728       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-087ccbcdc2f6b6ec6"]}
I1003 17:54:04.252769       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-0a8d5a8ea7a151c9f"]}
I1003 17:54:04.287591       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-0a0f8c5a0238ccdfa"]}
I1003 17:54:04.321406       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-02e0a5cb8fd98ac18"]}
I1003 17:54:04.460396       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-0aac79850fe933dd5"]}
I1003 17:54:04.809059       1 ec2.go:907] Request: ec2/DescribeInstanceStatus, Payload: {  InstanceIds: ["i-07ff51c4c34c4a741"]}
I1003 17:54:04.861474       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
E1003 17:54:04.861541       1 api.go:2420] Failed request: elasticloadbalancing/ModifyLoadBalancerAttributes, Payload: {  LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}, Error: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
I1003 17:54:04.861681       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30206, i-0a8d5a8ea7a151c9f:30206, i-0c8358f14848d4ee3:30206) and removing ()
I1003 17:54:04.861687       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
I1003 17:54:04.861800       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-3555f5c80e528c4a040/39b93498dd8b98c8",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30206    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30206    },{      Id: "i-0c8358f14848d4ee3",      Port: 30206    }]}
I1003 17:54:05.109877       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-3555f5c80e528c4a040 target group modified
I1003 17:54:05.109989       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:32093, i-0a8d5a8ea7a151c9f:32093, i-0c8358f14848d4ee3:32093) and removing ()
I1003 17:54:05.110140       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c521eaf10d11bddebbb/fdbf2d2b0d51fc7d",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 32093    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 32093    },{      Id: "i-0c8358f14848d4ee3",      Port: 32093    }]}
I1003 17:54:05.270408       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-c521eaf10d11bddebbb target group modified
I1003 17:54:05.270463       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30176, i-0a8d5a8ea7a151c9f:30176, i-0c8358f14848d4ee3:30176) and removing ()
I1003 17:54:05.270654       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7d3bdb0744ac24b9f26/c3287c165ed3de4a",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30176    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30176    },{      Id: "i-0c8358f14848d4ee3",      Port: 30176    }]}
I1003 17:54:05.464772       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-7d3bdb0744ac24b9f26 target group modified
I1003 17:54:05.464828       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:32047, i-0a8d5a8ea7a151c9f:32047, i-0c8358f14848d4ee3:32047) and removing ()
I1003 17:54:05.464971       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-4859fc0b28a3290192c/5d1c37cdc56a8d6d",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 32047    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 32047    },{      Id: "i-0c8358f14848d4ee3",      Port: 32047    }]}
I1003 17:54:05.623474       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-4859fc0b28a3290192c target group modified
I1003 17:54:05.623539       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:31137, i-0a8d5a8ea7a151c9f:31137, i-0c8358f14848d4ee3:31137) and removing ()
I1003 17:54:05.623683       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-ce9a200e90a867b99a7/ece35d3f2802a9fc",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 31137    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 31137    },{      Id: "i-0c8358f14848d4ee3",      Port: 31137    }]}
I1003 17:54:05.745185       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-ce9a200e90a867b99a7 target group modified
I1003 17:54:05.745220       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:31530, i-0a8d5a8ea7a151c9f:31530, i-0c8358f14848d4ee3:31530) and removing ()
I1003 17:54:05.745338       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-63852ff858d0ca72924/05d0cbb0aa44ab5c",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 31530    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 31530    },{      Id: "i-0c8358f14848d4ee3",      Port: 31530    }]}
I1003 17:54:05.955579       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30227, i-0a8d5a8ea7a151c9f:30227, i-0c8358f14848d4ee3:30227) and removing ()
I1003 17:54:05.955564       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-63852ff858d0ca72924 target group modified
I1003 17:54:05.955713       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-890615733aaba15d8f3/b329dbb932e49996",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30227    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30227    },{      Id: "i-0c8358f14848d4ee3",      Port: 30227    }]}
I1003 17:54:06.190928       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-890615733aaba15d8f3 target group modified
I1003 17:54:06.190959       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:31300, i-0a8d5a8ea7a151c9f:31300, i-0c8358f14848d4ee3:31300) and removing ()
I1003 17:54:06.191077       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-6af4c75866403501406/3f2a37d676f71b38",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 31300    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 31300    },{      Id: "i-0c8358f14848d4ee3",      Port: 31300    }]}
I1003 17:54:06.338874       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-6af4c75866403501406 target group modified
I1003 17:54:06.338894       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30623, i-0a8d5a8ea7a151c9f:30623, i-0c8358f14848d4ee3:30623) and removing ()
I1003 17:54:06.339071       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-7f17e498d2e43fa36b6/591977f76643560f",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30623    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30623    },{      Id: "i-0c8358f14848d4ee3",      Port: 30623    }]}
I1003 17:54:06.474468       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-7f17e498d2e43fa36b6 target group modified
I1003 17:54:06.474503       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30617, i-0a8d5a8ea7a151c9f:30617, i-0c8358f14848d4ee3:30617) and removing ()
I1003 17:54:06.474634       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-69c0c521ce3fbdc4216/5945094385f820d8",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30617    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30617    },{      Id: "i-0c8358f14848d4ee3",      Port: 30617    }]}
I1003 17:54:06.632070       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30568, i-0a8d5a8ea7a151c9f:30568, i-0c8358f14848d4ee3:30568) and removing ()
I1003 17:54:06.632108       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-69c0c521ce3fbdc4216 target group modified
I1003 17:54:06.632200       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9fcc8775ee0b49c84dd/6e6f4e1752698edd",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30568    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30568    },{      Id: "i-0c8358f14848d4ee3",      Port: 30568    }]}
I1003 17:54:06.833409       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-9fcc8775ee0b49c84dd target group modified
I1003 17:54:06.833446       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30486, i-0a8d5a8ea7a151c9f:30486, i-0c8358f14848d4ee3:30486) and removing ()
I1003 17:54:06.833586       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-170cb8c82d9b825fc36/448a6090e0a46756",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30486    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30486    },{      Id: "i-0c8358f14848d4ee3",      Port: 30486    }]}
I1003 17:54:06.990710       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:31480, i-0a8d5a8ea7a151c9f:31480, i-0c8358f14848d4ee3:31480) and removing ()
I1003 17:54:06.990739       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-170cb8c82d9b825fc36 target group modified
I1003 17:54:06.990840       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-acfdd4a0299afd27111/29e643b05fabe757",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 31480    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 31480    },{      Id: "i-0c8358f14848d4ee3",      Port: 31480    }]}
I1003 17:54:07.156928       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:31810, i-0a8d5a8ea7a151c9f:31810, i-0c8358f14848d4ee3:31810) and removing ()
I1003 17:54:07.156910       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-acfdd4a0299afd27111 target group modified
I1003 17:54:07.157068       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-8d9500bd62694e563ce/c010b33bd9bcc9d0",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 31810    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 31810    },{      Id: "i-0c8358f14848d4ee3",      Port: 31810    }]}
I1003 17:54:07.339206       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:32606, i-0a8d5a8ea7a151c9f:32606, i-0c8358f14848d4ee3:32606) and removing ()
I1003 17:54:07.339223       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-8d9500bd62694e563ce target group modified
I1003 17:54:07.339339       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-e56291c330627b5e5e3/aac942f46948142a",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 32606    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 32606    },{      Id: "i-0c8358f14848d4ee3",      Port: 32606    }]}
I1003 17:54:07.531041       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-e56291c330627b5e5e3 target group modified
I1003 17:54:07.531073       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30177, i-0a8d5a8ea7a151c9f:30177, i-0c8358f14848d4ee3:30177) and removing ()
I1003 17:54:07.531199       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-b219b7ee87885f1a15d/65e5b218c6e30ff7",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30177    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30177    },{      Id: "i-0c8358f14848d4ee3",      Port: 30177    }]}
I1003 17:54:07.646882       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-b219b7ee87885f1a15d target group modified
I1003 17:54:07.646905       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30405, i-0a8d5a8ea7a151c9f:30405, i-0c8358f14848d4ee3:30405) and removing ()
I1003 17:54:07.647031       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-fd057f9812a7f6ed65e/2b071a2f3ac98b61",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30405    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30405    },{      Id: "i-0c8358f14848d4ee3",      Port: 30405    }]}
I1003 17:54:07.828350       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-fd057f9812a7f6ed65e target group modified
I1003 17:54:07.828366       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:32107, i-0a8d5a8ea7a151c9f:32107, i-0c8358f14848d4ee3:32107) and removing ()
I1003 17:54:07.828521       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-a0a8295f0ef3f3b4020/a16ea67367ae2d36",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 32107    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 32107    },{      Id: "i-0c8358f14848d4ee3",      Port: 32107    }]}
I1003 17:54:07.988579       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-a0a8295f0ef3f3b4020 target group modified
I1003 17:54:07.988627       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30819, i-0a8d5a8ea7a151c9f:30819, i-0c8358f14848d4ee3:30819) and removing ()
I1003 17:54:07.988752       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-c2154a10e8d7c038283/1ebb80feb52552f7",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30819    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30819    },{      Id: "i-0c8358f14848d4ee3",      Port: 30819    }]}
I1003 17:54:08.167954       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-c2154a10e8d7c038283 target group modified
I1003 17:54:08.168064       1 targetgroup.go:342] XXXXXXXX/YYYYYY-ing: Modifying target group targets. Adding (i-087ccbcdc2f6b6ec6:30300, i-0a8d5a8ea7a151c9f:30300, i-0c8358f14848d4ee3:30300) and removing ()
I1003 17:54:08.168188       1 targetgroup.go:471] Request: elasticloadbalancing/RegisterTargets, Payload: {  TargetGroupArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/45f8e3d7-9d9337148ba46cfe852/7511d6d7a6ee9b9c",  Targets: [{      Id: "i-087ccbcdc2f6b6ec6",      Port: 30300    },{      Id: "i-0a8d5a8ea7a151c9f",      Port: 30300    },{      Id: "i-0c8358f14848d4ee3",      Port: 30300    }]}
I1003 17:54:08.446605       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Normal' reason: 'MODIFY' 45f8e3d7-9d9337148ba46cfe852 target group modified
E1003 17:54:08.453012       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1003 17:54:08.453031       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1003 17:54:08.453034       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:08.453036       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1003 17:54:08.453044       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 500.685478ms
I1003 17:54:18.886050       1 ec2.go:856] Request: ec2/DescribeSubnets, Payload: {  Filters: [{      Name: "subnet-id"    }]}
I1003 17:54:19.001762       1 ec2.go:856] Request: ec2/DescribeSubnets, Payload: {  Filters: [{      Name: "subnet-id"    }]}
I1003 17:54:19.005310       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
E1003 17:54:19.005366       1 api.go:2420] Failed request: elasticloadbalancing/ModifyLoadBalancerAttributes, Payload: {  LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}, Error: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
I1003 17:54:19.005473       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:19.013988       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1003 17:54:19.014007       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1003 17:54:19.014010       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:19.014013       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1003 17:54:19.014021       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 1.001495484s
I1003 17:54:19.054344       1 ec2.go:856] Request: ec2/DescribeSubnets, Payload: {  Filters: [{      Name: "subnet-id"    }]}
I1003 17:54:19.116975       1 ec2.go:286] Request: ec2/DescribeSecurityGroups, Payload: {  Filters: [{      Name: "tag:Name",      Values: ["XXXXXXXX-dev-east-Alb-Ingress"]    },{      Name: "vpc-id",      Values: ["vpc-1772806d"]    }]}
I1003 17:54:19.203173       1 ec2.go:856] Request: ec2/DescribeSubnets, Payload: {  Filters: [{      Name: "subnet-id"    }]}
I1003 17:54:19.271600       1 ec2.go:856] Request: ec2/DescribeSubnets, Payload: {  Filters: [{      Name: "subnet-id"    }]}
I1003 17:54:22.338325       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
E1003 17:54:22.338380       1 api.go:2420] Failed request: elasticloadbalancing/ModifyLoadBalancerAttributes, Payload: {  LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}, Error: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
I1003 17:54:22.338559       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:22.346732       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1003 17:54:22.346748       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1003 17:54:22.346751       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:22.346753       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1003 17:54:22.346761       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 2.004733868s
I1003 17:54:25.671366       1 loadbalancer.go:522] XXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [].
E1003 17:54:25.671428       1 api.go:2420] Failed request: elasticloadbalancing/ModifyLoadBalancerAttributes, Payload: {  LoadBalancerArn: "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/45f8e3d7-XXXXXXXX-YYYYYYin-6698/ddfaecfc789ae1c4"}, Error: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
I1003 17:54:25.671519       1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"XXXXXXXX", Name:"YYYYYY-ing", UID:"189c4ec4-c5f5-11e8-8c43-0e903259af56", APIVersion:"extensions/v1beta1", ResourceVersion:"23082529", FieldPath:""}): type: 'Warning' reason: 'ERROR' 45f8e3d7-XXXXXXXX-YYYYYYin-6698 attributes modification failed: InvalidParameter: 1 validation error(s) found.
- missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:25.679710       1 albingress.go:290] XXXXXXXX/YYYYYY-ing: Failed to reconcile state on this ingress
E1003 17:54:25.679726       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:  - Failed modifying attributes: InvalidParameter: 1 validation error(s) found.
E1003 17:54:25.679729       1 albingress.go:292] XXXXXXXX/YYYYYY-ing: - missing required field, ModifyLoadBalancerAttributesInput.Attributes.
E1003 17:54:25.679731       1 albingress.go:292] XXXXXXXX/YYYYYY-ing:
E1003 17:54:25.679739       1 albingress.go:295] XXXXXXXX/YYYYYY-ing: Will retry to reconcile in 4.039050118s
M00nF1sh commented 6 years ago

@natefox Cool 👍 , I know the root cause of this problem now. (To reproduce, adding annotation: alb.ingress.kubernetes.io/load-balancer-attributes: 'deletion_protection.enabled=true' and then remove it) I'll check whether @bigkraig 's recent refactor fixes this problem.

M00nF1sh commented 6 years ago

confirming it's already fixed by bigkraig's lb attributes refactor.

you can use this image: quay.io/coreos/alb-ingress-controller:13b731a which contains the fix.

BTW, i'd prefer to use the more stable beta.7 version for now(since there are refactoring work in master branch, and is not fully tested) , one way to mitigate this issue on beta.7 is adding alb.ingress.kubernetes.io/load-balancer-attributes: 'deletion_protection.enabled=false' annotation to your ingresses.

natefox commented 6 years ago

I'm using the image you mentioned and it's now giving me a good (very fixable) error:

I1004 16:15:48.753729       1 attributes.go:144] XXXXXXXXX/YYYYYY-ing: Modifying ELBV2 attributes to [{    Key: "access_logs.s3.enabled",    Value: "false"  },{    Key: "access_logs.s3.bucket",    Value: ""  }].

Apparently our security team enabled/mandated all LBs to turn on logging, so the controller is fighting that setting. I've updated beta.7 with the proper load-balancer-attributes and the error has gone away!

However, I did get this error with the 13b731a image:

E1004 16:11:43.814356       1 albingress.go:301] XXXXXXXXX/YYYYYY-ing:  - failed configuration of target group tags due to AccessDeniedException: User: arn:aws:sts::12345678912:assumed-role/YYYYYY-AlbIngressController-Role/kiam-kiam is not authorized to perform: tag:TagResources

Adding tag:TagResources to the policy fixed that. However, I don't see that in the examples/iam-policy.json file. I'll make a quick PR for that assuming it's new functionality.

M00nF1sh commented 6 years ago

Hi natefox, Thanks for reporting this error 😄 . The new version of alb ingress controller tries to set all settings to the default value. You can get around that issue by setting the attributes(e.g. logging) via annotation. However, from my point, i think the ingress controller should only mandate settings that are explicitly specified by users via annotation, and leave other settings as it as.(That allows more flexibility for use cases like yours).
We can discuss more about whether this should be the correct behavior.

natefox commented 6 years ago

I personally think a desired state system should set all the things, accounting for built in defaults. So if I dont have an annotation, it should set it to whatever the default is (in this case, no attributes like it was trying to do). This effectively undoes any outside changes (manual or in my case, a lambda enforcing a rule) and allows you to expect what is in the definition file is what is in the actual architecture. While infra tests are always good, if a desired state system doesnt undo things then tests become mandatory. If you cant rely on your code enforcing your infrastructure, you're forced to validate it some other way.

The enhanced error logging was the big win using the new refactor - it showed me why it was tanking so I could take appropriate action.

jh-sz commented 5 years ago

I can confirm that the issue is still there for access_logs.s3.bucket field for v1, beta.7 and beta.6. To reproduce, you can do it by either adding the annotation "alb.ingress.kubernetes.io/attributes": "access_logs.s3.enabled=true,access_logs.s3.bucket=my-bucket" (or just enable logging manually from aws console), then remove the annotation. You will get the error ValidationError: The value of 'access_logs.s3.bucket' cannot be empty

From debug logs, on DescribeLoadBalancer you will get access_logs.s3.bucket=my-bucket and when ModifyLoadBalancer it's setting access_logs.s3.bucket="", is the fix simply checking for empty string on this line?

jh-sz commented 5 years ago

@M00nF1sh can you confirm, thanks ^

M00nF1sh commented 5 years ago

@jh-sz ,i can confirm this issue exists 😄 , will fix it today