search

aws-controllers-k8s / community

AWS Controllers for Kubernetes (ACK) is a project enabling you to manage AWS services from Kubernetes
https://aws-controllers-k8s.github.io/community/
Apache License 2.0
2.41k stars 254 forks source link

New UpdateAssumeRolePolicy is causing unexpected spike in cloudtrail events/cost #1869

Closed twstewart42 closed 11 months ago

twstewart42 commented 1 year ago

Describe the bug Since my team rolled out the update to ack-iam-controller 1.2.1 and maybe related to this issue. we've noticed a big uptick in guardDuty/cloudtrail events for UpdateAssumeRolePolicy on any IAM Role controlled by this controller. The controller is sending an update request every second for every role. We have ~7 unique Roles generated by this controller. This change has doubled our guardDuty bill from the previous month.

redacted screenshot of activity

Screenshot 2023-08-04 at 2 12 04 PM

Steps to reproduce Deploy an IAM role associated with the ack-iam-controller. View cloudtrail events filtered on event name = UpdateAssumeRolePolicy

Expected outcome A way to tune how often the controller is attempting to update IAM roles or slow down how often the controller is sending an Update command to these IAM roles.

Environment

jaikanth-arcadia commented 1 year ago

Related to #1837 AWS API minifies the JSON while the CRD definition has the pretty formatted version of JSON.

a-hilaly commented 1 year ago

Hey @jaikanth-arcadia @twstewart42 i'm currently working on fixing this, expect a new release by end of this week. /assign

a-hilaly commented 1 year ago

@jaikanth-arcadia @twstewart42 expect a release in the upcoming days.. once https://github.com/aws-controllers-k8s/iam-controller/pull/85 is merged we'll ship a new release.

universam1 commented 11 months ago

@a-hilaly This turns out to be a major issue causing our CI to fail since we are monitoring the status ACK.ResourceSynced to pass. Is it possible to bump the priority of the fix?

a-hilaly commented 11 months ago

Fixed in iam-controller 1.3.1 /close

ack-prow[bot] commented 11 months ago

@a-hilaly: Closing this issue.

In response to [this](https://github.com/aws-controllers-k8s/community/issues/1869#issuecomment-1821336538): >Fixed in `iam-controller` 1.3.1 >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.