search

aws-ia / terraform-aws-control_tower_account_factory

AWS Control Tower Account Factory
Apache License 2.0
604 stars 386 forks source link

Too many call on Service Catalog ? #403

Open Menahem1 opened 8 months ago

Menahem1 commented 8 months ago

AFT Version: 1.10.4

Bug Description Too many API call on Service Catalog Actually i have more than 30.000 API Calls on Service Catalog, is this normal ?

On CloudTrail i see many recurrence Opera Instantané_2023-10-25_103151_eu-west-3 console aws amazon com

Any idea why ?

Thanks

PeterBengtson commented 8 months ago

I've been using AFT for years, and I'm also worried about this. In a system where no new accounts are created and no accounts updated, I still see lots of Service Catalog calls, to the point where the free tier is exhausted many times over.

AFT is obviously quite inefficient and expensive in this respect. I see no reason why this incessant polling should take place. Perhaps its creators can elucidate.

On Wed, 25 Oct 2023 at 10:45, Ménahem @.***> wrote:

AFT Version: 1.10.4

Bug Description Too many API call on Service Catalog Actually i have more than 30.000 API Calls on Service Catalog, is this normal ?

On CloudTrail i see many recurrence [image: Opera Instantané_2023-10-25_103151_eu-west-3 console aws amazon com] https://user-images.githubusercontent.com/1558463/277929966-85d60364-f484-472d-8fc2-ebec2daf9311.png

Any idea why ?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/aws-ia/terraform-aws-control_tower_account_factory/issues/403, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAA6OMQTBPUGAH2WNA2MCDYBDGS5AVCNFSM6AAAAAA6O7QCCWVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3DAOBXGE3TKMA . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

hanafya commented 8 months ago

Hey @Menahem1! Can you tell us the time range where you are seeing the API calls? In addition, how many customizations are you running?

Menahem1 commented 8 months ago

Hey @hanafya No time range is configured on the screenshot (on the monthly bill i see more that 30.000 api calls on September) Juste one customizations and very few accounts created

hanafya commented 8 months ago

Hey @Menahem1 How many accounts are in your AWS Organization?

Menahem1 commented 8 months ago

about 15

giedriuskilcauskas commented 7 months ago

The guilty one is here: https://github.com/aws-ia/terraform-aws-control_tower_account_factory/blob/main/modules/aft-account-request-framework/eventbridge.tf#L68 That lambda runs every 5 mins and makes many calls to Service catalog API. As a quick workaround we just manually changed that to like 12hours (even that is too much). Ideally, it should be triggered by SQS only

PeterBengtson commented 5 months ago

Has this been fixed?

robbycuenot commented 1 week ago

I've noticed this as well. Removing the NAT gateway requirement dramatically reduced the cost of running AFT (from $100/month to $50/month, roughly). Now, the Service Catalog calls make up the bulk of the cost.

image

If this could be changed in a new minor release, it would halve the cost again. The screenshot above is for an AFT setup with 8 accounts, and no new accounts created this calendar year.

wellsiau-aws commented 1 week ago

I suspect this part of the aft_account_request_processor is causing excessive call.

it might be a good intention to throw warning earlier, but I argue this call should begin only when there's a new message in the SQS queue.