search

pulumi / pulumi-aws-native

AWS Native Provider for Pulumi
Apache License 2.0
95 stars 17 forks source link

Throttling exceptions trying to provision AWS Organization accounts using the native provider #1687

Open automagic opened 3 months ago

automagic commented 3 months ago

What happened?

Attempting to create several accounts in an Organization at the same time, and getting throttled by the Cloud Control API

aws-native:organizations:Account (iam-nonprod):
error: creating resource: creating resource (await): operation CREATE failed with "Throttling": AWS Organizations can't complete your request because another request is already in progress. Try again later. (Service: Organizations, Status Code: 400, Request ID: 0266cd90-48cb-45d4-aeb9-066206c6db10)

Example

import os
from pulumi_aws_native import organizations
from src.organization import OrganizationTree

org = OrganizationTree(
 config_file=os.path.realpath("config/organization_structure.yaml")
)

for ou in org.get_all_organizational_units():
 if ou.name == "Root":
 ou.aws_ou_id = organizations.get_organization_output(id=ou.name).root_id
 else:
 ou.provision()

for acct in org.get_all_organization_accounts():
 acct.provision()
class OrganizationAccount:
 def __init__(self, name: str, description: str, email: str):
 self.name = name
 self.description = description
 self.email = email
 self.parent = None

 # todo: re-evaluate switching back to the native aws provider
 # 8/26/2024 reverted to the classic provider as the native provider doesn't
 # gracefully handle throttling exceptions
 def provision(self):
 return aws.organizations.Account(
 self.name,
 email=self.email,
 close_on_deletion=True,
 create_govcloud=False,
 iam_user_access_to_billing="ALLOW",
 name=self.name,
 parent_id=self.parent.aws_ou_id,
 role_name="OrganizationAdminRole",
 tags={},

Output of pulumi about

CLI
Version 3.130.0 Go Version go1.22.6 Go Compiler gc

Plugins KIND NAME VERSION resource aws 6.50.1 resource aws-native 0.120.0 resource awsx 2.14.0 resource docker 4.5.5 language python unknown

Host
OS darwin Version 14.5 Arch arm64

This project is written in python: executable='/Users/dan/Library/CloudStorage/Dropbox/Local/Blunatech/Jona Health/src/jona-infra/venv/bin/python' version='3.12.4'

Current Stack: bt_dang/jona-infra/sandbox

TYPE URN pulumi:pulumi:Stack urn:pulumi:sandbox::jona-infra::pulumi:pulumi:Stack::jona-infra-sandbox pulumi:providers:aws urn:pulumi:sandbox::jona-infra::pulumi:providers:aws::default_6_50_1 pulumi:providers:aws-native urn:pulumi:sandbox::jona-infra::pulumi:providers:aws-native::default_0_120_0 pulumi:providers:aws urn:pulumi:sandbox::jona-infra::pulumi:providers:aws::crossAccountProvider aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>infrastructure aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>apps aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>infrastructure>nonproduction aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>apps>development aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>infrastructure>production aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>apps>production aws-native:organizations:OrganizationalUnit urn:pulumi:sandbox::jona-infra::aws-native:organizations:OrganizationalUnit::root>apps>staging aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::networking-nonprod aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::iam-nonprod aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::jona-app-dev aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::networking-prod aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::iam-prod aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::jona-app-staging aws:organizations/account:Account urn:pulumi:sandbox::jona-infra::aws:organizations/account:Account::jona-app-prod

Found no pending operations associated with sandbox

Backend
Name pulumi.com URL https://app.pulumi.com/bt_dang User bt_dang Organizations bt_dang Token type personal

Dependencies: NAME VERSION pip 24.2 pulumi_aws_native 0.120.0 pulumi_awsx 2.14.0 ruff 0.6.2 setuptools 73.0.1 wheel 0.44.0

Pulumi locates its logs in /var/folders/m5/pt2vwvn10yn72zvgn6zlb_qc0000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

corymhall commented 2 months ago

@automagic we do have some logic around retrying operations, but it is very generic. I can think of a couple of ways to workaround this issue.

  1. Run pulumi up --parallel 1 to limit pulumi to creating 1 resource at a time
  2. Use the dependsOn resource option so that each account depends on the previous account. This is also the recommendation if you are using CloudFormation

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-organizations-account.html

If you include multiple accounts in a single template, you must use the DependsOn attribute on each account resource type so that the accounts are created sequentially. If you create multiple accounts at the same time, Organizations returns an error and the stack operation fails.

  1. Switch to using the aws provider which has better specific retry logic around this scenario.