search

aws-ia / terraform-aws-control_tower_account_factory

AWS Control Tower Account Factory
Apache License 2.0
604 stars 386 forks source link

Urgent - AFT Module Deployment Failure and Pipeline Errors #426

Closed Bckarnati2018 closed 4 months ago

Bckarnati2018 commented 4 months ago

Terraform Version & Prov: Using Terraform Cloud (TFC) version 1.5.7

AFT Version: (Can be found in the AFT Management Account in the SSM Parameter /aft/config/aft/version) 1.11.1

Terraform Version & Provider Versions Please provide the outputs of terraform version and terraform providers from within your AFT environment

terraform version 1.5.7

terraform providers Terraform Cloud

terraform distribution tfc

Bug Description A clear and concise description of what the bug is. Successfully deploying AFT module and completed the post deployment tasks, by enabling the CodeStar connection and granting AFT access to service catalog portfolio. Then proceed ahead to rerun the provisioned account and also rerun the account request pipelines. But, both the pipelines fail with he below error.

ERROR: [Container] 2024/02/04 21:25:46.375008 Command did not exit successfully if [ $TF_DISTRIBUTION = "oss" ]; then TF_BACKEND_REGION=$(aws ssm get-parameter --name "/aft/config/oss-backend/primary-region" --query "Parameter.Value" --output text)

ERROR LOGS: Successfully created workspace ct-aft-account-request with ID ws-LqBcpPpaSmsHdJ2m Successfully placed AWS credentials on workspace for arn:aws:iam::113634643425:role/AWSAFTAdmin Handling errors: [{'status': '404', 'title': 'not found'}] Traceback (most recent call last): File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/workspace_manager.py", line 275, in setup_and_run_workspace( File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/workspace_manager.py", line 22, in setup_and_run_workspace run_id = stage_run(workspace_id, assume_role_arn, role_session_name, api_token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/workspace_manager.py", line 48, in stage_run cv_id, upload_url = terraform.create_configuration_version(workspace_id, api_token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/terraform_client.py", line 74, in create_configuration_version response = __post(endpoint, headers, payload) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/terraform_client.py", line 215, in post handle_errors(response) File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/terraform_client.py", line 253, in __handle_errors raise ClientError(status=error["status"], message=error["title"]) terraform_client.ClientError: not found

[Container] 2024/02/04 21:25:46.375008 Command did not exit successfully if [ $TF_DISTRIBUTION = "oss" ]; then TF_BACKEND_REGION=$(aws ssm get-parameter --name "/aft/config/oss-backend/primary-region" --query "Parameter.Value" --output text)

To Reproduce Steps to reproduce the behavior:

  1. Deploy the AFT module as per the article in this url https://developer.hashicorp.com/terraform/tutorials/aws/aws-control-tower-aft
  2. Ensure to use the Terraform Cloud (TFC) distribution, Organization name & Terraform token
  3. Enable the CodeStar to use the GitHub repo and grant AWSAFTExecution role to access the service catalog AWS Control Tower Account Factory Portfolio
  4. Rerun the code pipelines "ct-aft-account-provisioning-customizations" & "ct-aft-account-request"
  5. Both the pipelines are failing with the above error.

Expected behavior A clear and concise description of what you expected to happen. Both the pipelines should be running to test and validate all the four scenarios listed below.

  1. Kick off new account provisioning using AFT using "learn-terraform-aft-account-request" repository.
  2. Apply customizations to all accounts created by AFT using "learn-terraform-aft-global-customizations" repository.
  3. Configuration of account specific customization by using "learn-terraform-aft-account-customizations" repository.
  4. Apply configuration for provisioning-time customizations to accounts by using "learn-terraform-aft-account-provisioning-customizations" repository.

Related Logs Provide any related logs or error messages to help explain your problem.

1| [Container] 2024/02/04 21:24:59.831294 Waiting for agent ping 2 | [Container] 2024/02/04 21:25:00.032584 Waiting for DOWNLOAD_SOURCE 3 | [Container] 2024/02/04 21:25:01.485484 Phase is DOWNLOAD_SOURCE 4 | [Container] 2024/02/04 21:25:01.487826 CODEBUILD_SRC_DIR=/codebuild/output/src202587511/src 5 | [Container] 2024/02/04 21:25:01.488214 YAML location is /codebuild/readonly/buildspec.yml 6 | [Container] 2024/02/04 21:25:01.490696 Setting HTTP client timeout to higher timeout for S3 source 7 | [Container] 2024/02/04 21:25:01.490931 Processing environment variables 8 | [Container] 2024/02/04 21:25:01.767446 No runtime version selected in buildspec. 9 | [Container] 2024/02/04 21:25:01.796794 Moving to directory /codebuild/output/src202587511/src 10 | [Container] 2024/02/04 21:25:01.799896 Unable to initialize cache download: no paths specified to be cached 11 | [Container] 2024/02/04 21:25:01.876460 Configuring ssm agent with target id: codebuild:47a409b5-8403-4868-8371-cb36933e52aa 12 | [Container] 2024/02/04 21:25:01.886807 Successfully updated ssm agent configuration 13 | [Container] 2024/02/04 21:25:01.887154 Registering with agent 14 | [Container] 2024/02/04 21:25:01.887168 Phases found in YAML: 3 15 | [Container] 2024/02/04 21:25:01.887175 POST_BUILD: 1 commands 16 | [Container] 2024/02/04 21:25:01.887180 PRE_BUILD: 20 commands 17 | [Container] 2024/02/04 21:25:01.887190 BUILD: 1 commands 18 | [Container] 2024/02/04 21:25:01.887416 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED 19 | [Container] 2024/02/04 21:25:01.887430 Phase context status code: Message: 20 | [Container] 2024/02/04 21:25:02.074508 Entering phase INSTALL 21 | [Container] 2024/02/04 21:25:02.077543 Phase complete: INSTALL State: SUCCEEDED 22 | [Container] 2024/02/04 21:25:02.077558 Phase context status code: Message: 23 | [Container] 2024/02/04 21:25:02.165830 Entering phase PRE_BUILD 24 | [Container] 2024/02/04 21:25:02.166242 Running command DEFAULT_PATH=$(pwd) 25 |   26 | [Container] 2024/02/04 21:25:02.173178 Running command TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S') 27 |   28 | [Container] 2024/02/04 21:25:02.181509 Running command AWS_MODULE_SOURCE=$(aws ssm get-parameter --name "/aft/config/aft-pipeline-code-source/repo-url" --query "Parameter.Value" --output text) 29 |   30 | [Container] 2024/02/04 21:25:14.249401 Running command AWS_MODULE_GIT_REF=$(aws ssm get-parameter --name "/aft/config/aft-pipeline-code-source/repo-git-ref" --query "Parameter.Value" --output text) 31 |   32 | [Container] 2024/02/04 21:25:14.929991 Running command TF_VERSION=$(aws ssm get-parameter --name "/aft/config/terraform/version" --query "Parameter.Value" --output text) 33 |   34 | [Container] 2024/02/04 21:25:15.641795 Running command TF_DISTRIBUTION=$(aws ssm get-parameter --name "/aft/config/terraform/distribution" --query "Parameter.Value" --output text) 35 |   36 | [Container] 2024/02/04 21:25:16.358261 Running command CT_MGMT_REGION=$(aws ssm get-parameter --name "/aft/config/ct-management-region" --query "Parameter.Value" --output text) 37 |   38 | [Container] 2024/02/04 21:25:17.056081 Running command AFT_MGMT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text) 39 |   40 | [Container] 2024/02/04 21:25:17.721088 Running command AFT_EXEC_ROLE_ARN=arn:$AWS_PARTITION:iam::$AFT_MGMT_ACCOUNT:role/AWSAFTExecution 41 |   42 | [Container] 2024/02/04 21:25:17.728997 Running command AFT_ADMIN_ROLE_NAME=$(aws ssm get-parameter --name /aft/resources/iam/aft-administrator-role-name \| jq --raw-output ".Parameter.Value") 43 |   44 | [Container] 2024/02/04 21:25:18.350353 Running command AFT_ADMIN_ROLE_ARN=arn:$AWS_PARTITION:iam::$AFT_MGMT_ACCOUNT:role/$AFT_ADMIN_ROLE_NAME 45 |   46 | [Container] 2024/02/04 21:25:18.356169 Running command ROLE_SESSION_NAME=$(aws ssm get-parameter --name /aft/resources/iam/aft-session-name \| jq --raw-output ".Parameter.Value") 47 |   48 | [Container] 2024/02/04 21:25:18.961563 Running command ssh_key_parameter=$(aws ssm get-parameter --name /aft/config/aft-ssh-key --with-decryption 2> /dev/null \|\| echo "None") 49 | if [[ $ssh_key_parameter != "None" ]]; then 50 | ssh_key=$(jq --raw-output ".Parameter.Value" <<< $ssh_key_parameter) 51 | mkdir -p ~/.ssh 52 | echo "Host *" >> ~/.ssh/config 53 | echo "StrictHostKeyChecking no" >> ~/.ssh/config 54 | echo "UserKnownHostsFile=/dev/null" >> ~/.ssh/config 55 | echo "$ssh_key" > ~/.ssh/ssh_key 56 | echo -e "\n\n" >> ~/.ssh/ssh_key 57 | chmod 600 ~/.ssh/ssh_key 58 | eval "$(ssh-agent -s)" 59 | ssh-add ~/.ssh/ssh_key 60 | fi 61 |   62 |   63 | [Container] 2024/02/04 21:25:19.583621 Running command git config --global credential.helper '!aws codecommit credential-helper $@' 64 |   65 | [Container] 2024/02/04 21:25:19.590588 Running command git config --global credential.UseHttpPath true 66 |   67 | [Container] 2024/02/04 21:25:19.598115 Running command git clone -b $AWS_MODULE_GIT_REF $AWS_MODULE_SOURCE aws-aft-core-framework 68 | Cloning into 'aws-aft-core-framework'... 69 | Note: switching to '26667e52d0e2f46e3213239933a1c8fcf1a83166'. 70 |   71 | You are in 'detached HEAD' state. You can look around, make experimental 72 | changes and commit them, and you can discard any commits you make in this 73 | state without impacting any branches by switching back to a branch. 74 |   75 | If you want to create a new branch to retain commits you create, you may 76 | do so (now or later) by using -c with the switch command. Example: 77 |   78 | git switch -c 79 |   80 | Or undo this operation with: 81 |   82 | git switch - 83 |   84 | Turn off this advice by setting config variable advice.detachedHead to false 85 |   86 |   87 | [Container] 2024/02/04 21:25:21.082364 Running command python3 -m venv ./venv 88 |   89 | [Container] 2024/02/04 21:25:31.812569 Running command source ./venv/bin/activate 90 |   91 | [Container] 2024/02/04 21:25:31.820232 Running command pip install jinja2-cli==0.7.0 Jinja2==3.0.1 MarkupSafe==2.0.1 boto3==1.18.56 requests==2.26.0 92 | Collecting jinja2-cli==0.7.0 93 | Downloading jinja2_cli-0.7.0-py2.py3-none-any.whl (6.2 kB) 94 | Collecting Jinja2==3.0.1 95 | Downloading Jinja2-3.0.1-py3-none-any.whl (133 kB) 96 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.7/133.7 kB 3.2 MB/s eta 0:00:00 97 | Collecting MarkupSafe==2.0.1 98 | Downloading MarkupSafe-2.0.1.tar.gz (18 kB) 99 | Installing build dependencies: started 100 | Installing build dependencies: finished with status 'done' 101 | Getting requirements to build wheel: started 102 | Getting requirements to build wheel: finished with status 'done' 103 | Preparing metadata (pyproject.toml): started 104 | Preparing metadata (pyproject.toml): finished with status 'done' 105 | Collecting boto3==1.18.56 106 | Downloading boto3-1.18.56-py3-none-any.whl (131 kB) 107 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.6/131.6 kB 33.5 MB/s eta 0:00:00 108 | Collecting requests==2.26.0 109 | Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 110 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.3/62.3 kB 20.7 MB/s eta 0:00:00 111 | Collecting botocore<1.22.0,>=1.21.56 (from boto3==1.18.56) 112 | Downloading botocore-1.21.65-py3-none-any.whl (8.0 MB) 113 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 126.1 MB/s eta 0:00:00 114 | Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.18.56) 115 | Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 116 | Collecting s3transfer<0.6.0,>=0.5.0 (from boto3==1.18.56) 117 | Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 118 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.5/79.5 kB 24.6 MB/s eta 0:00:00 119 | Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) 120 | Obtaining dependency information for urllib3<1.27,>=1.21.1 from https://files.pythonhosted.org/packages/b0/53/aa91e163dcfd1e5b82d8a890ecf13314e3e149c05270cc644581f77f17fd/urllib3-1.26.18-py2.py3-none-any.whl.metadata 121 | Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 122 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 9.5 MB/s eta 0:00:00 123 | Collecting certifi>=2017.4.17 (from requests==2.26.0) 124 | Obtaining dependency information for certifi>=2017.4.17 from https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl.metadata 125 | Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB) 126 | Collecting charset-normalizer~=2.0.0 (from requests==2.26.0) 127 | Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 128 | Collecting idna<4,>=2.5 (from requests==2.26.0) 129 | Obtaining dependency information for idna<4,>=2.5 from https://files.pythonhosted.org/packages/c2/e7/a82b05cf63a603df6e68d59ae6a68bf5064484a0718ea5033660af4b54a9/idna-3.6-py3-none-any.whl.metadata 130 | Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB) 131 | Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.22.0,>=1.21.56->boto3==1.18.56) 132 | Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 133 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 51.6 MB/s eta 0:00:00 134 | Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.22.0,>=1.21.56->boto3==1.18.56) 135 | Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 136 | Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 137 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 33.2 MB/s eta 0:00:00 138 | Downloading idna-3.6-py3-none-any.whl (61 kB) 139 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 20.0 MB/s eta 0:00:00 140 | Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 141 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.8/143.8 kB 37.5 MB/s eta 0:00:00 142 | Building wheels for collected packages: MarkupSafe 143 | Building wheel for MarkupSafe (pyproject.toml): started 144 | Building wheel for MarkupSafe (pyproject.toml): finished with status 'done' 145 | Created wheel for MarkupSafe: filename=MarkupSafe-2.0.1-cp311-cp311-linux_x86_64.whl size=28257 sha256=1ffc0b2cff31e42028d6042766427958c333d545752252f712a75a29b01c087f 146 | Stored in directory: /root/.cache/pip/wheels/ea/18/79/6266ea508b8164a77b95aa19534c77eb805f2878612c37efca 147 | Successfully built MarkupSafe 148 | Installing collected packages: urllib3, six, MarkupSafe, jmespath, idna, charset-normalizer, certifi, requests, python-dateutil, Jinja2, jinja2-cli, botocore, s3transfer, boto3 149 | Successfully installed Jinja2-3.0.1 MarkupSafe-2.0.1 boto3-1.18.56 botocore-1.21.65 certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.6 jinja2-cli-0.7.0 jmespath-0.10.0 python-dateutil-2.8.2 requests-2.26.0 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.18 150 |   151 | [notice] A new release of pip is available: 23.2.1 -> 24.0 152 | [notice] To update, run: pip install --upgrade pip 153 |   154 | [Container] 2024/02/04 21:25:43.212502 Running command if [ $TF_DISTRIBUTION = "oss" ]; then 155 | TF_BACKEND_REGION=$(aws ssm get-parameter --name "/aft/config/oss-backend/primary-region" --query "Parameter.Value" --output text) 156 | TF_KMS_KEY_ID=$(aws ssm get-parameter --name "/aft/config/oss-backend/kms-key-id" --query "Parameter.Value" --output text) 157 | TF_DDB_TABLE=$(aws ssm get-parameter --name "/aft/config/oss-backend/table-id" --query "Parameter.Value" --output text) 158 | TF_S3_BUCKET=$(aws ssm get-parameter --name "/aft/config/oss-backend/bucket-id" --query "Parameter.Value" --output text) 159 | TF_S3_KEY=account-request/terraform.tfstate 160 | cd /tmp 161 | echo "Installing Terraform" 162 | curl -o terraform_${TF_VERSION}_linux_amd64.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip 163 | unzip -o terraform_${TF_VERSION}_linux_amd64.zip && mv terraform /usr/bin 164 | terraform --version 165 | cd $DEFAULT_PATH/terraform 166 | for f in *.jinja; do jinja2 $f -D timestamp="$TIMESTAMP" -D tf_distribution_type=$TF_DISTRIBUTION -D provider_region=$CT_MGMT_REGION -D region=$TF_BACKEND_REGION -D bucket=$TF_S3_BUCKET -D key=$TF_S3_KEY -D dynamodb_table=$TF_DDB_TABLE -D kms_key_id=$TF_KMS_KEY_ID -D aft_admin_role_arn=$AFT_EXEC_ROLE_ARN -D tf_version=$TF_VERSION >> ./$(basename $f .jinja).tf; done 167 | for f in *.tf; do echo "\n \n"; echo $f; cat $f; done 168 | JSON=$(aws sts assume-role --role-arn ${AFT_ADMIN_ROLE_ARN} --role-session-name ${ROLE_SESSION_NAME}) 169 | #Make newly assumed role default session 170 | export AWS_ACCESS_KEY_ID=$(echo ${JSON} \| jq --raw-output ".Credentials[\"AccessKeyId\"]") 171 | export AWS_SECRET_ACCESS_KEY=$(echo ${JSON} \| jq --raw-output ".Credentials[\"SecretAccessKey\"]") 172 | export AWS_SESSION_TOKEN=$(echo ${JSON} \| jq --raw-output ".Credentials[\"SessionToken\"]") 173 | terraform init -no-color 174 | else 175 | TF_ORG_NAME=$(aws ssm get-parameter --name "/aft/config/terraform/org-name" --query "Parameter.Value" --output text) 176 | TF_TOKEN=$(aws ssm get-parameter --name "/aft/config/terraform/token" --with-decryption --query "Parameter.Value" --output text) 177 | TF_ENDPOINT=$(aws ssm get-parameter --name "/aft/config/terraform/api-endpoint" --query "Parameter.Value" --output text) 178 | TF_WORKSPACE_NAME="ct-aft-account-request" 179 | TF_CONFIG_PATH="./temp_configuration_file.tar.gz" 180 | cd $DEFAULT_PATH/terraform 181 | for f in *.jinja; do jinja2 $f -D timestamp="$TIMESTAMP" -D provider_region=$CT_MGMT_REGION -D tf_distribution_type=$TF_DISTRIBUTION -D terraform_org_name=$TF_ORG_NAME -D terraform_workspace_name=$TF_WORKSPACE_NAME -D aft_admin_role_arn=$AFT_EXEC_ROLE_ARN >> ./$(basename $f .jinja).tf; done 182 | for f in *.tf; do echo "\n \n"; echo $f; cat $f; done 183 | cd $DEFAULT_PATH 184 | tar -czf temp_configuration_file.tar.gz -C terraform --exclude .git --exclude venv . 185 | python3 $DEFAULT_PATH/aws-aft-core-framework/sources/scripts/workspace_manager.py --operation "deploy" --organization_name $TF_ORG_NAME --workspace_name $TF_WORKSPACE_NAME --assume_role_arn $AFT_ADMIN_ROLE_ARN --assume_role_session_name $ROLE_SESSION_NAME --api_endpoint $TF_ENDPOINT --api_token $TF_TOKEN --terraform_version $TF_VERSION --config_file $TF_CONFIG_PATH 186 | fi 187 |   188 | \n \n 189 | aft-providers.tf 190 | ## Auto generated providers.tf ## 191 | ## Updated on: 2024-02-04 21:25:02 ## 192 |   193 | provider "aws" { 194 | region = "us-east-1" 195 | assume_role { 196 | role_arn = "arn:aws:iam::113634643425:role/AWSAFTExecution" 197 | } 198 | default_tags { 199 | tags = { 200 | managed_by = "AFT" 201 | } 202 | } 203 | } 204 | \n \n 205 | backend.tf 206 | ## Auto generated backend.tf ## 207 | ## Updated on: 2024-02-04 21:25:02 ## 208 |   209 | terraform { 210 | backend "remote" { 211 | organization = "HNO" 212 | workspaces { 213 | name = "ct-aft-account-request" 214 | } 215 | } 216 | } 217 | \n \n 218 | sandbox-accounts.tf 219 | module "sandbox_account_01" { 220 | source = "./modules/aft-account-request" 221 |   222 | control_tower_parameters = { 223 | AccountEmail = "aakarnati+sandbox01@gmail.com" 224 | AccountName = "sandbox-account-01" 225 | ManagedOrganizationalUnit = "Learn AFT" 226 | SSOUserEmail = "aakarnati+sandbox01@gmail.com" 227 | SSOUserFirstName = "Sandbox" 228 | SSOUserLastName = "User" 229 | } 230 |   231 | account_tags = { 232 | "BOOTCAMP:Owner" = "aakarnati@gmail.com" 233 | "BOOTCAMP:Environment" = "Dev" 234 | "BOOTCAMP:Project" = "AFT Tutorial" 235 | } 236 |   237 | change_management_parameters = { 238 | change_requested_by = "BOOTCAMP" 239 | change_reason = "BOOTCAMP AWS Control Tower Account Factory for Terraform" 240 | } 241 |   242 | custom_fields = { 243 | group = "non-prod" 244 | } 245 |   246 | account_customizations_name = "sandbox-customizations" 247 | } 248 | Successfully created workspace ct-aft-account-request with ID ws-LqBcpPpaSmsHdJ2m 249 | Successfully placed AWS credentials on workspace for arn:aws:iam::113634643425:role/AWSAFTAdmin 250 | Handling errors: [{'status': '404', 'title': 'not found'}] 251 | Traceback (most recent call last): 252 | File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/workspace_manager.py", line 275, in 253 | setup_and_run_workspace( 254 | File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/workspace_manager.py", line 22, in setup_and_run_workspace 255 | run_id = stage_run(workspace_id, assume_role_arn, role_session_name, api_token) 256 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 257 | File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/workspace_manager.py", line 48, in stage_run 258 | cv_id, upload_url = terraform.create_configuration_version(workspace_id, api_token) 259 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 260 | File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/terraform_client.py", line 74, in create_configuration_version 261 | response = __post(endpoint, headers, payload) 262 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 263 | File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/terraform_client.py", line 215, in __post 264 | __handle_errors(response) 265 | File "/codebuild/output/src202587511/src/aws-aft-core-framework/sources/scripts/terraform_client.py", line 253, in __handle_errors 266 | raise ClientError(status=error["status"], message=error["title"]) 267 | terraform_client.ClientError: not found 268 |   269 | [Container] 2024/02/04 21:25:46.375008 Command did not exit successfully if [ $TF_DISTRIBUTION = "oss" ]; then 270 | TF_BACKEND_REGION=$(aws ssm get-parameter --name "/aft/config/oss-backend/primary-region" --query "Parameter.Value" --output text) 271 | TF_KMS_KEY_ID=$(aws ssm get-parameter --name "/aft/config/oss-backend/kms-key-id" --query "Parameter.Value" --output text) 272 | TF_DDB_TABLE=$(aws ssm get-parameter --name "/aft/config/oss-backend/table-id" --query "Parameter.Value" --output text) 273 | TF_S3_BUCKET=$(aws ssm get-parameter --name "/aft/config/oss-backend/bucket-id" --query "Parameter.Value" --output text) 274 | TF_S3_KEY=account-request/terraform.tfstate 275 | cd /tmp 276 | echo "Installing Terraform" 277 | curl -o terraform_${TF_VERSION}_linux_amd64.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip 278 | unzip -o terraform_${TF_VERSION}_linux_amd64.zip && mv terraform /usr/bin 279 | terraform --version 280 | cd $DEFAULT_PATH/terraform 281 | for f in *.jinja; do jinja2 $f -D timestamp="$TIMESTAMP" -D tf_distribution_type=$TF_DISTRIBUTION -D provider_region=$CT_MGMT_REGION -D region=$TF_BACKEND_REGION -D bucket=$TF_S3_BUCKET -D key=$TF_S3_KEY -D dynamodb_table=$TF_DDB_TABLE -D kms_key_id=$TF_KMS_KEY_ID -D aft_admin_role_arn=$AFT_EXEC_ROLE_ARN -D tf_version=$TF_VERSION >> ./$(basename $f .jinja).tf; done 282 | for f in *.tf; do echo "\n \n"; echo $f; cat $f; done 283 | JSON=$(aws sts assume-role --role-arn ${AFT_ADMIN_ROLE_ARN} --role-session-name ${ROLE_SESSION_NAME}) 284 | #Make newly assumed role default session 285 | export AWS_ACCESS_KEY_ID=$(echo ${JSON} \| jq --raw-output ".Credentials[\"AccessKeyId\"]") 286 | export AWS_SECRET_ACCESS_KEY=$(echo ${JSON} \| jq --raw-output ".Credentials[\"SecretAccessKey\"]") 287 | export AWS_SESSION_TOKEN=$(echo ${JSON} \| jq --raw-output ".Credentials[\"SessionToken\"]") 288 | terraform init -no-color 289 | else 290 | TF_ORG_NAME=$(aws ssm get-parameter --name "/aft/config/terraform/org-name" --query "Parameter.Value" --output text) 291 | TF_TOKEN=$(aws ssm get-parameter --name "/aft/config/terraform/token" --with-decryption --query "Parameter.Value" --output text) 292 | TF_ENDPOINT=$(aws ssm get-parameter --name "/aft/config/terraform/api-endpoint" --query "Parameter.Value" --output text) 293 | TF_WORKSPACE_NAME="ct-aft-account-request" 294 | TF_CONFIG_PATH="./temp_configuration_file.tar.gz" 295 | cd $DEFAULT_PATH/terraform 296 | for f in *.jinja; do jinja2 $f -D timestamp="$TIMESTAMP" -D provider_region=$CT_MGMT_REGION -D tf_distribution_type=$TF_DISTRIBUTION -D terraform_org_name=$TF_ORG_NAME -D terraform_workspace_name=$TF_WORKSPACE_NAME -D aft_admin_role_arn=$AFT_EXEC_ROLE_ARN >> ./$(basename $f .jinja).tf; done 297 | for f in *.tf; do echo "\n \n"; echo $f; cat $f; done 298 | cd $DEFAULT_PATH 299 | tar -czf temp_configuration_file.tar.gz -C terraform --exclude .git --exclude venv . 300 | python3 $DEFAULT_PATH/aws-aft-core-framework/sources/scripts/workspace_manager.py --operation "deploy" --organization_name $TF_ORG_NAME --workspace_name $TF_WORKSPACE_NAME --assume_role_arn $AFT_ADMIN_ROLE_ARN --assume_role_session_name $ROLE_SESSION_NAME --api_endpoint $TF_ENDPOINT --api_token $TF_TOKEN --terraform_version $TF_VERSION --config_file $TF_CONFIG_PATH 301 | fi 302 | exit status 1 303 | [Container] 2024/02/04 21:25:46.380135 Phase complete: PRE_BUILD State: FAILED 304 | [Container] 2024/02/04 21:25:46.380152 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: if [ $TF_DISTRIBUTION = "oss" ]; then 305 | TF_BACKEND_REGION=$(aws ssm get-parameter --name "/aft/config/oss-backend/primary-region" --query "Parameter.Value" --output text) 306 | TF_KMS_KEY_ID=$(aws ssm get-parameter --name "/aft/config/oss-backend/kms-key-id" --query "Parameter.Value" --output text) 307 | TF_DDB_TABLE=$(aws ssm get-parameter --name "/aft/config/oss-backend/table-id" --query "Parameter.Value" --output text) 308 | TF_S3_BUCKET=$(aws ssm get-parameter --name "/aft/config/oss-backend/bucket-id" --query "Parameter.Value" --output text) 309 | TF_S3_KEY=account-request/terraform.tfstate 310 | cd /tmp 311 | echo "Installing Terraform" 312 | curl -o terraform_${TF_VERSION}_linux_amd64.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip 313 | unzip -o terraform_${TF_VERSION}_linux_amd64.zip && mv terraform /usr/bin 314 | terraform --version 315 | cd $DEFAULT_PATH/terraform 316 | for f in *.jinja; do jinja2 $f -D timestamp="$TIMESTAMP" -D tf_distribution_type=$TF_DISTRIBUTION -D provider_region=$CT_MGMT_REGION -D region=$TF_BACKEND_REGION -D bucket=$TF_S3_BUCKET -D key=$TF_S3_KEY -D dynamodb_table=$TF_DDB_TABLE -D kms_key_id=$TF_KMS_KEY_ID -D aft_admin_role_arn=$AFT_EXEC_ROLE_ARN -D tf_version=$TF_VERSION >> ./$(basename $f .jinja).tf; done 317 | for f in *.tf; do echo "\n \n"; echo $f; cat $f; done 318 | JSON=$(aws sts assume-role --role-arn ${AFT_ADMIN_ROLE_ARN} --role-session-name ${ROLE_SESSION_NAME}) 319 | #Make newly assumed role default session 320 | export AWS_ACCESS_KEY_ID=$(echo ${JSON} \| jq --raw-output ".Credentials[\"AccessKeyId\"]") 321 | export AWS_SECRET_ACCESS_KEY=$(echo ${JSON} \| jq --raw-output ".Credentials[\"SecretAccessKey\"]") 322 | export AWS_SESSION_TOKEN=$(echo ${JSON} \| jq --raw-output ".Credentials[\"SessionToken\"]") 323 | terraform init -no-color 324 | else 325 | TF_ORG_NAME=$(aws ssm get-parameter --name "/aft/config/terraform/org-name" --query "Parameter.Value" --output text) 326 | TF_TOKEN=$(aws ssm get-parameter --name "/aft/config/terraform/token" --with-decryption --query "Parameter.Value" --output text) 327 | TF_ENDPOINT=$(aws ssm get-parameter --name "/aft/config/terraform/api-endpoint" --query "Parameter.Value" --output text) 328 | TF_WORKSPACE_NAME="ct-aft-account-request" 329 | TF_CONFIG_PATH="./temp_configuration_file.tar.gz" 330 | cd $DEFAULT_PATH/terraform 331 | for f in *.jinja; do jinja2 $f -D timestamp="$TIMESTAMP" -D provider_region=$CT_MGMT_REGION -D tf_distribution_type=$TF_DISTRIBUTION -D terraform_org_name=$TF_ORG_NAME -D terraform_workspace_name=$TF_WORKSPACE_NAME -D aft_admin_role_arn=$AFT_EXEC_ROLE_ARN >> ./$(basename $f .jinja).tf; done 332 | for f in *.tf; do echo "\n \n"; echo $f; cat $f; done 333 | cd $DEFAULT_PATH 334 | tar -czf temp_configuration_file.tar.gz -C terraform --exclude .git --exclude venv . 335 | python3 $DEFAULT_PATH/aws-aft-core-framework/sources/scripts/workspace_manager.py --operation "deploy" --organization_name $TF_ORG_NAME --workspace_name $TF_WORKSPACE_NAME --assume_role_arn $AFT_ADMIN_ROLE_ARN --assume_role_session_name $ROLE_SESSION_NAME --api_endpoint $TF_ENDPOINT --api_token $TF_TOKEN --terraform_version $TF_VERSION --config_file $TF_CONFIG_PATH 336 | fi 337 | . Reason: exit status 1
snebhu3 commented 4 months ago

@Bckarnati2018 thank you for reaching out. From the error logs it seems the failure occurs during create_configuration_version. You could ensure that the api_token being used is valid and has the right permissions. I would recommend reaching out to AWS premium support for help in deep diving the issue in the environment.