search

aws-samples / aws-service-catalog-terraform-reference-architecture

Apply Terraform configurations using CloudFormation through a proxy lambda
Apache License 2.0
134 stars 70 forks source link

When trying to provision the product through service catalog, even after product gets provisioned status on service catalog and cloud formation stays CREATE_IN_PROGRESS (later changes to ROLLBACK_IN_PROGRESS) #16

Closed mejuhi closed 4 years ago

mejuhi commented 4 years ago

After deploying the infrastructure, and then provisioning a product (using end user) via service catalog, I can see the provisioned product using admin's console but the end user doesnt receive the success message since both service catalog and cloud formation show status as "CREATE_IN_PROGRESS" (which later changes to "ROLLBACK_IN_PROGRESS")

Following is the error stack trace received

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/__main__.py", line 240, in main
    run(cleanups, args, args.request, config, s3, response_poster)
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/__main__.py", line 216, in run
    state_file_location=state_file_location)
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/response_poster.py", line 39, in post_response_with_expiration_check
    state_file_location=state_file_location, reason=reason)
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/response_poster.py", line 44, in _post_response
    output_url = self.create_proxy_object()
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/response_poster.py", line 80, in create_proxy_object
    WebsiteRedirectLocation=presigned_url
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sc-terraform-wrapper", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/__main__.py", line 243, in main
    response_poster.post_response_with_expiration_check('FAILED', reason=msg)
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/response_poster.py", line 39, in post_response_with_expiration_check
    state_file_location=state_file_location, reason=reason)
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/response_poster.py", line 44, in _post_response
    output_url = self.create_proxy_object()
  File "/usr/local/lib/python3.7/site-packages/sc_terraform_wrapper/response_poster.py", line 80, in create_proxy_object
    WebsiteRedirectLocation=presigned_url
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
failed to run commands: exit status 1

Also going through stdout to find details i get following 

Attempt to load configuration at: /usr/local/var/sc-config.json
Creating workspace
Downloading artifact file
Writing backend configuration to file
Creating AWS provider override file
Writing variables to file
Starting Terraform execution
Tagging resources with tags: {'Name': 'trial21', 'CfnStackId': 'arn:aws:cloudformation:us-west-2:<<MyAccountID>>:stack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52', 'TfResourceGroupName': 'SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc'}
Tagging try #1. Attempt to tag ARNs: ['arn:aws:sqs:us-west-2:<<MyAccountID>>:trial21.fifo']
Creating resource group if not exist
Created resource group: {'ResponseMetadata': {'RequestId': 'c4cc57dd-d7d6-44ab-a782-0a49f2b80aa5', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 10 Dec 2019 05:54:06 GMT', 'content-type': 'application/json', 'content-length': '994', 'connection': 'keep-alive', 'x-amzn-requestid': 'c4cc57dd-d7d6-44ab-a782-0a49f2b80aa5', 'x-amz-apigw-id': 'EeTnxFaGvHcFsuQ=', 'x-amzn-trace-id': 'Root=1-5def32fe-ee9600089e7e5bf06419bbbc;Sampled=0'}, 'RetryAttempts': 0}, 'Group': {'GroupArn': 'arn:aws:resource-groups:us-west-2:<<MyAccountID>>:group/SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc', 'Name': 'SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc', 'Description': 'Auto-created from Terraform wrapper script'}, 'ResourceQuery': {'Type': 'TAG_FILTERS_1_0', 'Query': '{"ResourceTypeFilters": ["AWS::AllSupported"], "TagFilters": [{"Key": "TfResourceGroupName", "Values": ["SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc"]}]}'}, 'Tags': {'Name': 'trial21', 'CfnStackId': 'arn:aws:cloudformation:us-west-2:<<MyAccountID>>:stack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52', 'TfResourceGroupName': 'SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc'}}
Posting SUCCESS response to https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3A<<MyAccountID>>%3Astack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52%7CMyTerraformStack%7Ce9f9f920-4d2b-4b66-b4ca-fce4ac306b60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20191210T055333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SAVTEM6XA%2F20191210%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=31b97d9a634a137ec23206394437adc36c66f9bed668cfca869a66347b673865
Posting FAILED response to https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3A<<MyAccountID>>%3Astack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52%7CMyTerraformStack%7Ce9f9f920-4d2b-4b66-b4ca-fce4ac306b60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20191210T055333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SAVTEM6XA%2F20191210%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=31b97d9a634a137ec23206394437adc36c66f9bed668cfca869a66347b673865
Remove workspace

Curious about why it is trying to post on "cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com" bucket, since it is not in my account (or mentioned anywhere in code), and if at all this bucket is causing the error. Moreover reason for two "SUCCESS" & "FAILED" response to that bucket

chapmancl commented 4 years ago

cloudformation-custom-resource-response is not a bucket, you are looking at the stack of the custom resource lambda which is not the error. check the output in the tf_wrapper_script_output for why terraform cannot create the resources. the cloud formation stack resource "MyTerraformStack" has a status reason which will have the full S3 path to the output.

What TF file are you trying to run? make sure the TerraformResourceCreationRole in IAM has permission to create the resources in that TF file.

chapmancl commented 4 years ago

duplicate of #10

mejuhi commented 4 years ago

I am trying to provision SQS, and made sue that role "TerraformResourceCreationRole" has adequate permission, i already checked the contents of "tf_wrapper_script_output", terraform is able to create the resource (provisioned resource can be seen from admin's AWS console as well)

Attaching contents of tf_wrapper_script_output file 

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (2.41.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.41"

Terraform has been successfully initialized!
aws_sqs_queue.terraform_queue: Creating...
  arn:                               "" => "<computed>"
  content_based_deduplication:       "" => "true"
  delay_seconds:                     "" => "0"
  fifo_queue:                        "" => "true"
  kms_data_key_reuse_period_seconds: "" => "<computed>"
  max_message_size:                  "" => "262144"
  message_retention_seconds:         "" => "345600"
  name:                              "" => "trial21.fifo"
  policy:                            "" => "<computed>"
  receive_wait_time_seconds:         "" => "0"
  visibility_timeout_seconds:        "" => "30"
aws_sqs_queue.terraform_queue: Creation complete after 0s (ID: https://sqs.us-west-2.amazonaws.com/<<MyAccountID>>/trial21.fifo)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

queue = SQS queue created trial21.fifo

==========TERRAFORM WRAPPER SCRIPT OUTPUT==========
Attempt to load configuration at: /usr/local/var/sc-config.json
Creating workspace
Downloading artifact file
Writing backend configuration to file
Creating AWS provider override file
Writing variables to file
Starting Terraform execution
Tagging resources with tags: {'Name': 'trial21', 'CfnStackId': 'arn:aws:cloudformation:us-west-2:<<MyAccountID>>:stack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52', 'TfResourceGroupName': 'SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc'}
Tagging try #1. Attempt to tag ARNs: ['arn:aws:sqs:us-west-2:<<MyAccountID>>:trial21.fifo']
Creating resource group if not exist
Created resource group: {'ResponseMetadata': {'RequestId': 'c4cc57dd-d7d6-44ab-a782-0a49f2b80aa5', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Tue, 10 Dec 2019 05:54:06 GMT', 'content-type': 'application/json', 'content-length': '994', 'connection': 'keep-alive', 'x-amzn-requestid': 'c4cc57dd-d7d6-44ab-a782-0a49f2b80aa5', 'x-amz-apigw-id': 'EeTnxFaGvHcFsuQ=', 'x-amzn-trace-id': 'Root=1-5def32fe-ee9600089e7e5bf06419bbbc;Sampled=0'}, 'RetryAttempts': 0}, 'Group': {'GroupArn': 'arn:aws:resource-groups:us-west-2:<<MyAccountID>>:group/SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc', 'Name': 'SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc', 'Description': 'Auto-created from Terraform wrapper script'}, 'ResourceQuery': {'Type': 'TAG_FILTERS_1_0', 'Query': '{"ResourceTypeFilters": ["AWS::AllSupported"], "TagFilters": [{"Key": "TfResourceGroupName", "Values": ["SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc"]}]}'}, 'Tags': {'Name': 'trial21', 'CfnStackId': 'arn:aws:cloudformation:us-west-2:<<MyAccountID>>:stack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52', 'TfResourceGroupName': 'SC-<<MyAccountID>>-pp-spkckdpjc3mmk-MyTerraformStack-03192b6e60d98c9c066769b0214eb57392de066f88c9ac888ad133b1006206bc'}}
Posting SUCCESS response to https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3A<<MyAccountID>>%3Astack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52%7CMyTerraformStack%7Ce9f9f920-4d2b-4b66-b4ca-fce4ac306b60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20191210T055333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SAVTEM6XA%2F20191210%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=31b97d9a634a137ec23206394437adc36c66f9bed668cfca869a66347b673865
Posting FAILED response to https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3A<<MyAccountID>>%3Astack/SC-<<MyAccountID>>-pp-spkckdpjc3mmk/64352560-1b11-11ea-a978-02b749140c52%7CMyTerraformStack%7Ce9f9f920-4d2b-4b66-b4ca-fce4ac306b60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20191210T055333Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SAVTEM6XA%2F20191210%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=31b97d9a634a137ec23206394437adc36c66f9bed668cfca869a66347b673865
Remove workspace