aws cloudformation deploy s3-bucket should work for different regions

[badaldavda]

Now when using aws cloudformation deploy with s3-bucket flag - CLI does these operations -

• Redirect to other endpoint.

  -------
  Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x105152450>>
  ------

• PutObject - to url like - https://abcbucket123.s3.us-west-2.amazonaws.com -Works
• CreateChangeSet - 'TemplateURL': 'https://s3-ap-south-1.amazonaws.com/abcbucket123/2be6ca6a006bd83e306ae0f307076535.template'

In my tests, I found that if we use - create-change-set with template-url like https://abcbucket123.s3.amazonaws.com/newtest2.yaml

use template-url as https://.s3.amazonaws.com instead of https://s3-.amazonaws.com/ which will enable the use of same bucket for different regions in future.

This will mean that even PutObject is being done in correct bucket and even CreateChangeSet from the same bucket.

This will enable use of just single bucket in single region for multiple stacks in different region.

[jamesls]

Just adding a bit more detail here. The issue appears to be that when auto-populating the TemplateURL parameter for the underlying create_change_set operation, we use path style addressing for the S3 object. The comment in the code suggests that you have to use path style addressing:

https://github.com/aws/aws-cli/blob/a53dd3a87b6b0bed14882c6f644781e99c37e0ab/awscli/customizations/cloudformation/deployer.py#L130-L132

In this case we'll use whatever region you have configured. This will fail if your bucket is in a different region:

<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">
  <Error>
    <Type>Sender</Type>
    <Code>ValidationError</Code>
    <Message>S3 error: Unable to get the object https://s3.us-west-1.amazonaws.com/<my-bucket-in-us-west-2>/0ec60de5986a0c04cf1a90efcc10a296.template</Message>
  </Error>
  <RequestId>43f13b59-b883-11e9-aab2-874a0bd414ac</RequestId>
</ErrorResponse>

However, I've confirmed that if you use <bucket>.s3.amazonaws.com format, then you can specify s3 buckets that are in a different region than your stack.

The docs don't mention the path-style restriction so perhaps this comment is no longer accurate.

That being said, switching to virtual hosted addressing isn't always possible so there's some edge cases to work through.

I'd like to get more feedback from others on whether or not we'd want to make this change.

cc @sanathkr

[manindravarma]

When I tried to use "aws cloudformation deploy" command using .s3.amazonaws.com format for bucket, it failed.

aws cloudformation deploy --template-file Securitygroup.json --s3-bucket test-ap-south-1-xxxx.s3.amazonaws.com --stack-name tests3 --region ap-southeast-2

S3 Bucket does not exist. Execute the command to create a new bucket aws s3 mb s3://test-ap-south-1-xxxx.s3.amazonaws.com

Is this the right format that you suggested for using cross-region s3 bucket for "aws cloudformation deploy" command.

If so, please let me know what I am doing wrong with the above command.

Note: S3 bucket I am using exists and when I use "aws cloudformation deploy --template-file Securitygroup.json --s3-bucket test-ap-south-1-xxxx --stack-name tests3 --region ap-southeast-2" command, I was able to see that the file is being uploaded to the S3 bucket but fails while creating change set.

aws cloudformation deploy --template-file Securitygroup.json --s3-bucket test-ap-south-1-xxxx --stack-name tests3 --region ap-southeast-2 Uploading to e5acf580b1396e83722e686fb6363532.template 502 / 502.0 (100.00%) An error occurred (ValidationError) when calling the CreateChangeSet operation: S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

[badaldavda]

@manindravarma we need not use test-ap-south-1-xxxx.s3.amazonaws.com in s3-bucket flag but just s3 bucket name.

Since the bucket is in ap-south-1 and you are trying to deploy in ap-southeast-2, it is failing. Due to the reason I created this issue.

[nhakmiller]

I know this issue is a bit old, but we just ran into it ourselves. In our case it manifests when a user has one region defined in their .aws/config file, but specifies a different region with either environment variables or flags while running the aws cloudformation package command. Although I believe it is the same issue, the TemplateURL is constructed incorrectly, using the region specified in the users .aws/config instead of the region specified in the environment variables or flags.

We worked around this by adding some post processing to every CloudFormation template we construct based on @jamesls comment and re-writing every TemplateURL from the https://s3.region.amazonaws.com/bucket/path/1.template format to https://bucket.s3.amazonaws.com/path/1.template. It works for us for now, but just wondering if this issue is still being worked on? Or if there is a workaround to this?

[penguinpowernz]

I thought S3 was supposed to be regionless? Why can't the cloudformation command just lookup the correct location of the bucket?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

[kdaily]

Bumping this to keep it open.

[julienbonastre]

Yes, experiencing the same issue, tried all variations of path I have seen throughout these posts and others to no avail..

How is this still an issue and not addressed yet? 🤪 🙃 🤷🏻‍♂️

[codechrysalis-doug]

I have to deploy my ACM Cert into us-east-1, but my bucket for templates is in a different region. Would really love this fixed eventually.

[drAlberT]

It's a P3 .. not for ppl not being able to deploy a stack ;(

Any chance to have this bug addressed?

[apogrebnyak]

For me specifying the S3 endpoint URL in the correct region worked.

export AWS_ENDPOINT_URL_S3="https://s3.us-west-2.amazonaws.com"

aws cloudformation deploy --region ap-south-1 --s3-bucket my-bucket-in-us-west-2 ...

If you don't want to fix it, it's OK, but at least document this behavior.