aws-solutions / aws-crr-monitor

A solution for near real-time monitoring of replication of objects in Amazon S3 between a source bucket and a destination bucket across multiple regions.
https://aws.amazon.com/solutions/cross-region-replication-monitor
Apache License 2.0
39 stars 16 forks source link

Issue with cross region & cross account replication #4

Closed mingzhu-abb closed 5 years ago

mingzhu-abb commented 6 years ago

I encountered an error deploying this stack. Here's our setup:

  1. Source bucket bucket1 is in our Prod AWS account, with home region us-east-1.

  2. Dest bucket bucket1-dest is in our Disaster Recovery aws account, with home region us-west-2.

  3. We have setup cross region & cross account replication from bucket1 to bucket1-dest.

When I create the stack, it fails in Custom::DeployAgent with following error log:

An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
[ERROR] 2018-09-10T21:27:05.46Z 265a5aaa-b540-11e8-ba4c-09fc0e12c98a    Failed to execute resource function
Traceback (most recent call last):
File "/var/task/cfn_resource.py", line 43, in wrapper_func
response.update(func(event, context))
File "/var/task/CRRdeployagent.py", line 89, in create_agent
agent_regions = get_agent_regions()
File "/var/task/CRRdeployagent.py", line 72, in get_agent_regions
raise e
ClientError: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
[INFO]  2018-09-10T21:27:05.46Z 265a5aaa-b540-11e8-ba4c-09fc0e12c98a    Responding to 'Create' request with:
{
    "StackId": "arn:aws:cloudformation:us-east-1:172631448019:stack/S3CRRMonitor201809101425/0f6d5d00-b540-11e8-b4b3-50fae98974c5",
    "Status": "FAILED",
    "Reason": "Exception was raised while handling custom resource",
    "RequestId": "4fc165d5-0ae7-49ab-9999-64c5968d7a1e",
    "LogicalResourceId": "CustomDeploy"
}

After debugging, it turns out to be caused by some fault in GetBucketLocation:

  1. The stack is setup in us-east-1, so the following Lambda code creates an S3 client with us-east-1 as default region:

    client = boto3.client('s3')
  2. When it tries to call GetBucketLocation on the dest bucket, which is created in us-west-2, S3 returns the above error.

I googled this issue, and found an work around solution by using HeadBucket to get bucket location, which worked well. Please also fix this issue here, so that other people could benefit.

Thanks,

hnishar commented 6 years ago

mingzhu-abb@, Please submit the PR with the fix.

hnishar commented 6 years ago

mingzhu-abb@ The solution does not support Cross-Account i.e. source and destination buckets should be in same AWS account, as noted in the documentation "Once deployed, the CRR Monitor applies the appropriate configuration for monitoring the replicated Amazon S3 buckets across all AWS Regions in a single account."

https://docs.aws.amazon.com/solutions/latest/crr-monitor/considerations.html

We have a pending feature request to update the solution to support Cross-Account scenarios.

mingzhu-abb commented 6 years ago

Please check PR #5 . Would you mind merging this PR , even though you don't support x-account yet? I think my PR is harmless after all.

groverlalit commented 5 years ago

The pull request was accepted and will be part of the next release. @mingzhu-abb It would be great if we can close this ticket and track it via the pull request ticket. Thanks.

georgebearden commented 5 years ago

Closing this issue as the latest update contains this PR.