aws-samples / sync-buckets-state-machine

A sample AWS Step Functions (SFN) state machine, designed to one-way synchronize an Amazon S3 source bucket into another S3 destination bucket.
Apache License 2.0
111 stars 43 forks source link

Cross-region migration not supported #6

Open schainks opened 7 years ago

schainks commented 7 years ago

It wasn't clear to me in the docs, so filing an issue. I'm still learning the ins and outs of lambda, but what is the limitation for this tool to not work cross-region? Will gladly volunteer a PR to make this work :)

glez-aws commented 7 years ago

Hi, and thanks for filing!

Cross-region requires different code in the two lambda functions for copying/deleting files and handling of the separate regions. Not a huge effort, but still something to implement and test carefully, so I've postponed this to be done at a later time.

As soon as I find some time, I'll work on adding it.

Thanks, Constantin

schainks commented 7 years ago

Constantin, Thanks for the clear reply and dedicated support! I'm also following this issue issue from awscli about preserving ACL as part of the copy. How would the lambda function also be built to ensure we preserve object ACL on copy?

glez-aws commented 7 years ago

Hi David,

you can read out the ACL information from the source and the (potentially existing) object with the read_object_acl() call in Python, compare the two, then use the put_object_acl() call to set it on the destination object if different. See:

http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.get_object_acl http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.put_object_acl

I've created a new issue (https://github.com/awslabs/sync-buckets-state-machine/issues/8) to track this independently.

Thanks, Constantin

schainks commented 7 years ago

Thanks Constantin for the API refresher. Will continue studying your lambda state machine!

yunw commented 6 years ago

Hi Constantin, I made the change to support multi-region: https://github.com/yunw/sync-buckets-state-machine/commit/e2a0f9f762f4af22693c6adc654b0e648778c5d1

Can you review?