search

aws-solutions / data-transfer-hub

Seamless User Interface for replicating data into AWS.
https://aws-solutions.github.io/data-transfer-hub/en/
Apache License 2.0
136 stars 22 forks source link

KMS-encrypted S3 Transfers with S3 Events enabled not possible with IAM user #113

Open kaplanan opened 1 year ago

kaplanan commented 1 year ago

Use case: Using DTH to transfer any changes made to the source S3 bucket into the corresponding destination bucket in cn-north-1. The source S3 bucket is in the same account as the DTH, however, it is KMS encrypted. That is, the vanilla S3 Transfer Task for Source Buckets in the current region will not work for this scenario. One possible solution to this is described in Issue https://github.com/awslabs/data-transfer-hub/issues/73

The approach is to create an IAM User in the source bucket account and pass the credentials for it to the DTH S3 Transfer Task. For doing so, one must select "Is bucket in this account?" with "No" in the Source Settings. This will disable the option to listen to S3 events from the source bucket as "Enable S3 Event?" will not be available when selecting that the source bucket is not in the current account.

Expected behavior

We should make sure that either:

evalzy commented 1 year ago

Hi Kaplana

Thanks for the feedback and contribute the PR meanwhile. I have discussed with team and right now for the KMS-encrypted S3 we suggest to use https://github.com/awslabs/data-transfer-hub/blob/main/docs/S3-SSE-KMS-Policy.md approach, and we are going to enhance UI to provide better user experience.

For the suggestion that use AK/SK to access all the S3 bucket including in/ out same account, this might impact too many customers, especially that for those who didn't initiate KMS in source S3, might leading them addition effort to create AK/ SK compare with existing feature.

Let me know if you have other thoughts.

Best Regard/ Eva DTH Product Manager

kaplanan commented 1 year ago

Hi @evalzy , thanks for your reply. I see that but there must be a way in which one could do it via UI instead of manual modification within the Cloudformation stack after creating said S3 transfer task, right? Did I understand this correctly that there are UI features in plan which will incorporate this? Or even select from available KMS keys of that account (just like the ASM secrets) would be nice!

I think if this is what is coming up, that's totally fine!

Best regards Ayhan

evalzy commented 1 year ago

We add this into UI and feature design backlog. Will release within next 2 version.

kaplanan commented 1 year ago

We're looking forward to it. Thank you!

bassemwanis commented 1 month ago

Hi @kaplanan,

After reviewing the documentation and the available workarounds mentioned in https://github.com/aws-solutions/data-transfer-hub/issues/73, I believe it would be more appropriate to classify this as an enhancement rather than a bug. The reason for this change is to ensure that our labels are accurate, which will help us prioritize backlog items more effectively.

Thank you Bassem