awslabs / amazon-s3-data-replication-hub-plugin

The Amazon S3 Transfer Plugin for Data Transfer Hub(https://github.com/awslabs/data-transfer-hub). Transfer objects from S3(in other partition), Alibaba Cloud OSS, Tencent COS, Qiniu Kodo into Amazon S3.
Apache License 2.0
47 stars 16 forks source link

Deletions can't be replicated after enabling bucket versioning for the source S3 bucket #86

Open shikunwei opened 2 years ago

shikunwei commented 2 years ago

Describe the bug The S3 replication plugin stops working for file/folder deletions after enabling bucket versioning for the source S3 bucket.

To Reproduce

  1. Create two S3 buckets without bucket versioning.
  2. Install this S3 replication plugin for the two buckets above.
  3. Do some file upload/delete operations to the source S3 bucket, and verify that the replication to the destination bucket works correctly.
  4. Enable the bucket versioning for the source bucket. Delete some files or folders in the source bucket, and you will see that those files/folders won't be deleted in the destination bucket no matter how long you wait.

So Enable the bucket versioning for the source bucket caused the S3 replication plugin won't work for file/folder deletions.

Expected behavior Expect the files and folders to be deleted in the destination bucket after they have been deleted from the source bucket. Otherwise, the difference between the two buckets would keep growing.

Please complete the following information about the solution:

Screenshots None

Additional context None

YikaiHu commented 2 years ago

Hi @shikunwei , thanks for call out for this issue.

DTH S3 plugin currently doesn't support sync object's version and sync deletion when object enable version.