Region af-south-1 is NOT supported

[xinlaoda]

Describe the bug when copy S3 objects from af-south-1, error happens.

To Reproduce

1. create data replication hub in ap-southeast-1 region
2. create data replication task, from af-south-1 to ZHY cn-northwest-1

Expected behavior Support af-south-1 and me-south-1

Screenshots ==== CloudWatch Log ====

INFO:JobSender> Start comparing...

Traceback (most recent call last): File "job_sender.py", line 105, in find_and_send_jobs(src_client, des_client, sqs_queue_name, File "job_sender.py", line 32, in find_and_send_jobs job_sender.send_jobs(include_version) File "/app/migration_lib/job.py", line 122, in send_jobs self._get_delta_and_send(include_version) File "/app/migration_lib/job.py", line 83, in _get_delta_and_send src_file_set = self._get_source_set(src_list_gen, include_version) File "/app/migration_lib/job.py", line 38, in _get_source_set src_list = next(src_list_gen) File "/app/migration_lib/client.py", line 218, in _list_objects_without_version response = self._list_objects_func(list_kwargs) File "/app/migration_lib/client.py", line 202, in _list_objects_func return self._client.list_objects_v2(kwargs) File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the ListObjectsV2 operation: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

Please complete the following information about the solution:

• [ ] Version: [e.g. v1.0.0]
• [ ] Region: [af-south-1]
• [x] Was the solution modified from the version published on this repository?
• [ ] If the answer to the previous question was yes, are the changes available on GitHub?
• [ ] Have you checked your service quotas for the sevices this solution uses?
• [x] Were there any errors in the CloudWatch Logs?

Additional context Add any other context about the problem here.

[xinlaoda]

The S3 requests must be sent to the destination region. As in this case AWS Data Replication Hub have configured default region to be the source region, It will need to do an override to destination region, for example:

From EC2 in SG, ap-southeast-1, do following s3 cp CLI:

$ aws s3 cp s3://xxin-package-af-south-1/cm6/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /tmp download failed: s3://xxin-package-af-south-1/cm6/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm to ../../tmp/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm An error occurred (IllegalLocationConstraintException) when calling the GetObject operation: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.

If using "--region " override default one, it works. $ aws s3 cp s3://xxin-package-af-south-1/cm6/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /tmp --region=af-south-1 download: s3://xxin-package-af-south-1/cm6/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm to ../../tmp/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm

[daixba]

Hi @xinlaoda

The cause of your failure is because you deployed the solution in ap-southeast-1 and this tool assumed the region of the source bucket is ap-southeast-1.

We can't support the deployment directly in af-south-1 or me-south-1 for some reason.

However, you can still use this solution to replicate data from those two regions to China regions. The suggestion is that you can deploy this tool in China regions directly and set the source region as those two. If you have some issues deploying data replication hub in China regions like you don't have a ICP licensed domain name etc, you can directly deploy this plugin (follow the readme of this repo) instead.

Hope this clarifies, Thanks for using this solution.

[xinlaoda]

Two issues & comments:

1. If I deploy it in CN region, then I guess it can not automatically trigger data replication worker(Lambda) when a S3 object CREATE in HK or other regions. It just can leverage ECS-docker checking the difference each 1 hour by default. Currently, the customer want to replicate the S3 objects into CN region ASAP, and must be completed 1G-size object in 30 minutes.
2. The S3 objects will be created in 6 WW regions in my case, include eu-, sa-, af-, me-, ap-, but from my test, only af-south-1 and me-south-1 happen error when deploying this hub in HK region, so I think fixing it is reasonable.

[JoeShi]

make sense. we will upgrade to support af-south-1 and me-south-1 regions.