Open shikunwei opened 2 years ago
We do IAM key rotation monthly for security reasons.
The brief steps of the rotation are:
This amazon-s3-data-replication-hub-plugin works fine during steps one and two when the first AK/SK is active and can be used in these 37 days.
But on the 38th day, it stops working and throw this error: "2022/08/01 03:16:17 S3> Got an error uploading file - operation error S3: PutObject, https response error StatusCode: 403, RequestID: xxxxxxxx, HostID: xxxxxxxx, api error InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records."
It seems that the worker node is caching the AK/SK in the secret manager from day 1 and not updating the cache after that. Although we updated the secret manager with the second AK/SK on day 30, the cached AK/SK won't be updated. From day 38, after the first key is inactive, the plugin will start getting 403 errors and still keeps trying with the outdated AK/SK in its cache.
If the worker node has some cache update mechanism, the issue should be resolved.
Could you please take a look at this issue?
Hi @shikunwei , sorry for the late reply.
Thanks for reporting this issue to us, Data Transfer Hub doesn't support auto-rotated access key and we are trying to find out a way to support this scenario.
Here we provide a workaround:
You can using Event Bridge rule and a Lambda function to terminate all the active worker instance when your source access key is rotated.
To Reproduce
2022/08/01 03:16:17 ----->Transferred 1 object xxxxxxxx/xxxxxxxx.json with status ERROR