Adds the basic ability to export tables from RDS to S3. Follows a similar pattern to DynamoDB/DocumentDB source that the leader node creates ExportPartition in coordination store and the node that picks up the ExportPartition will call RDS APIs to take snapshot and export it to S3.
There will be a follow-up PR to process the S3 files in the rds source. This PR only targets RDS instances, will add support for RDS/Aurora clusters in separate PRs (the RDS APIs for instances and clusters are different).
Testing
Created RDS MySQL instance as source and created a DDB table as coordination store and tested with the following pipeline config and verified that the data is exported to S3.
[ ] New functionality has a documentation issue. Please link to it in this PR.
[ ] New functionality has javadoc added
[x] Commits are signed with a real name per the DCO
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Description
Adds the basic ability to export tables from RDS to S3. Follows a similar pattern to DynamoDB/DocumentDB source that the leader node creates ExportPartition in coordination store and the node that picks up the ExportPartition will call RDS APIs to take snapshot and export it to S3.
There will be a follow-up PR to process the S3 files in the rds source. This PR only targets RDS instances, will add support for RDS/Aurora clusters in separate PRs (the RDS APIs for instances and clusters are different).
Testing
Created RDS MySQL instance as source and created a DDB table as coordination store and tested with the following pipeline config and verified that the data is exported to S3.
Issues Resolved
Contributes to #4561
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.