apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.36k stars 14.35k forks source link

Adding DMS serverless options to aws provider #39954

Open prakharcode opened 6 months ago

prakharcode commented 6 months ago

Description

Sometime back AWS launched serverless DMS (Data Migration Service). This lets users create replication jobs without explicitly setting up a replication instance.

Currently, in the AWS provider, we have support for classic DMS, which requires users to set up a replication instance. It would be nice, if we can provide dms serverless support using the following boto3 functions:

Use case/motivation

Giving airflow[aws] a serverless DMS option would unlock specific DMS use cases that are becoming increasingly common via serverless, like -- full-load of a new table and patching a CDC table using the full load on serverless.

The serverless implementation is also quite appealing for source database replication that happens at a larger frequency than a couple of days (where a dedicated instance doesn't make sense) or with source data that changes or grows a lot.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 6 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Taragolis commented 6 months ago

Feel free to raise a PR.

ellisms commented 2 months ago

If there hasn't been any work on this, I can pick it up.

eladkal commented 1 month ago

If there hasn't been any work on this, I can pick it up.

Assigned