Use base aws classes in amazon provider Operators/Sensors/Triggers

[Taragolis]

Body

Right now some inconsistency exists between different Operators/Sensors/Triggers which are use AwsBaseGeneric hook. After #34784 and #34884 we could make all of them (except Transfer Operators) use same subset of generic attributes.

It might make more easier to end users use this objects (don't need to think is it region or region_name or is it even support this arguments), as well as make easier extend by new new hook parameters to all this operators in the one go.

General Approach for solve each case:

• Use AwsBaseOperator for operators, AwsBaseSensor for sensors. In some very specific case e.g. multi inheritance required to use AwsBaseHookMixin (see: AppflowRecordsShortCircuitOperator)
• Provide additional arguments to AwsBaseWaiterTrigger (if required)
• Wrap template_fields parameters
• Attach generic documentation into the service specific page into the Amazon Provider Documentation
• Validate that all parameters propagated to operator/sensor specific hook

All this steps already done for some of the operators, so it could be use as a reference

• [x] Use base aws classes in Amazon AppFlow Operators (https://github.com/apache/airflow/pull/35082)
  • operator(s)
  • [x] Use base aws classes in Amazon Athena Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/35133)
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [ ] Use base aws classes in AWS Batch Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/35226)
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in AWS CloudFormation Operators/Sensors https://github.com/apache/airflow/pull/36771
  • operator(s)
  • sensor(s)
  • [x] Use base aws classes in AWS Datasync Operators https://github.com/apache/airflow/pull/36766
  • operator(s)
  • [x] Use base aws classes in AWS DMS Operators/Sensors https://github.com/apache/airflow/pull/36772
  • operator(s)
  • sensor(s)
  • [x] Use base aws classes in Amazon DynamoDB Sensors https://github.com/apache/airflow/pull/36770
  • sensor(s)
  • [ ] Use base aws classes in Amazon EC2 Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in Amazon ECS Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/36393)
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [ ] Use base aws classes in Amazon EKS Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [ ] Use base aws classes in Amazon EMR Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in Amazon EventBridge Operators https://github.com/apache/airflow/pull/36765
  • operator(s)
  • [ ] Use base aws classes in AWS Glue Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in AWS Glue Data Catalog Sensors https://github.com/apache/airflow/pull/40492
  • sensor(s)
  • [x] Use base aws classes in AWS Glue Crawlers Operators/Sensors/Triggers https://github.com/apache/airflow/pull/40504
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [ ] Use base aws classes in AWS Glue DataBrew Operators/Triggers
  • operator(s)
  • trigger(s)
  • [x] Use base aws classes in AWS Lambda Operators/Sensors/Triggers (https://github.com/apache/airflow/pull/34890)
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in Amazon QuickSight Operators/Sensors https://github.com/apache/airflow/pull/36776
  • operator(s)
  • sensor(s)
  • [ ] Use base aws classes in Amazon RDS Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [ ] Use base aws classes in Amazon Redshift clusters Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in AWS Redshift Data API Operators https://github.com/apache/airflow/pull/36764
  • operator(s)
  • [ ] Use base aws classes in Amazon S3 Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in Amazon S3 Glacier Operators/Sensors (https://github.com/apache/airflow/pull/35108)
  • operator(s)
  • sensor(s)
  • [ ] Use base aws classes in Amazon SageMaker Operators/Sensors/Triggers
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in Amazon SNS Operators https://github.com/apache/airflow/pull/36615
  • operator(s)
  • [x] Use base aws classes in Amazon SQS Operators/Sensors/Triggers https://github.com/apache/airflow/pull/36613
  • operator(s)
  • sensor(s)
  • trigger(s)
  • [x] Use base aws classes in AWS Step Functions Operators/Sensors/Triggers https://github.com/apache/airflow/pull/36468
  • operator(s)
  • sensor(s)
  • trigger(s)

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ferruzzi]

Thanks for getting this set up!

[gopidesupavan]

@Taragolis i am happy to look into this Use base aws classes in AWS Glue Crawlers Operators/Sensors/Triggers ? or anyone is working already?

[prakharcode]

@Taragolis I am happy to take EMR, EC2 and S3 (operators) here

[gopidesupavan]

@vincbeck @eladkal Working on this Use base aws classes in AWS Glue Data Catalog Sensors didnt find anything in the documentation and example code for this. Planning to add them here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/glue.html is this fine?

[vincbeck]

@vincbeck @eladkal Working on this Use base aws classes in AWS Glue Data Catalog Sensors didnt find anything in the documentation and example code for this. Planning to add them here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/glue.html is this fine?

Please yes :)

[eladkal]

@gopidesupavan do you plan to address the other Glue services that were left?

[gopidesupavan]

@eladkal Yes will update other services, coming week.