search

astronomer / astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
https://astronomer-providers.rtfd.io/
Apache License 2.0
137 stars 26 forks source link

Deprecate amazon async provider #1410

Closed Lee-W closed 8 months ago

Lee-W commented 10 months ago

Is your feature request related to a problem? Please describe.

as most of the functionality has been contributed back to OSS airflow, we're going to leverage the code there and deprecate this project

Describe the solution you'd like

  1. Contribute the delta to the OSS provider
  2. Point the astronomer provider to the OSS provider
  3. Do the release and wait for a month or so before we remove the astronomer provider in a major release

Describe alternatives you've considered

Additional context

Current Progress

  1. [x] EmrContainerSensorAsync
  2. [x] EmrStepSensorAsync
  3. [x] EmrJobFlowSensorAsync
  4. [x] RedshiftClusterSensorAsync
  5. [x] BatchSensorAsync
  6. [x] S3KeySensorAsync
  7. [x] S3KeySizeSensorAsync
    • Note: already deprecated
    • Action: -
    • Deprecation PR: -
    • OSS Airflow PR: -
    • Minimum Provider Version: -
  8. [x] S3KeysUnchangedSensorAsync
  9. [x] S3PrefixSensorAsync
    • Note: already deprecated
    • Action: -
    • Deprecation PR: -
    • OSS Airflow PR: -
    • Minimum Provider Version: -
  10. [x] EmrContainerOperatorAsync
  11. [x] RedshiftDeleteClusterOperatorAsync
  12. [x] RedshiftResumeClusterOperatorAsync
  13. [x] RedshiftPauseClusterOperatorAsync
  14. [x] BatchOperatorAsync
  15. [x] RedshiftSQLOperatorAsync
    • Note: deprecated in OSS airflow
    • Action: keep it as it is
    • Deprecation PR: -
    • OSS Airflow PR: -
    • Minimum Provider Version: -
  16. [x] SageMakerProcessingOperatorAsync
  17. [x] SageMakerTransformOperatorAsync
  18. [x] SageMakerTrainingOperatorAsync (WIP)
  19. [x] RedshiftDataOperatorAsync
phanikumv commented 10 months ago

Refer to https://github.com/apache/airflow/pull/36523

Lee-W commented 10 months ago

got some push back in https://github.com/apache/airflow/pull/36416 and https://github.com/apache/airflow/pull/36523. already started the discussion with the community. started working on operator we've not yet contributed back instead of missing the check

Lee-W commented 10 months ago

Add draft https://github.com/apache/airflow/pull/36578 and https://github.com/apache/airflow/pull/36550

Lee-W commented 10 months ago

Add draft https://github.com/apache/airflow/pull/36586

Different from its original RedshiftDataOperatorAsync implementation, I follow how it's implemented on OSS airflow. The main difference is checking statement_id vs query_ids.

Lee-W commented 10 months ago

Wrap up https://github.com/apache/airflow/pull/36578 and add https://github.com/apache/airflow/pull/36658

Lee-W commented 10 months ago

As of Jan 10th

Lee-W commented 10 months ago

As of Jan 11th

Lee-W commented 10 months ago

As of Jan 15th

Lee-W commented 10 months ago

As of Jan 16th

Lee-W commented 10 months ago

As of Jan 17th

Lee-W commented 9 months ago

As of Jan 19th

Lee-W commented 9 months ago

As of Jan 24th

Lee-W commented 9 months ago

After working on RedshiftSQLOperatorAsync a bit, I think we should not deprecate it and keep it as it is just like the snowflake operator instead.

Lee-W commented 9 months ago

As of Jan 29th

phanikumv commented 9 months ago

Waiting for OSS provider release