Status of testing Providers that were prepared on August 03, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.27.0rc2

• [x] Add RedriveExecution support to StepFunctionStartExecutionOperator (#40976): @gopidesupavan
• [ ] AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski
• [x] Introduce Amazon Kinesis Analytics V2 (Managed Service for Apache Flink application) (#40765): @gopidesupavan
• [ ] Make EMR Container Trigger max attempts retries match the Operator (#41008): @o-nikolas Linked issues:
  • [ ] Linked Issue #40483: @akomisarek
• [ ] Fix RdsStopDbOperator operator in deferrable mode (#41059): @vincbeck
• [ ] Fix RedshiftCreateClusterOperator to always specify PubliclyAccessible (#40872): @vincbeck
• [ ] Fix AWS Redshift operators and sensors (#41191): @vincbeck
• [ ] Fix EmrServerlessStartJobOperator (#41103): @vincbeck
• [ ] Update example_redshift and example_redshift_s3_transfers to use RedshiftDataHook instead of RedshiftSQLHook (#40970): @vincbeck
• [x] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [x] [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda Linked issues:
  • [x] Linked Issue #40335: @JDarDagran
  • [x] Linked Issue #38767: @mobuchowski
• [x] Move AWS Managed Service for Apache Flink sensor states to Hook (#40896): @gopidesupavan
• [ ] Replace usages of task context logger with the log table (#40867): @dstandish
• [ ] Deprecate SageMakerTrainingPrintLogTrigger (#41158): @vincbeck

  Provider apache.beam: 5.7.2rc2

• [ ] Update default value for job_name in DataflowConfiguration (#40645): @e-galan Linked issues:
  • [ ] Linked Issue #40515: @turb

    Provider apache.drill: 2.7.3rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider apache.druid: 3.10.2rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider apache.impala: 1.4.2rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider apache.pinot: 4.4.2rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider apprise: 1.3.2rc1

• [ ] Fix default behaviour for init function in AppriseNotifier (#41054): @romsharon98

  Provider celery: 3.7.3rc1

• [ ] Increase broker's visibility timeout to 24hrs (#40879): @ephraimbuddy

  Provider cncf.kubernetes: 8.3.4rc1

• [ ] Pass content of kube/config file to triggerer as a dictionary (#41178): @VladaZakharova
• [ ] Fix confusing log message in kubernetes executor (#41035): @RNHTTR
• [ ] Fix ApiException handling when adopting completed pods (#41109): @jedcunningham

  Provider common.compat: 1.1.0rc1

• [ ] AIP-62: add method to common.compat to not force hooks to try/except every time (#40812): @mobuchowski
• [x] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [ ] AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski

  Provider common.io: 1.4.0rc1

• [x] [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda Linked issues:
  • [x] Linked Issue #40335: @JDarDagran
  • [x] Linked Issue #38767: @mobuchowski
• [ ] AIP-62: add file dataset type support into common.io provider (#40817): @mobuchowski
• [x] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [ ] AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski

  Provider common.sql: 1.15.0rc1

• [x] Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669): @dabla

  Provider databricks: 6.8.0rc1

• [x] Add DatabricksWorkflowPlugin (#40724): @pankajkoti Linked issues:
  • [x] Linked Issue #40153: @pankajkoti
  • [x] Linked Issue #40714: @jscheffl
  • [x] Linked Issue #40708: @jscheffl
• [x] DatabricksPlugin - Fix dag view redirect URL by using url_for redirect (#41040): @pankajkoti
• [x] Fix named parameters templating in Databricks operators (#40864): @boraberke Linked issues:
  • [x] Linked Issue #40788: @vatsrahul1001
  • [x] Linked Issue #40471: @boraberke
• [x] [Databricks Provider] Revert PRs #40864 and #40471 (#41050): @pankajkoti Linked issues:
  • [x] Linked Issue #40864: @boraberke
  • [x] Linked Issue #35433: @josh-fell
  • [x] Linked Issue #40471: @boraberke

    Provider docker: 3.12.3rc1

• [x] DockerSwarmOperator: Support line breaks in service logs (#40705): @mr1holmes Linked issues:
  • [x] Linked Issue #40571: @bladerail

    Provider elasticsearch: 5.4.2rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider exasol: 4.5.3rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider ftp: 3.10.1rc1

• [ ] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran

  Provider google: 10.21.1rc2

• [x] Dynamic import MySqlHook/PostgresHook in CloudSQLDatabaseHook (#41009): @jhongy1994 Linked issues:
  • [x] Linked Issue #40175: @bourliam
• [x] Make google_analytics_admin system tests self sufficient (#40951): @moiseenkov
• [ ] Fix Custom Training Job operators to accept results without a model in def mode (#40685): @e-galan Linked issues:
  • [ ] Linked Issue #40476: @vignesh-sc
• [x] Fix behavior for reattach_state parameter in BigQueryInsertJobOperator (#40664): @VladaZakharova
• [x] Fix cloudsql-query system tests (#41092): @moiseenkov
• [x] Refactor dataproc system tests (#40720): @VladaZakharova
• [ ] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [ ] Resolve google cloud sql deprecation in tests (#40834): @dirrao
• [ ] Update default value for job_name in DataflowConfiguration (#40645): @e-galan Linked issues:
  • [ ] Linked Issue #40515: @turb

    Provider jdbc: 4.4.0rc1

• [ ] Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669): @dabla
• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider microsoft.azure: 10.3.0rc1

• [ ] Added priority to Azure Container Instances (#40616): @liamalsopp Linked issues:
  • [ ] Linked Issue #40574: @liamalsopp
• [x] Bump minimum version for azure containerinstance. (#40767): @potiuk Linked issues:
  • [x] Linked Issue #40616: @liamalsopp

    Provider microsoft.mssql: 3.8.0rc1

• [x] Added support for replace option when using insert_rows with MsSqlHook (#40836): @dabla
• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider mysql: 5.6.3rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider odbc: 4.6.3rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider openlineage: 1.10.0rc1

• [x] openlineage: add AirflowRun on COMPLETE/FAIL events (#40996): @mobuchowski
• [x] openlineage: extend custom_run_facets to on_failed and on_complete (#40953): @kacpermuda Linked issues:
  • [ ] Linked Issue #38982: @anandhimurali
• [x] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [x] openlineage: Add AirflowDagRunFacet to dag runEvents (#40854): @dolfinus Linked issues:
  • [x] Linked Issue #40798: @dolfinus
• [x] [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda Linked issues:
  • [x] Linked Issue #40335: @JDarDagran
  • [x] Linked Issue #38767: @mobuchowski
• [x] Ability to add custom facet in OpenLineage events (#38982): @anandhimurali
• [x] AIP-62: add method to common.compat to not force hooks to try/except every time (#40812): @mobuchowski
• [x] openlineage: use airflow provided getters in conf (#40790): @kacpermuda Linked issues:
  • [x] Linked Issue #40780: @mobuchowski
  • [x] Linked Issue #40589: @mobuchowski
• [x] openlineage: add config to include 'full' task info based on conf setting (#40589): @mobuchowski
• [x] Add log_url to OpenLineage AirflowRunFacet (#40797): @dolfinus
• [x] openlineage: add deferrable information to task info in airflowRunFacet (#40682): @kacpermuda Linked issues:
  • [x] Linked Issue #38264: @kacpermuda
• [x] openlineage: adjust DefaultExtractor's on_failure detection for airflow 2.10 fix (#41094): @kacpermuda Linked issues:
  • [x] Linked Issue #41053: @kacpermuda
• [x] openlineage: make value of slots in attrs.define consistent (#40992): @kacpermuda Linked issues:
  • [x] Linked Issue #40972: @JDarDagran
• [x] openlineage: Set slots to True for facets used in dag level (#40972): @JDarDagran
• [x] openlineage: fix / add some task attributes in AirflowRunFacet (#40725): @kacpermuda Linked issues:
  • [x] Linked Issue #40371: @mobuchowski
• [x] openlineage: replace dt.now with airflow.utils.timezone.utcnow (#40887): @kacpermuda
• [x] openlineage: remove deprecated parentRun facet key (#40681): @kacpermuda

  Provider pgvector: 1.2.2rc1

• [ ] updating pgvector to !=0.3.0 to avoid recent issue in latest releases(0.3.0) (#40683): @vatsrahul1001

  Provider postgres: 5.11.3rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider qdrant: 1.1.2rc1

• [x] Qdrant Provider: Bump Qdrant client to latest (#40831): @Anush008

  Provider sftp: 4.10.3rc1

• [x] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran

  Provider slack: 8.8.0rc1

• [ ] Add the ability to toggle the unfurl behavior for slack notifier (#40694): @armand-sauzay
• [ ] Add newly added unfurl args to the docstring of SlackNotifier (#40709): @armand-sauzay

  Provider snowflake: 5.6.1rc1

• [x] openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider sqlite: 3.8.2rc1

• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider ssh: 3.12.0rc1

• [x] Add on kill to ssh (#40377): @MRLab12 Linked issues:
  • [x] Linked Issue #40343: @potiuk

    Provider teradata: 2.5.0rc1

• [x] Implemented Query Band Support for the Teradata provider (#40716): @sc250072
• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

  Provider ydb: 1.2.0rc1

• [x] ydb provider: add bulk upsert support (#40631): @uzhastik
• [x] Clean up remaining getattr connection DbApiHook (#40665): @dabla

All users involved in the PRs: @MRLab12 @uzhastik @dstandish @kacpermuda @jedcunningham @mr1holmes @gopidesupavan @moiseenkov @Anush008 @dirrao @o-nikolas @vatsrahul1001 @dabla @romsharon98 @e-galan @JDarDagran @ephraimbuddy

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jhongy1994]

I checked #41009 (Linked issue #40175) and working as expected :)

[moiseenkov]

40951, #41092 work as expected

[Anush008]

Qdrant Provider: Bump Qdrant client to latest (#40831): @Anush008

Tested and verified working.

[VladaZakharova]

https://github.com/apache/airflow/pull/40664 and https://github.com/apache/airflow/pull/40720 work as expected

[potiuk]

Checked my changes - both look good !

[pankajkoti]

Tested my changes in the Databricks provider, all look good!

[dolfinus]

40854 and #40797 are fine

[sc250072]

https://github.com/apache/airflow/pull/40716 Tested and working fine. All system tests are getting success.

[dabla]

Tested following PR with odbc/jdbc/mssql and looking good:

• Clean up remaining getattr connection DbApiHook (#40665)
• Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)
• Added support for replace option when using insert_rows with MsSqlHook (#40836)

[kacpermuda]

I've tested all OpenLineage provider changes and my PRs with some example DAGs, all seems to work fine.

[gopidesupavan]

Have tested #40976 , #40765 and #40896. All looks good. thank you.

[JDarDagran]

Tested all my changes - all look fine.

[uzhastik]

ydb provider works as expected, checked https://github.com/apache/airflow/pull/40631 thank you!

[mr1holmes]

Tested https://github.com/apache/airflow/pull/40705 Works as expected.

[eladkal]

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.