Status of testing Providers that were prepared on August 26, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.6.0rc1

• [x] Added Amazon SageMaker Notebook hook and operators (#33219): @ellisms
• [x] Add deferrable option to LambdaCreateFunctionOperator (#33327): @vincbeck
• [ ] Add Deferrable mode to GlueCatalogPartitionSensor (#33239): @syedahsn
• [x] Add sql_hook_params parameter to S3ToSqlOperator (#33427): @alexbegg
• [x] Add sql_hook_params parameter to SqlToS3Operator (#33425): @alexbegg
• [ ] Add parameter to pass role ARN to GlueJobOperator (#33408): @erdos2n
• [ ] Add new RdsStartExportTaskOperator parameters (#33251): @covidium
• [x] Fix bug in task logs when using AWS CloudWatch. Do not set start_time (#33673): @vincbeck
• [x] Fix AWS Batch waiter failure state (#33656): @yermalov-here
• [x] [bugfix] fix AWS Appflow waiter (#33613): @vandonr-amz
• [x] bugfix: strip tags when falling back to update in SageMakerEndpointOperator (#33487): @vandonr-amz
• [x] Remove non-public interface usage in EcsRunTaskOperator (#29447): @Taragolis
• [x] Upgrade botocore/aiobotocore minimum requirements (#33649): @potiuk
• [x] always push ECS task ARN to xcom in EcsRunTaskOperator (#33703): @vandonr-amz
• [x] Use boto3.client linked to resource meta instead of create new one for waiters (#33552): @Taragolis

  Provider apache.beam: 5.2.2rc1

• [ ] Fix impersonate beam condition lower than 2.39.0 always evaluating true (#33308): @tiranux

  Provider apache.hive: 6.1.5rc1

• [x] Bring back hive support for Python 3.11 (#32607): @potiuk
• [x] Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala

  Provider apache.pinot: 4.1.3rc1

• [x] Referencing correct hooks for Apache Pinot (#33601): @alexbegg

  Provider celery: 3.3.3rc1

• [x] Fix dependencies for celery and opentelemetry for Python 3.8 (#33579): @potiuk
• [x] Make auth managers provide their own airflow CLI commands (#33481): @vandonr-amz
• [x] Refactor Sqlalchemy queries to 2.0 style (Part 7) (#32883): @phanikumv

  Provider cncf.kubernetes: 7.5.0rc1

• [x] Add istio test, use curl /quitquitquit to exit sidecar, and some othe… (#33306): @Owen-CH-Leung
• [x] Add active_deadline_seconds parameter to KubernetesPodOperator (#33379): @amoghrajesh
• [ ] Make cluster_context templated in KubernetesPodOperator (#33604): @mati-o
• [x] Fix KubernetesPodOperator duplicating logs when interrupted (#33500): @fdemiane
• [x] Fix 2.7.0 db migration job errors (#33652): @LipuFei
• [ ] Inspect container state rather than last_state when deciding whether to skip (#33702): @SamWheating
• [x] Introducing class constant to make worker pod log lines configurable (#33378): @amoghrajesh
• [x] Adding typing for KPO SCC objects (#33381): @amoghrajesh
• [ ] Refactor Sqlalchemy queries to 2.0 style (Part 7) (#32883): @phanikumv

  Provider databricks: 4.4.0rc1

• [x] Add Service Principal OAuth for Databricks. (#33005): @johnbrandborg
• [ ] Update docs in databricks.py - we use 2.1 now (#32340): @zbstof

  Provider elasticsearch: 5.0.1rc1

• [ ] Add backward compatibility for elasticsearch<8 (#33281): @sunank200
• [x] Fix urlparse schemaless-behaviour on Python 3.9+ (#33289): @potiuk
• [x] Upgrade Elasticsearch to 8 (#33135): @Owen-CH-Leung

  Provider exasol: 4.2.4rc1

• [x] Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala

  Provider influxdb: 2.2.2rc1

• [x] Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala

  Provider microsoft.azure: 6.3.0rc1

• [x] Add AzureBatchOperator example (#33716): @Lee-W
• [x] feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerInstanceHook (#33467): @Lee-W
• [x] Add DefaultAzureCredential auth for ADX service (#33627): @pankajastro
• [x] Fix updating account url for WasbHook (#33457): @ephraimbuddy
• [x] Fix Azure Batch Hook instantation (#33731): @potiuk
• [x] Bump azure-kusto-data>=4.1.0 (#33598): @pankajastro
• [x] Set logging level to WARNING (#33314): @arjunanan6

  Provider mysql: 5.3.0rc1

• [x] Add init_command parameter to MySqlHook (#33359): @alexbegg
• [ ] Add local_infile parameter to S3ToMySqlOperator (#33459): @benchoncy

  Provider openlineage: 1.0.2rc1

• [x] openlineage: don't run task instance listener in executor (#33366): @mobuchowski
• [x] openlineage: do not try to redact Proxy objects from deprecated config (#33393): @mobuchowski
• [x] openlineage: defensively check for provided datetimes in listener (#33343): @mobuchowski
• [x] openlineage, trino: add OpenLineage support for Trino. (#32910): @JDarDagran

  Provider oracle: 3.7.3rc1

• [x] Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala

  Provider salesforce: 5.4.2rc1

• [x] Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala

  Provider sftp: 4.6.0rc1

• [ ] Add parameter sftp_prefetch to SFTPToGCSOperator (#33274): @functicons

  Provider slack: 8.0.0rc1

• [x] Remove deprecated parts from Slack provider (#33557): @Taragolis
• [x] Replace deprecated slack notification in provider.yaml with new one (#33643): @potiuk
• [x] Avoid importing pandas and numpy in runtime and module level (#33483): @hussein-awala

  Provider snowflake: 5.0.0rc1

• [x] Remove deprecated S3ToSnowflake and SnowflakeToSlack operators (#33558): @Taragolis
• [ ] Set snowflake_conn_id on Snowflake Operators to avoid error (#33681): @SamWheating
• [x] Bump apache-airflow-providers-snowflake due to breaking changes (#33615): @Taragolis

  Provider trino: 5.3.0rc1

• [x] openlineage, trino: add OpenLineage support for Trino. (#32910): @JDarDagran

  Provider yandex: 3.4.0rc1

• [x] Add support for Yandex Dataproc cluster labels (#29811): @zinal

All users involved in the PRs: @amoghrajesh @mobuchowski @zbstof @johnbrandborg @vincbeck @JDarDagran @benchoncy @ellisms @SamWheating @pankajastro @alexbegg @sunank200 @zinal @LipuFei @functicons @hussein-awala @ephraimbuddy @Lee-W @arjunanan6 @erdos2n @vandonr-amz @fdemiane @Owen-CH-Leung @phanikumv @Taragolis @mati-o @syedahsn @tiranux @covidium @yermalov-here @potiuk

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[alexbegg]

Hi, https://github.com/apache/airflow/pull/33427, https://github.com/apache/airflow/pull/33425, https://github.com/apache/airflow/pull/33359 works as expected, even tested using an external database and all looks good.

[alexbegg]

I also tested https://github.com/apache/airflow/pull/33601 and it no longer has the ImportError warning, as expected

[arjunanan6]

I have just tested #33314, works as expected.

[LipuFei]

I haved tested https://github.com/apache/airflow/pull/33652, looks good.

[johnbrandborg]

I have tested #33005 this morning. Messages indicate Service Principal Token is being used, and data was retrieved from a Databricks Workspace.

[potiuk]

Checked all my changes. All look good!

[ephraimbuddy]

Tested https://github.com/apache/airflow/pull/33457 and it works fine

[Owen-CH-Leung]

Tested #33135 and it works as expected. For #33306 , it's not working as expected so I have created another PR which just got merged.

https://github.com/apache/airflow/pull/33779

[pankajastro]

https://github.com/apache/airflow/pull/33598 and https://github.com/apache/airflow/pull/33627 looks good. thanks!

[hussein-awala]

Checked #33483 and it is present in the different providers.

[Lee-W]

Tested https://github.com/apache/airflow/pull/33716 and https://github.com/apache/airflow/pull/33467. Both works fine. Thanks!

[amoghrajesh]

Tested my changes:

• https://github.com/apache/airflow/pull/33378 (This is a non functional one)
• https://github.com/apache/airflow/pull/33379
• https://github.com/apache/airflow/pull/33381 (This is a non functional one)

Things do not seem to have regressed and it works as expected. Ran a few dags to target the cncf provider too. It looks good to me.

Thanks!

[mobuchowski]

Tested my changes, everything looks good.

[zinal]

Hi! #29811 looks fine for me, processed the test dag successfully.

[phanikumv]

32883 looks good

[JDarDagran]

32910 works as expected.

[yermalov-here]

tested https://github.com/apache/airflow/pull/33656 and it looks good

[ellisms]

tested #33219 and it looks good.

[fdemiane]

tested #33500, works as expected

[vandonr-amz]

good for me

[vincbeck]

33327 and #33673 work as expected

[eladkal]

Thank you everyone. microsoft.azure and cncf.kubernetes are excluded from rc1. I will cut rc2 shortly. The rest of the providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.