Status of testing Providers that were prepared on July 06, 2023

[eladkal]

Body

Issue title: Status of testing Providers that were prepared on July 06, 2023

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.5.0rc2

• [x] Add Alibaba Cloud AnalyticDB Spark Support (#31787): @dcoliversun

  Provider amazon: 8.3.0rc2

• [ ] Add ChimeWebhookHook (#31939): @cjames23
• [ ] Add ChimeNotifier (#32222): @cjames23
• [ ] Add deferrable mode to S3KeysUnchangedSensor (#31940): @sunank200
• [ ] Add deferrable mode to RdsCreateDbInstanceOperator and RdsDeleteDbInstanceOperator (#32171): @syedahsn
• [x] Add deferrable mode for AthenaOperator (#32186): @vandonr-amz
• [x] Add a deferrable mode to BatchCreateComputeEnvironmentOperator (#32036): @vandonr-amz
• [ ] Add deferrable mode in EMR operator and sensor (#32029): @pankajastro
• [x] add async wait method to the "with logging" aws utils (#32055): @vandonr-amz
• [ ] Add custom waiters to EMR Serverless (#30463): @syedahsn
• [x] Add an option to GlueJobOperator to stop the job run when the TI is killed (#32155): @hussein-awala
• [x] deferrable mode for SageMakerTuningOperator and SageMakerEndpointOperator (#32112): @vandonr-amz
• [ ] EKS Create/Delete Nodegroup Deferrable mode (#32165): @syedahsn
• [x] Deferrable mode for ECS operators (#31881): @vandonr-amz
• [ ] Add job_poll_interval argument to AWS GlueJobOperator (#32147): @raphaelauv
• [ ] Added AzureBlobStorageToS3Operator transfer operator (#32270): @Adaverse
• [x] bugfix: break down run+wait method in ECS operator (#32104): @vandonr-amz
• [x] Handle UnboundLocalError while parsing invalid s3_url (#32120): @Adaverse
• [ ] Fix LambdaInvokeFunctionOperator payload parameter type (#32259): @eladi99
• [x] Bug fix GCSToS3Operator: avoid ValueError when replace=False with files already in S3 (#32322): @Adaverse
• [ ] Deprecate delimiter param and source object's wildcards in GCS, introduce match_glob param. (#31261): @shahar1
• [x] aws waiter util: log status info with error level on waiter error (#32247): @vandonr-amz
• [x] rewrite method used in ecs to fetch less logs (#31786): @vandonr-amz
• [ ] Refactor Eks Create Cluster Operator code (#31960): @syedahsn
• [x] Use a waiter in AthenaHook (#31942): @vandonr-amz
• [x] Add on_finish_action to KubernetesPodOperator (#30718): @hussein-awala
• [x] Add default_deferrable config (#31712): @Lee-W
• [x] deprecate arbitrary parameter passing to RDS hook (#32352): @vandonr-amz

  Provider apache.hive: 6.1.2rc2

• [x] Add proxy_user template check (#32334): @potiuk

  Provider apache.impala: 1.1.2rc2

• [x] Add kerberos dependency to Impala Provider (#32304): @Joffreybvn

  Provider apache.kafka: 1.1.2rc2

• [x] Break AwaitMessageTrigger execution when finding a message with the desired format (#31803): @hussein-awala

  Provider apache.livy: 3.5.2rc2

• [ ] Fix LivyHook TypeError exception on session_id log format (#32051): @joaopamaral
• [x] Add default_deferrable config (#31712): @Lee-W

  Provider cncf.kubernetes: 7.2.0rc2

• [x] Add on_finish_action to KubernetesPodOperator (#30718): @hussein-awala
• [ ] Fix KubernetesPodOperator validate xcom json and add retries (#32113): @aagateuip
• [x] Fix KubernetesPodTrigger waiting strategy (#31348): @hussein-awala
• [ ] fix spark-kubernetes-operator compatibility (#31798): @blcksrx
• [x] Add default_deferrable config (#31712): @Lee-W

  Provider common.sql: 1.6.0rc2

• [x] openlineage, common.sql: provide OL SQL parser as internal OpenLineage provider API (#31398): @JDarDagran
• [x] openlineage: pass SQLAlchemy engine to construct information schema query. (#32371): @JDarDagran

  Provider databricks: 4.3.1rc2

• [x] Modify template_fields of DatabricksSqlOperator to support parent class fields (#32253): @eladkal
• [x] Add default_deferrable config (#31712): @Lee-W

  Provider dbt.cloud: 3.2.2rc2

• [x] Add default_deferrable config (#31712): @Lee-W

  Provider elasticsearch: 5.0.0rc2

• [ ] Deprecate the 2 non-official elasticsearch libraries (#31920): @Owen-CH-Leung

  Provider google: 10.3.0rc2

• [x] Add on_finish_action to KubernetesPodOperator (#30718): @hussein-awala
• [ ] Add deferrable mode to CloudSQLExportInstanceOperator (#30852): @bkossakowska
• [ ] Adding src_fmt_configs to the list of template fields. (#32097): @nathadfield
• [ ] Fix name format in the batch requests (#32070): @HaeSe0ng
• [x] Fix BigQueryInsertJobOperator error handling in deferrable mode (#32034): @phanikumv
• [ ] Fix BIGQUERY_JOB_DETAILS_LINK_FMT in BigQueryConsoleLink (#31953): @bkossakowska
• [x] Make the deferrable version of DataprocCreateBatchOperator handle a b… (#32216): @kristopherkane
• [ ] Switch Google Ads API version from v13 to v14 (#32028): @ahidalgob
• [ ] Deprecate delimiter param and source object's wildcards in GCS, introduce match_glob param. (#31261): @shahar1
• [ ] Refactor GKECreateClusterOperator's body validation (#31923): @moiseenkov
• [x] Optimize deferrable mode execution for BigQueryValueCheckOperator (#31872): @phanikumv
• [ ] Add default_deferrable config (#31712): @Lee-W

  Provider hashicorp: 3.4.2rc2

• [ ] Add default retry to hvac client requests (#31073): @notatallshaw-gts

  Provider microsoft.azure: 6.2.0rc2

• [x] Adds connection test for ADLS Gen2 (#32126): @Adaverse
• [x] Add option to pass extra configs to ClientSecretCredential (#31783): @Adaverse
• [x] Added AzureBlobStorageToS3Operator transfer operator (#32270): @Adaverse
• [x] feat(providers/azure): cancel pipeline if unexpected exception caught (#32238): @Lee-W
• [x] Fix where account url is build if not provided using login (account name) (#32082): @Adaverse
• [x] refresh connection if an exception is caught in "AzureDataFactory" (#32323): @Lee-W
• [x] Add default_deferrable config (#31712): @Lee-W

  Provider postgres: 5.5.2rc2

• [x] postgres: deprecate runtime_parameters in favor of options in hook_params (#32345): @JDarDagran

  Provider snowflake: 4.3.0rc2

• [x] Add Deferrable switch to SnowflakeSqlApiOperator (#31596): @utkarsharma2

  Provider trino: 5.2.0rc2

• [x] Trino Hook: Add ability to read JWT from file (#31950): @jhbigler

  Provider vertica: 3.5.0rc2

• [x] Add various Vertica connection parameters (#32089): @darkag

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs: @darkag @raphaelauv @blcksrx @vandonr-amz @jhbigler @potiuk @Lee-W @bkossakowska @nathadfield @dcoliversun @phanikumv @pankajastro @aagateuip @joaopamaral @Adaverse @JDarDagran @moiseenkov @Joffreybvn @eladi99 @hussein-awala @notatallshaw-gts @Owen-CH-Leung @shahar1 @eladkal @utkarsharma2 @sunank200 @syedahsn @kristopherkane @cjames23 @ahidalgob @HaeSe0ng

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Joffreybvn]

32304 Works fine. Kerberos is installed when doing pip install apache-airflow-providers-apache-impala[kerberos] ✅

[JDarDagran]

32345 works fine - tested both with hook_params and deprecated runtime_parameters.

31398 , #32371 are added too. There's however a bug fixed in #32400 which is not exposed though as there's no release of OpenLineage provider yet (method with a bug is never called without the OL provider). Would be nice to include this in this release cycle, I guess?

[eladkal]

There's however a bug fixed in https://github.com/apache/airflow/pull/32400 which is not exposed though as there's no release of OpenLineage provider yet (method with a bug is never called without the OL provider). Would be nice to include this in this release cycle, I guess?

Thanks for verifying This is not a blocker for this release since the OL integration doesn't work anyway (as OL provider is not yet available) the bug is locallized to OL part. For the moment I see no reason to have RC3, The fix will be included in future release of the provider.

[Lee-W]

We tested our example, DAGs with the following rc without encountering any error

• apache-airflow-providers-amazon
• apache-airflow-providers-apache-hive
• apache-airflow-providers-apache-livy
• apache-airflow-providers-cncf-kubernetes
• apache-airflow-providers-databricks
• apache-airflow-providers-dbt-cloud
• apache-airflow-providers-google
• apache-airflow-providers-microsoft-azure
• apache-airflow-providers-snowflake

[pankajkoti]

For #31920 I have a question which I have asked here https://github.com/apache/airflow/pull/31920#issuecomment-1623629476

[darkag]

tested #32089, works as expected

[vandonr-amz]

good for me, even if I'd prefer an RC3 to sneak some not-yet-breaking changes in there.

[Adaverse]

Tested #32126 #31783 #32082 #32120 #32322 works as expected.

[potiuk]

Checked #32334 :+1:

vs (6.2.1)

[dcoliversun]

Checked #31787, thanks for your work @eladkal

• ADB Spark is installed when apache-airflow-providers-alibaba==2.5.0rc2
• DAGs works fine

[eladkal]

Tested #32126 #31783 #32082 #32120 #32322 works as expected.

Can you also check 32270 ?

[phanikumv]

32034 and #31872 are working fine

[jhbigler]

31950 is working, I used apache-airflow-providers-trino==5.2.0rc2 and tested it against our development trino instance, I can verify the Trino hook reads the jwt file from disk and successfully authenticates with trino.

[Adaverse]

Tested #32126 #31783 #32082 #32120 #32322 works as expected.

Can you also check 32270 ?

Tested with the above versions of providers of Amazon and Microsoft #32270 works as expected. Thanks, @eladkal

[kristopherkane]

Tested #32216 in local Airflow against real GCP Dataproc jobs.

[eladkal]

Amazon and Elasticsearch are excluded from RC2 (RC3 will be cut seperatly) The rest of providers are not affected

[utkarsharma2]

Tested #31596 https://github.com/apache/airflow/pull/31596, working as expected.

On Sat, Jul 8, 2023 at 12:21 AM eladkal @.***> wrote:

Amazon and Elasticsearch are excluded from RC2 (RC3 will be cut seperatly) The rest of providers are not affected

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/32389#issuecomment-1625886602, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDLAHKTQ3NR6BIU7ECEWUDXPBLCXANCNFSM6AAAAAAZ74VIAU . You are receiving this because you were mentioned.Message ID: @.***>

[hussein-awala]

I have tested my changes and all of them work as expected. For #31803, it addresses a part of the problem, but there are still remaining issues in some use cases that need to be resolved, however no need for a new RC, we can release them in next releases, and the merged PR is necessary.

[eladkal]

Thank you everyone. Providers are released except Amazon and Elasticsearch which we will have RC3.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[raphaelauv]

Hey , I did not test on time the K8S providers -> there is a bug on the release

https://github.com/apache/airflow/issues/32458

[eladkal]

Hey , I did not test on time the K8S providers -> there is a bug on the release

32458

Provider already released. When fix is merged it will be released in the next provider wave

[potiuk]

Provider already released. When fix is merged it will be released in the next provider wave

We could consider yanking it.

[eladkal]

We could consider yanking it.

We shouldn't I don't think its a major bug and if we yank kubernetes we must also yank google and cancel amazon rc4 (currently in vote) as they use new feature released in kubernetes provider.

I will cut new release for kubernetes as soon as possible (can't right now because we have another provider vote in progress)

[potiuk]

Sure. No problem with that :)