Status of testing Providers that were prepared on November 24, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.12.0rc1

• [x] Add EC2HibernateInstanceOperator and EC2RebootInstanceOperator (#35790): @dashton90
• [ ] Add OpenLineage support to S3FileTransformOperator (#35819): @kacpermuda
• [ ] Add OpenLineage support to S3Operators - Copy, Delete and Create Object (#35796): @kacpermuda
• [ ] Added Feature: retry strategy parameter to Amazon AWS Batch Operator (#35789): @evgenyslab
• [x] Added name field to template_fields in EmrServerlessStartJobOperator (#35648): @sseung00921
• [ ] openlineage, aws: Add OpenLineage support for AthenaOperator. (#35090): @JDarDagran
• [ ] Implement login and logout in AWS auth manager (#35488): @vincbeck
• [x] Fix recent change to Batch operator's retry_strategy (#35808): @ferruzzi
• [x] Fix and reapply templates for provider documentation (#35686): @potiuk
• [ ] Make EksPodOperator exec config not rely on log level (#35771): @syedahsn
• [ ] Update emr.py fix serverless operator without runtime override (#35787): @tnyz
• [x] Updated docstring: check_key_async is now in line with description of _check_key_async (#35799): @bolleyboll
• [ ] Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
• [ ] Allow a wider range of watchtower versions (#35713): @o-nikolas
• [x] Extend task context logging support for remote logging using AWS S3 (#32950): @pankajkoti
• [ ] Log failure reason for containers if a task fails for ECS Executor (#35496): @syedahsn

  Provider apache.impala: 1.2.1rc1

• [x] Remove backcompat inheritance for DbApiHook (#35754): @Taragolis

  Provider atlassian.jira: 2.3.0rc1

• [ ] Add Jira Notifier implementation (#35397): @pubudu91

  Provider cncf.kubernetes: 7.10.0rc1

• [x] Add annotations to template_fields in KubernetesPodOperator (#35641): @shohamy7
• [x] feat: K8S resource operator - CRD (#35600): @raphaelauv
• [ ] Revert Remove PodLoggingStatus object #35422 (#35822): @dstandish
• [x] Fix K8S executor override config using pod_override_object (#35185): @hussein-awala
• [x] Fix and reapply templates for provider documentation (#35686): @potiuk
• [ ] Remove inconsequential code bits in KPO logging (#35416): @dstandish
• [x] KubernetesExecutor observability Improvements (#35579): @dirrao

  Provider common.io: 1.1.0rc1

• [x] Refactor ObjectStorage (#35612): @bolkedebruin

  Provider common.sql: 1.8.1rc1

• [x] Make pyodbc.Row and databricks.Row JSON-serializable via new make_serializable method (#32319): @Joffreybvn
• [x] Remove backcompat inheritance for DbApiHook (#35754): @Taragolis
• [x] Use reproducible builds for provider packages (#35693): @potiuk
• [x] Fix and reapply templates for provider documentation (#35686): @potiuk
• [ ] Work around typing issue in examples and providers (#35494): @uranusjr
• [x] Pre-upgrade ruff==0.0.292 changes in providers (#35053): @Taragolis

  Provider databricks: 5.0.1rc1

• [x] Make pyodbc.Row and databricks.Row JSON-serializable via new make_serializable method (#32319): @Joffreybvn

  Provider dbt.cloud: 3.4.1rc1

• [ ] added cancelled handling in dbt deferred (#35597): @ccorleis

  Provider docker: 3.8.2rc1

• [x] Refactor docker operator attribute validations and docs (#35571): @Taragolis

  Provider elasticsearch: 5.2.0rc1

• [x] Add task context logging feature to allow forwarding messages to task logs (#32646): @pankajkoti
• [x] Extend task context logging support for remote logging using Elasticsearch (#32977): @pankajkoti
• [ ] Update es read query to not use body (#34792): @dstandish
• [x] Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
• [x] Remove backcompat inheritance for DbApiHook (#35754): @Taragolis

  Provider google: 10.12.0rc1

• [ ] Add PubSub Topic params for schema_settings and message_retention_duration (#35767): @dmedora
• [ ] Add OpenLineage support to GCSToBigQueryOperator (#35778): @kacpermuda
• [ ] Add dedicated OpenLineage support for bigquery_to_gcs transfer (#35660): @kacpermuda
• [ ] Add support for driver pool, instance flexibility policy, and min_num_instances for Dataproc (#34172): @ahzaz
• [ ] Add "NON_PREEMPTIBLE" as a valid value for Dataproc workers (#35669): @tskir
• [ ] Add ability to pass impersonation_chain to BigQuery triggers (#35629): @VladaZakharova
• [x] Add a filter for local files in GoogleDisplayVideo360CreateQueryOperator (#35635): @hussein-awala
• [x] Extend task context logging support for remote logging using GCS (#32970): @pankajkoti
• [x] Fix and reapply templates for provider documentation (#35686): @potiuk
• [x] Fix the logic of checking dataflow job state (#34785): @champon1020
• [ ] Remove usage of deprecated method from BigQueryToBigQueryOperator (#35605): @MaksYermak
• [ ] Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
• [x] Remove backcompat with Airflow 2.3/2.4 in providers (#35727): @Taragolis
• [ ] Restore delegate_to param in GoogleDiscoveryApiHook (#35728): @austinletson
• [ ] Remove usage of deprecated methods from BigQueryCursor (#35606): @MaksYermak

  Provider microsoft.azure: 8.3.0rc1

• [ ] Update provider.yaml: To view Azure Synapse Pipeline Connection type in Airflow UI (#35709): @ambika-garg
• [x] Add task context logging feature to allow forwarding messages to task logs (#32646): @pankajkoti
• [x] Add operator to invoke Azure-Synapse pipeline (#35091): @ambika-garg
• [x] Extend task context logging support for remote logging using WASB (Azure Blob Storage) (#32972): @pankajkoti
• [ ] Check attr on parent not self re TaskContextLogger set_context (#35780): @dstandish
• [x] Remove backcompat with Airflow 2.3/2.4 in providers (#35727): @Taragolis

  Provider odbc: 4.2.0rc1

• [x] Make pyodbc.Row and databricks.Row JSON-serializable via new make_serializable method (#32319): @Joffreybvn

  Provider openai: 1.0.1rc1

• [ ] Refine Type Handling in OpenAI Embedding Operator to Match OpenAI Typings (#35547): @ylnsnv

  Provider opsgenie: 5.3.0rc1

• [ ] feature(providers): added OpsgenieNotifier (#35530): @Eric-Mendes

  Provider papermill: 3.5.0rc1

• [x] Extend papermill operator to support remote kernels (#34840): @akshaychitneni

  Provider redis: 3.4.1rc1

• [x] Add task context logging feature to allow forwarding messages to task logs (#32646): @pankajkoti

  Provider snowflake: 5.1.2rc1

• [x] OpenLineage integration tried to use non-existed method in SnowflakeHook (#35752): @Taragolis

  Provider trino: 5.4.1rc1

• [x] Remove backcompat inheritance for DbApiHook (#35754): @Taragolis

All users involved in the PRs: @ferruzzi @champon1020 @tskir @Eric-Mendes @pubudu91 @dashton90 @o-nikolas @raphaelauv @bolleyboll @ccorleis @MaksYermak @kacpermuda @uranusjr @akshaychitneni @dmedora @Joffreybvn @evgenyslab @ambika-garg @pankajkoti @potiuk @hussein-awala @ahzaz @tnyz @syedahsn @VladaZakharova @shohamy7 @austinletson @sseung00921 @dstandish @JDarDagran @bolkedebruin @vincbeck @ylnsnv @dirrao @Taragolis

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[raphaelauv]

https://github.com/apache/airflow/pull/35600 work :white_check_mark:

[sseung00921]

https://github.com/apache/airflow/pull/35648 work ✅

[bolleyboll]

35799 work ✅

[dashton90]

35790 works

It has caused a problem with the example DAG (addressed in #35839) but the operator changes themselves work

[eladkal]

35790 works

It has caused a problem with the example DAG (addressed in #35839) but the operator changes themselves work

Thanks. Since functionality work as expected this will not block rc1 release.

[hussein-awala]

I tested https://github.com/apache/airflow/pull/35185 and https://github.com/apache/airflow/pull/35635, they work as expected.

[pankajkoti]

Tested my set of PRs, all work fine, thank you for the release efforts!

[bolkedebruin]

I would like to see a re-discussion of https://github.com/apache/airflow/pull/32319 before including it.

[potiuk]

I would like to see a re-discussion of #32319 before including it.

I think this is misunderstanding what #32319 wants to achieve. Commented there.

[potiuk]

Conflirmed my template + reproducible builds worl (works great! PR shortly to a better PMC verification step is coming soon :) )

[dirrao]

Tested my PR #35579 and it is working as expected.

[Lee-W]

We tested it with our example DAGs with apache-airflow-providers-amazon, apache-airflow-providers-cncf-kubernetes, apache-airflow-providers-databricks, apache-airflow-providers-dbt-cloud, apache-airflow-providers-google, apache-airflow-providers-cncf-kubernetes, apache-airflow-providers-microsoft-azure, apache-airflow-providers-snowflake and everything works as expected. Thanks!

[ferruzzi]

https://github.com/apache/airflow/pull/35808 appears to be working as expected.

[Joffreybvn]

32319 works as expected !

[akshaychitneni]

https://github.com/apache/airflow/pull/34840 works as expected.

[champon1020]

34785 works as expected

[ambika-garg]

35091 https://github.com/apache/airflow/pull/35091 and #35709

https://github.com/apache/airflow/pull/35709 work as expected.

On Tue, 28 Nov 2023 at 00:14, Yoshiki Nagasaki @.***> wrote:

34785 https://github.com/apache/airflow/pull/34785 works as expected

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/35845#issuecomment-1829119603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ3NQEZK57KWQF5RAV3UQQTYGVXKFAVCNFSM6AAAAAA7ZOR6K2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRZGEYTSNRQGM . You are receiving this because you were mentioned.Message ID: @.***>

[eladkal]

Thank you everyone. Providers are released I invite everyone to help improve providers for the next release, a list of open issues can be found here.