Status of testing Providers that were prepared on February 12, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.18.0rc1

• [ ] ECS Executor - Add backoff on failed task retry (#37109): @ferruzzi
• [ ] SqlToS3Operator: feat/ add max_rows_per_file parameter (#37055): @selimchergui
• [x] Adding Amazon Neptune Hook and Operators (#37000): @ellisms
• [ ] Fix log_query to format SQL statement correctly in AthenaOperator (#36962): @ferruzzi
• [x] check sagemaker training job status before deferring SageMakerTrainingOperator (#36685): @Lee-W
• [ ] Merge all ECS executor configs following recursive python dict update (#37137): @o-nikolas
• [ ] Update default value for BatchSensor (#37234): @vincbeck
• [x] Remove info log from amazon provider-> S3 Hook -> download_file (#37211): @rawwar
• [ ] feature: S3ToRedshiftOperator templating aws_conn_id (#37195): @raphaelauv
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider apache.beam: 5.6.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider apache.drill: 2.6.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider apache.druid: 3.8.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider apache.hive: 7.0.0rc1

• [ ] Simplify hive client connection (#37043): @romsharon98
• [x] Fix pyhive hive_pure_sasl extra name (#37323): @hussein-awala

  Provider apache.livy: 3.7.2rc1

• [x] Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala

  Provider apprise: 1.2.2rc1

• [x] Simplified configuration parsing in Apprise Connection (#37202): @infohash

  Provider celery: 3.6.0rc1

• [x] Add task_acks_late configuration to Celery Executor (#37066): @ninsbl
• [ ] prevent celery command autoscale misconfig (#36576): @Bowrna

  Provider cncf.kubernetes: 8.0.0rc1

• [ ] The task is stuck in a queued state forever in case of pod launch errors (#36882): @dirrao
• [ ] Modify KPO to log container log periodically (#37279): @pankajastro
• [x] Fix occasional attr-undefined for the python_kubernetes_script (#37318): @potiuk
• [ ] Fix hanging KPO on deferrable task with do_xcom_push (#37300): @vchiapaikeo
• [x] Fix rendering SparkKubernetesOperator.template_body (#37271): @Taragolis
• [ ] Fix assignment of template field in __init__ in KubernetesPodOperator (#37010): @romsharon98
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda
• [ ] Kubernetes version bump (#37040): @dirrao

  Provider common.io: 1.3.0rc1

• [ ] AIP-58: Add object storage backend for xcom (#37058): @bolkedebruin

  Provider common.sql: 1.11.0rc2

• [ ] Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246): @dabla
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider databricks: 6.2.0rc1

• [x] Update DatabricksSqlOperator to work with namedtuple (#37025): @SuperBo
• [x] Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda
• [x] Get rid of pytest-httpx as dependency (#37334): @potiuk

  Provider dbt.cloud: 3.6.1rc1

• [x] Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider elasticsearch: 5.3.3rc1

• [x] Avoid pendulum.from_timestamp usage (#37160): @Taragolis
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider google: 10.15.0rc1

• [ ] add service_file support to GKEPodAsyncHook (#37081): @Lee-W
• [ ] Update GCP Dataproc ClusterGenerator to support GPU params (#37036): @dmedora
• [x] Create DataprocStartClusterOperator and DataprocStopClusterOperator (#36996): @molcay
• [x] Implement deferrable mode for CreateHyperparameterTuningJobOperator (#36594): @moiseenkov
• [ ] Enable '_enable_tcp_keepalive' functionality for GKEPodHook (#36999): @MaksYermak
• [ ] fix how GKEPodAsyncHook.service_file_as_context is used (#37306): @Lee-W
• [x] Fix metadata override for ComputeEngineSSHHook (#37192): @molcay
• [ ] Fix assignment of template field in __init__ in custom_job (#36789): @romsharon98
• [ ] Fix location requirement in DataflowTemplatedJobStartOperator (#37069): @shardbread
• [ ] Fix assignment of template field in __init__ in CloudDataTransferServiceCreateJobOperator (#36909): @romsharon98
• [x] Fixed the hardcoded default namespace value for GCP Data Fusion links. (#35379): @synsh
• [x] Do not ignore the internal_ip_only if set to false in Dataproc cluster config (#37014): @ahzaz
• [x] Revert protection against back-compatibilty issue with google-core-api (#37111): @potiuk
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider hashicorp: 3.6.3rc1

• [x] feat(hashicorp): Add VAULT_TOKEN support (#37337): @V0lantis
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider http: 4.9.1rc1

• [x] Bump aiohttp min version to avoid CVE-2024-23829 and CVE-2024-23334 (#37110): @hussein-awala
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda
• [x] Get rid of pytest-httpx as dependency (#37334): @potiuk

  Provider influxdb: 2.4.1rc1

• [x] Get rid of pytest-httpx as dependency (#37334): @potiuk

  Provider jdbc: 4.2.2rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider microsoft.mssql: 3.6.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider mongo: 4.0.0rc1

• [x] Adding certificate validation for ssl in mongo hook (#37214): @amoghrajesh
• [x] Change MongoHook.conn_name_attr to mongo_conn_id (#37104): @Taragolis
• [ ] Change docs type in insert_many to list of dicts (#37230): @MezentsevIlya

  Provider mysql: 5.5.3rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider oracle: 3.9.2rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider pagerduty: 3.6.2rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider postgres: 5.10.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider qdrant: 1.0.0rc1

• [x] Add Qdrant Provider (#36805): @Anush008

  Provider sftp: 4.9.0rc1

• [x] Add deferrable param in SFTPSensor (#37117): @utkarsharma2

  Provider slack: 8.6.1rc1

• [x] Remove the remaining references to use the token bypassing the Connection in the Slack provider. (#37112): @Taragolis
• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider snowflake: 5.3.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider sqlite: 3.7.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider ssh: 3.10.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider tableau: 4.4.2rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider trino: 5.6.2rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider vertica: 3.7.1rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider weaviate: 1.3.2rc1

• [x] Standardize deprecations in providers [part1] (#36876): @kacpermuda

  Provider yandex: 3.9.0rc1

• [x] Add secrets-backends section into the Yandex provider yaml definition (#37065): @Taragolis
• [ ] fix: using endpoint from connection if not specified (#37076): @ImpressionableRaccoon

All users involved in the PRs: @ellisms @moiseenkov @synsh @Anush008 @infohash @shardbread @raphaelauv @hussein-awala @molcay @pankajastro @selimchergui @dabla @rawwar @SuperBo @ninsbl @V0lantis @kacpermuda @ahzaz @dirrao

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[moiseenkov]

36594 works as expected

[moiseenkov]

@potiuk , @eladkal , hi, Could we please include #37282 into the release?

[eladkal]

@potiuk , @eladkal , hi, Could we please include #37282 into the release?

If PR missed cut off time there is no option to add it. It will have to wait for next release / RC2 if we will have one.

[pankajkoti]

Observing an issue with common.sql: 1.11.0rc2

Posted a question on the PR

[potiuk]

Observing an issue with common.sql: 1.11.0rc2

Posted a question on the PR

Nice catch @pankajkoti

[pankajkoti]

Nice catch @pankajkoti

Actually, it was the (Astronomer) Providers RC automation test suite that reported this failure :) Thank you @potiuk for providing the fix PR https://github.com/apache/airflow/pull/37359 so quick. Appreciate it 🙏🏽

[infohash]

• gh-37202 is working with Provider apprise: 1.2.2rc1

[vchiapaikeo]

https://github.com/apache/airflow/pull/37300 looks good. Also, @Lee-W , thank you for that fix to GKE providers.

However, I have deferrable tasks that should be completing that are not completing due to https://github.com/apache/airflow/pull/37279. Will comment on the PR there. cc @pankajastro @pankajkoti

Link to comment: https://github.com/apache/airflow/pull/37279#issuecomment-1938757838

[pankajastro]

However, I have deferrable tasks that should be completing that are not completing due to https://github.com/apache/airflow/pull/37279. Will comment on the PR there. cc @pankajastro @pankajkoti

Thanks for testing it. please share the DAG and other details if possible.

[ninsbl]

Hmm... I am struggeling to build a recent version of Airflow with celery 3.6.0rc1. The issue seems unrelated to the changes in the provider. I tried to build a container with 2.9.0 but there FAB provider seems to be missing / not installable...

[potiuk]

Hmm... I am struggeling to build a recent version of Airflow with celery 3.6.0rc1. The issue seems unrelated to the changes in the provider. I tried to build a container with 2.9.0 but there FAB provider seems to be missing / not installable...

Yes. FAB provider and Airflow 2.9.0 are not yet usable to install. We might want to change that and release FAB provider in the next wave

[potiuk]

Added https://github.com/apache/airflow/pull/37362 to make the provider "ready" (we will need to do it when we come closer to 2.9.0 release.

[ellisms]

37000 is good.

[molcay]

https://github.com/apache/airflow/pull/36996 and https://github.com/apache/airflow/pull/37192 works as expected

[kacpermuda]

I tested a random sample of changes from #36876 and it works OK. The PR simply changed the warnings.warn to decorator.

[amoghrajesh]

Tried out https://github.com/apache/airflow/pull/37214 using some sample DAGs in breeze environment. Looks good 👍🏽

[rawwar]

Tried out #37211

amazon provider - 8.17.0:

amazon provider - 8.18.0rc1:

Info Log is no longer printed. hence, working as expected.

[V0lantis]

Tried out #37337 https://github.com/apache/airflow/pull/37337, works like a charm.

Provider apache-airflow-providers-hashicorp==3.6.3rc1

Le lun. 12 févr. 2024 à 17:37, Kalyan @.***> a écrit :

Tried out #37211 https://github.com/apache/airflow/pull/37211

amazon provider - 8.17.0: image.png (view on web) https://github.com/apache/airflow/assets/20266953/f96c054b-a6fe-4021-a0e5-ecfa5a76c23b

amazon provider - 8.18.0rc1: image.png (view on web) https://github.com/apache/airflow/assets/20266953/f727596c-51d6-4c88-a06f-9a74e408a2a9

Log is no longer printed. hence, working as expected.

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/37358#issuecomment-1939109668, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7LNNVW4RMJ3ZBCIZAWTGDYTJANRAVCNFSM6AAAAABDERZ2CGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZGEYDSNRWHA . You are receiving this because you were mentioned.Message ID: @.***>

[hussein-awala]

Tested #37110 and #37323 and they both work as expected.

[synsh]

35379 looks good.

[o-nikolas]

While testing #37137 I encountered an unexpected behaviour with the ECS executor not retrying on failures. I don't think it was caused by this change, it's a separate regression I believe. However, this executor is still experimental so I'm not sure it should block this release. CC @ferruzzi @syedahsn @shubham22

[potiuk]

Checked all my changes. LGTM.

[Lee-W]

https://github.com/apache/airflow/pull/36685 doesn't work as expected. It stucks somehow. Will take a deeper look these days.

https://github.com/apache/airflow/pull/37306 and https://github.com/apache/airflow/pull/37081 works fine without cncf.kubernetes: 8.0.0rc1 Looks like https://github.com/apache/airflow/pull/37279#issuecomment-1938782796 is the root cause

[Anush008]

Provider qdrant: 1.0.0rc1 https://github.com/apache/airflow/pull/36805

Tested DAGs. Works.

[eladkal]

Providers amazon, cncf.kubernetes and common.sql will be excluded from RC1.

[SuperBo]

Databricks providers, confirm #37025 works when xcom is disabled. However it raises error with xcom enabled when serializing named tuple to json.

[Taragolis]

However it raises error with xcom enabled when serializing named tuple to json.

Serializing named tuple should be add into the core first, e.g. it might be added by https://github.com/apache/airflow/pull/37168

[Taragolis]

Checked all my changes, looking good

[ahzaz]

37014 works as expected.

[pankajkoti]

37117 works as expected

[utkarsharma2]

Tested Add deferrable param in SFTPSensor (#37117) https://github.com/apache/airflow/pull/37117 and Fix metadata override for ComputeEngineSSHHook (#37192) https://github.com/apache/airflow/pull/37192: @molcay https://github.com/molcay works as expected.

Thanks, Utkarsh Sharma

On Wed, Feb 14, 2024 at 2:41 PM Pankaj Koti @.***> wrote:

37117 https://github.com/apache/airflow/pull/37117 works as expected

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/37358#issuecomment-1943347598, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDLAHM3XVNAOIW7XIDQ5GTYTR5SJAVCNFSM6AAAAABDERZ2CGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGM2DONJZHA . You are receiving this because you were mentioned.Message ID: @.***>

[raphaelauv]

Fix hanging KPO on deferrable task with do_xcom_push (#37300): @vchiapaikeo

work :+1:

[pankajastro]

However, I have deferrable tasks that should be completing that are not completing due to #37279. Will comment on the PR there. cc @pankajastro @pankajkoti

Link to comment: #37279 (comment)

Hey @vchiapaikeo would it be possible for you to test https://github.com/apache/airflow/pull/37363 with your use case?

[eladkal]

However, this executor is still experimental so I'm not sure it should block this release.

Since we are having RC2 for amazon provider there is time to fix it if you can raise a PR? @o-nikolas

[ninsbl]

37066 Seems to work fine. The environment variable is acknowledged in the airflow config and long running tasks have been finnishing successfull with Celery 3.6.0rc1.

[Lee-W]

36685 doesn't work as expected. It stucks somehow. Will take a deeper look these days.

37306 and #37081 works fine without cncf.kubernetes: 8.0.0rc1 Looks like #37279 (comment) is the root cause

I tested #36685 a few more times today. The issue can no longer be reproduced.

[eladkal]

Thank you everyone. Providers are released.

Providers: amazon, common.sql, cncf.kubernetes were excluded from this release.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.