Status of testing Providers that were prepared on October 27, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 9.1.0rc1

• [ ] feat: add OpenLineage support for RedshiftToS3Operator (#41632): @kacpermuda Linked issues:
  • [ ] Linked Issue #41575: @Artuz37
• [ ] Add SageMakerProcessingSensor (#43144): @vVv-AA
• [ ] Make RedshiftDataOperator handle multiple queries (#42900): @jroachgolf84
• [ ] Remove sqlalchemy-redshift dependency (#43271): @mobuchowski
• [ ] feat(providers/amazon): Use asset in common provider (#43110): @Lee-W
• [x] Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954): @gopidesupavan

  Provider apache.beam: 5.9.0rc1

• [ ] Add early job_id xcom_push for google provider Beam Pipeline operators (#42982): @olegkachur-e

  Provider apache.druid: 3.12.0rc1

• [ ] Add possibility to override the conn type for Druid (#42793): @Rasnar

  Provider apache.hdfs: 4.6.0rc1

• [ ] added MultipleFilesWebHdfsSensor (#43045): @eilon246810
• [x] Fixed failing static checks & provider tests (#43122): @kaxil

  Provider apache.livy: 3.9.2rc1

• [x] Add lower bound to asgiref in providers (#43001): @rawwar

  Provider apache.spark: 4.11.2rc1

• [ ] Changed conf property from str to dict in SparkSqlOperator (#42835): @duhizjame

  Provider cloudant: 4.0.2rc1

• [ ] More correctly work around uv sync issue with python 3.10+ only dep (#43227): @ashb
• [ ] Update cloudant versions to be py-version aware so uv lock can work. (#43217): @ashb Linked issues:
  • [ ] Linked Issue #41555: @topherinternational
  • [ ] Linked Issue #21004: @uranusjr

    Provider cncf.kubernetes: 9.0.1rc1

• [ ] (fix): HybridExecutor tasks of other executor rescheduled in kubernetes executor (#43003): @pavansharma36 Linked issues:
  • [ ] Linked Issue #42151: @iw-pavan
• [ ] fix: use instance base_container_name to fetch logs on trigger_reentry (#42960): @peloyeje
• [x] Add kubernetes_conn_id to templated fields (#42786): @gopidesupavan Linked issues:
  • [ ] Linked Issue #42546: @nikhilkarve
• [x] ✨ Allow node_selector templating in KPO (#43051): @bdsoha
• [ ] Adding support for Kubernetes package version 31.0.0 (#42907): @dirrao

  Provider common.sql: 1.19.0rc1

• [ ] Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357): @dabla
• [ ] Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891): @dabla
• [x] SqlSensor enhancement: #37437 (#43107): @vVv-AA Linked issues:
  • [x] Linked Issue #37437: @amolsr
• [x] fix mypy failure(provider) in SqlSensor (#43206): @rawwar

  Provider databricks: 6.12.0rc1

• [x] Add TimeoutError to be a retryable error in databricks provider (#43137): @rawwar Linked issues:
  • [x] Linked Issue #43128: @rawwar
• [x] Add ClientConnectorError to be a retryable error in databricks provider (#43091): @rawwar
• [ ] DatabricksHook: fix status property to work with ClientResponse used in async mode (#43333): @lucafurrer Linked issues:
  • [ ] Linked Issue #43269: @lucafurrer
• [x] [DatabricksHook] Respect connection settings (#42618): @xitep

  Provider dbt.cloud: 3.11.1rc1

• [x] Add lower bound to asgiref in providers (#43001): @rawwar

  Provider fab: 1.5.0rc1

• [ ] feat(providers/fab): Use asset in common provider (#43112): @Lee-W
• [ ] Fix revoke Dag stale permission on airflow < 2.10 (#42844): @joaopamaral Linked issues:
  • [ ] Linked Issue #42743: @RostD
• [ ] Bump Flask-AppBuilder to 4.5.2 (#43309): @kaxil
• [x] Upgrade FAB to 4.5.1 (#43251): @potiuk
• [ ] Move user and roles schemas to fab provider (#42869): @vincbeck
• [ ] Move the session auth backend to FAB auth manager (#42878): @vincbeck Linked issues:
  • [ ] Linked Issue #42634: @vincbeck

    Provider google: 10.25.0rc1

• [ ] feat: add Hook Level Lineage support for GCSHook (#42507): @kacpermuda Linked issues:
  • [ ] Linked Issue #40819: @mobuchowski
• [ ] feat: sensor to check status of Dataform action (#43055): @steve148
• [ ] Create Operators for Google Cloud Vertex AI Context Caching (#43008): @CYarros10
• [x] Fix outdated CloudRunExecuteJobOperator docs (#43195): @p13rr0m
• [ ] Fix TestTranslationLegacyModelPredictLink dataset_id error for google provider translation operator (#42463): @olegkachur-e
• [ ] Add a debug log for creating batch workloads in dataproc (#43265): @romsharon98
• [x] add min version to plyvel (#43129): @rawwar
• [ ] vertex ai training operators: add display_name to rendered fields (#43028): @WSHoekstra Linked issues:
  • [ ] Linked Issue #43027: @WSHoekstra
• [x] Make google provider pyarrow dependency explicit (#42996): @saucoide Linked issues:
  • [x] Linked Issue #42924: @saucoide
• [x] Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954): @gopidesupavan

  Provider http: 4.13.2rc1

• [x] fix: HttpSensorTrigger to include method when serializing (#42925): @rawwar
• [ ] Use url_from_endpoint inside HttpHook. (#42785): @simi
• [x] add lowerbound to requests-toolbelt and replace requests_toolbelt with requests-toolbelt (#43020): @rawwar Linked issues:
  • [x] Linked Issue #42989: @potiuk
• [x] Add lower bound to asgiref in providers (#43001): @rawwar

  Provider jenkins: 3.7.2rc1

• [ ] Remove unnecessary return value from jenkins_request_with_headers (#43207): @romsharon98

  Provider microsoft.azure: 11.0.0rc1

• [x] Add context to Azure Service Bus Message callback (#43370): @perry2of5
• [ ] Feature: Added event_handler parameter in MSGraphAsyncOperator (#42539): @dabla
• [ ] Add documentation for the PowerBIDatasetRefresh Operator. (#42754): @ambika-garg
• [x] Add upperbound to microsoft-kiota-abstractions (#43021): @rawwar
• [x] Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954): @gopidesupavan

  Provider mongo: 4.2.2rc1

• [x] fix(providers/mongo): prevent applying lower method on bool field (#43024): @josix Linked issues:
  • [x] Linked Issue #42930: @FarouziAbir

    Provider mysql: 5.7.3rc1

• [x] Improve the cursor type definition of mysql (#43376): @yangyulely

  Provider odbc: 4.8.0rc1

• [ ] Also use ODBC connection for sqlalchemy engine in OdbcHook like JdbcHook (#43145): @dabla

  Provider openlineage: 1.13.0rc1

• [ ] feat(providers/openlineage): Use asset in common provider (#43111): @Lee-W
• [ ] Ignore attr-defined for compat import (#43301): @uranusjr Linked issues:
  • [ ] Linked Issue #43142: @Lee-W
• [ ] nit: remove taskgroup's tooltip from OL's AirflowJobFacet (#43152): @kacpermuda
• [ ] require 1.2.1 common.compat for openlineage provider (#43039): @mobuchowski

  Provider oracle: 3.12.0rc1

• [x] Add sequence insert support to OracleHook (#42947): @Lee2532

  Provider slack: 8.9.1rc1

• [ ] adding support for snippet type in slack api (#43305): @Bowrna
• [ ] passing the filetype for SlackAPIFileOperator (#43069): @Bowrna Linked issues:
  • [ ] Linked Issue #42889: @hzlmn

    Provider ssh: 3.14.0rc1

• [x] SSHHook expose auth_timeout parameter (#43048): @gyandeeps

All users involved in the PRs: @steve148 @gyandeeps @mobuchowski @gopidesupavan @uranusjr @dabla @kacpermuda @eilon246810 @kaxil @ambika-garg @Bowrna @joaopamaral @pavansharma36 @rawwar @vVv-AA @bdsoha @olegkachur-e @p13rr0m @romshar

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Checked my changes - all look good !

[josix]

Checked mongo: 4.2.2rc1, it works as expected, thanks!

[bdsoha]

Checked cncf kubernetes/9.0.1rc1, looks OK!

[Lee2532]

checked oracle: 3.12.0rc1, it works, thanks!

[rawwar]

Checked following and don't see any issues.

43001 , #43206, #43001 ,#43129 ,#42925 ,#43020 , #43001 ,#43021

[gyandeeps]

Checked ssh: 3.14.0rc1 and it looks good. Thanks.

[perry2of5]

Azure Service Bus message callbacks look good in 11.0.0rc1 (from PR #43370)

[p13rr0m]

Checked https://github.com/apache/airflow/pull/43195. Looks good as well.

[xitep]

Verified #42618 locally with apache-airflow-providers-databricks==6.12.0rc1 ... works as expected for me

[saucoide]

checked #42996 / #42924

[yangyulely]

Checked mysql: 5.7.3rc1, all looks good.

[Lee-W]

amazon: 9.1.0rc1 and fab: 1.5.0rc1 do not work with airflow 2.10.x on dataset permissions.

corresponding PRs have been sent

https://github.com/apache/airflow/pull/43470 https://github.com/apache/airflow/pull/43469

[gopidesupavan]

Checked my changes all are okay!..

[eladkal]

I will exclude amazon and fab from this release

[vVv-AA]

Tested SqlSensor enhancement: #37437 (#43107) Looks ok.

On Wed, 30 Oct 2024, 12:35 Elad Kalif, @.***> wrote:

I will exclude amazon and fab from this release

— Reply to this email directly, view it on GitHub https://github.com/apache/airflow/issues/43416#issuecomment-2446020229, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAMDO67YQGZYERCZYBRO4LZ6CAJ5AVCNFSM6AAAAABQV42RR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBWGAZDAMRSHE . You are receiving this because you were mentioned.Message ID: @.***>

[dabla]

Will check mine tonight or tomorrow

[eladkal]

Thank you everyone. Providers are released. fab and amazon providers are excluded from this wave

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[dabla]

43357 43145

A bit late, but works as expected