Update Databricks provider to depend on databricks-sql-connector >= 3.1.0

[sunchao]

Description

We are trying to upgrade to urllib3 2.x but faced a few challenges in terms of dependencies w.r.t Databricks, in particular we use dbt-databricks as well as apache-airflow-providers-databricks:

• urllib3>=2 requires dbt-core1.8.x (e.g., dbt-core1.8.0b3)
• dbt-core1.8.0b3 requires dbt-databricks1.8.x (e.g., dbt-databricks1.8.0b2)
• dbt-databricks1.8.0b2 requires databricks-sql-connector<3.2.0,>=3.1.0
• However, apache-airflow-providers-databricks (latest 6.3.0 release) requires databricks-sql-connector!=2.9.0,<3.0.0,>=2.0.0.

Ideally, it'd be great if apache-airflow-providers-databricks can upgrade its dependency of databricks-sql-connector to >=3.1.0.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Feel free to attempt to upgrade it.

[Taragolis]

AFAIK databricks-sql-connector only supports SQLAlchemy 2, but Airflow and some components (FAB) only works with 1.4

[sunchao]

Thanks @Taragolis ! Yes, looks like this depends on https://github.com/apache/airflow/issues/28723