search

databricks / dbt-databricks

A dbt adapter for Databricks.
https://databricks.com
Apache License 2.0
226 stars 119 forks source link

upgrade databricks-sql-connector to allow 3.2.0 or 3.3.0 #775

Open stephen-bowser opened 3 months ago

stephen-bowser commented 3 months ago

Describe the feature

It would be helpful if dbt-databricks was compatible with databricks-sql-connector 3.3.0 or 3.2.0. This would help since these versions are compatible with pyarrow (">=14.0.1,<17"), whereas 3.1.0 is restricted to ^14.0.1

Describe alternatives you've considered

N/A

Additional context

N/A

Who will this benefit?

Anyone who is interested in using the latest versions of pyarrow. In particular, the latest versions (0.19.x) of deltalake require more modern versions of pyarrow than what is currently enabled by dbt-databricks.

benc-db commented 3 months ago

This will come in 1.9; we've become conservative about when we upgrade our dependency versions due to major issues in the migration to 3.0, since users on Databricks Workflows will always get latest available versions if we don't pin them down.