cannot import name 'HeaderFactory' from 'databricks.sdk.core'

[martin-mm1]

Describe the bug

Since the release of the new 0.29.0 version of databricks-sdk, the dbt job running on our databricks cluster, using dbt-databricks==1.6.5 version, started to fail with the error provided below. After downgrading to version 0.28.0, by explicitly specifying it on the databricks cluster, the error is no longer observed.

Steps To Reproduce

Install dbt-databricks==1.6.5 on a databricks cluster and run your dbt models.

Expected behavior

The expected behavior would be to successfully import HeaderFactory.

Screenshots and log output

cannot import name 'HeaderFactory' from 'databricks.sdk.core' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/databricks/sdk/core.py) 07:09:00 Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 87, in wrapper result, success = func(*args, *kwargs) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 72, in wrapper return func(args, **kwargs) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 140, in wrapper profile = load_profile(flags.PROJECT_DIR, flags.VARS, flags.PROFILE, flags.TARGET, threads) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/runtime.py", line 70, in load_profile profile = Profile.render( File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 436, in render return cls.from_raw_profiles( File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 401, in from_raw_profiles return cls.from_raw_profile_info( File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 355, in from_raw_profile_info credentials: Credentials = cls._credentials_from_profile( File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 165, in _credentials_from_profile cls = load_plugin(typename) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/factory.py", line 212, in load_plugin return FACTORY.load_plugin(name) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/factory.py", line 58, in load_plugin mod: Any = import_module("." + name, "dbt.adapters") File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 855, in exec_module File "", line 228, in _call_with_frames_removed File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/init.py", line 1, in from dbt.adapters.databricks.connections import DatabricksConnectionManager # noqa File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/connections.py", line 60, in from dbt.adapters.databricks.auth import token_auth, m2m_auth File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/auth.py", line 3, in from databricks.sdk.core import CredentialsProvider, HeaderFactory, Config, credentials_provider ImportError: cannot import name 'HeaderFactory' from 'databricks.sdk.core' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/databricks/sdk/core.py)

System information

dbt-databricks==1.6.5 dbt-core==1.6.7 dbt-spark==1.6.0 python==3.9.5

Additional context

Seems like there should be a limit to what version of the databricks-sdk is installed alongside the dbt-databricks adapter for version 1.6.5 instead of pulling the most recent one.

[BramVandendriessche]

I'm having the same issue for version 1.7.5.

[saxyogi]

Its happening at my end as well. I mitigated it using prev version.

[case-k-git]

This issue is happening in my side. when execute dbt source freshness command. library dependency issue. Downgrade the databricks-sdk version(from 0.29.0 to 0.28.0) solve the issue. . May be this lib issue better to handle inside dbt-databricks.

databricks-sdk==0.28.0

Seems databricks-sdk released yesterday https://pypi.org/project/databricks-sdk/

[benc-db]

This bug only exists in old versions, as newer versions pin the SDK to a particular known-good version. What are the reasons you are pinned to old versions? I cannot backport a fix to a particular patch version, so it's more useful for me to find out why you're not upgrading than to file a bug that only exists in outdated patch versions.