[ISSUE] Problems with dbt-databricks 1.6.4

[luisvicenteatprima]

Description

Release 0.29.0 broke dbt projects based on dbt-databricks 1.6.4

Reproduction

Doing a fresh reinstall of dbt-databricks 1.6.4 and running dbt list fails with the following error

12:24:40  Running with dbt=1.6.3
12:24:41  Encountered an error:
cannot import name 'HeaderFactory' from 'databricks.sdk.core' (.../miniconda3/envs/dbterror/lib/python3.10/site-packages/databricks/sdk/core.py)
12:24:41  Traceback (most recent call last):
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/cli/requires.py", line 87, in wrapper
    result, success = func(*args, **kwargs)
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/cli/requires.py", line 72, in wrapper
    return func(*args, **kwargs)
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/cli/requires.py", line 140, in wrapper
    profile = load_profile(flags.PROJECT_DIR, flags.VARS, flags.PROFILE, flags.TARGET, threads)
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/config/runtime.py", line 70, in load_profile
    profile = Profile.render(
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/config/profile.py", line 436, in render
    return cls.from_raw_profiles(
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/config/profile.py", line 401, in from_raw_profiles
    return cls.from_raw_profile_info(
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/config/profile.py", line 355, in from_raw_profile_info
    credentials: Credentials = cls._credentials_from_profile(
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/config/profile.py", line 165, in _credentials_from_profile
    cls = load_plugin(typename)
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/adapters/factory.py", line 212, in load_plugin
    return FACTORY.load_plugin(name)
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/adapters/factory.py", line 58, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
  File ".../miniconda3/envs/dbterror/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/adapters/databricks/__init__.py", line 1, in <module>
    from dbt.adapters.databricks.connections import DatabricksConnectionManager  # noqa
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/adapters/databricks/connections.py", line 59, in <module>
    from dbt.adapters.databricks.auth import token_auth, m2m_auth
  File ".../miniconda3/envs/dbterror/lib/python3.10/site-packages/dbt/adapters/databricks/auth.py", line 3, in <module>
    from databricks.sdk.core import CredentialsProvider, HeaderFactory, Config, credentials_provider
ImportError: cannot import name 'HeaderFactory' from 'databricks.sdk.core' (.../miniconda3/envs/dbterror/lib/python3.10/site-packages/databricks/sdk/core.py)

Expected behavior

dbt should not fail

Is it a regression?

It works with version 0.28.0.

Debug Logs

N/A

Other Information

• OS: WSL2

Additional context

This library is a transitive dependency of dbt-databricks