search

JenspederM / kedro-databricks

A Databricks Plugin for Kedro
MIT License
11 stars 4 forks source link

Exception: databricks CLI is not installed #60

Open noklam opened 4 days ago

noklam commented 4 days ago

Context

I create a fresh environment today and start with a new kedro project and run into this issue

  1. kedro new
  2. pip install
  3. pip install kedro-databricks
  4. kedro init

Please select your cloud provider:

  1. Azure
  2. AWS
  3. GCP ────────────────────────────────╮ │ /workspace/kedro/.venv/bin/kedro:8 in │ │ │ │ /workspace/kedro/kedro/framework/cli/cli.py:263 in main │ │ │ │ 260 │ cli_collection = KedroCLI( │ │ 261 │ │ project_path=_find_kedro_project(Path.cwd()) or Path.cwd() │ │ 262 │ ) │ │ ❱ 263 │ cli_collection() │ │ 264 │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/core.py:1157 in call │ │ │ │ /workspace/kedro/kedro/framework/cli/cli.py:163 in main │ │ │ │ 160 │ │ ) │ │ 161 │ │ │ │ 162 │ │ try: │ │ ❱ 163 │ │ │ super().main( │ │ 164 │ │ │ │ args=args, │ │ 165 │ │ │ │ prog_name=prog_name, │ │ 166 │ │ │ │ complete_var=complete_var, │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/core.py:1078 in main │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/core.py:1688 in invoke │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/core.py:1688 in invoke │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/core.py:1434 in invoke │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/core.py:783 in invoke │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/click/decorators.py:45 in new_func │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/kedro_databricks/plugin.py:114 in init │ │ │ │ 111 │ provider: str, │ │ 112 ): │ │ 113 │ """Initialize Databricks Asset Bundle configuration""" │ │ ❱ 114 │ write_bundle_template(metadata) │ │ 115 │ write_override_template(metadata, default, _PROVIDER_MAP.get(provider)) │ │ 116 │ if require_databricks_run_script(): │ │ 117 │ │ log = logging.getLogger(metadata.package_name) │ │ │ │ /workspace/kedro/.venv/lib/python3.11/site-packages/kedro_databricks/init.py:163 in │ │ write_bundle_template │ │ │ │ 160 │ project_path = metadata.project_path │ │ 161 │ log = logging.getLogger(package_name) │ │ 162 │ if shutil.which("databricks") is None: # pragma: no cover │ │ ❱ 163 │ │ raise Exception("databricks CLI is not installed") │ │ 164 │ │ │ 165 │ config = { │ │ 166 │ │ "project_name": package_name, │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ Exception: databricks CLI is not installed
JenspederM commented 3 days ago

Do you think we should do more in terms of documenting this, or is the new error message good enough?

What I really wanted to, was to add it as a system dependency. I'm just not entirely sure how to do that in Python...

noklam commented 3 days ago

I don't think it is possible to include system dependencies in a package. The alternative is using python sdk. I have not used it before so I do not know if it can replace the CLI