search

databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
https://dbx.readthedocs.io
Other
439 stars 120 forks source link

Support for Databricks CLI versions 0.205 and above #856

Open mrmasterplan opened 10 months ago

mrmasterplan commented 10 months ago

Expected Behavior

I expect the dbx tool to continue to support the lates and recommended versions of databricks tooling. Databricks now recommends all users to migrate to the new Databricks CLI version 0.205 or above. (Link: https://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/cli/) The version 0.18 is considered deprecated. DBX currently requires databricks-cli version 0.17. (See link to repo here: https://github.com/databrickslabs/dbx/blob/main/setup.py#L14)

Can we have a discussion how we continue to support the latests tool versions?

Current Behavior

DBX is incompatible with Databricks CLI version 0.205 or above.

Context

see above

Your Environment

mrmasterplan commented 10 months ago

I see two ways to proceed:

Option 1: Use the databricks-sdk package which is available on pypi and depend on that instead of on the CLI tool.

Option 2: Depend on Databricks CLI version 0.205 and above which needs to be installed independently. It is not avialable on Pypi.

What do people prefer? I feel that option 2 is probably easier to code.