search

databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
https://dbx.readthedocs.io
Other
443 stars 122 forks source link

Support for Databricks CLI versions 0.205 and above #856

Open mrmasterplan opened 1 year ago

mrmasterplan commented 1 year ago

Expected Behavior

I expect the dbx tool to continue to support the lates and recommended versions of databricks tooling. Databricks now recommends all users to migrate to the new Databricks CLI version 0.205 or above. (Link: https://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/cli/) The version 0.18 is considered deprecated. DBX currently requires databricks-cli version 0.17. (See link to repo here: https://github.com/databrickslabs/dbx/blob/main/setup.py#L14)

Can we have a discussion how we continue to support the latests tool versions?

Current Behavior

DBX is incompatible with Databricks CLI version 0.205 or above.

Context

see above

Your Environment

mrmasterplan commented 1 year ago

I see two ways to proceed:

Option 1: Use the databricks-sdk package which is available on pypi and depend on that instead of on the CLI tool.

Option 2: Depend on Databricks CLI version 0.205 and above which needs to be installed independently. It is not avialable on Pypi.

What do people prefer? I feel that option 2 is probably easier to code.