Replace Databricks CLI subprocess calls with Databricks SDK for Python?

JenspederM / kedro-databricks

A Databricks Plugin for Kedro

MIT License

10 stars 4 forks source link

Replace Databricks CLI subprocess calls with Databricks SDK for Python? #8

Open astrojuanlu opened 1 month ago

astrojuanlu commented 1 month ago

https://github.com/databricks/databricks-sdk-py/

The Databricks SDK for Python is beta but at least

it can be installed with pip (unlike the newest Databricks CLI)
and users would probably get better error messages when things go wrong (instead of cryptic "subprocess exited with status 1" kind of thing)
and maybe easier to maintain even?

Wdyt @JenspederM ?

JenspederM commented 1 month ago

AFAIK, the SDK doesn't support asset bundles. The bundle architecture is more than just deployment, it's also validation and environment separation, all things that would have to be implemented from scratch with the sdk.

Even validation would have to be implemented separately, as they chose to use dataclasses instead of a proper validation tool such as pydantic.

I can give it another look, but I feel as if I went pretty deep the first time around 😊

astrojuanlu commented 1 month ago

Indeed https://github.com/databricks/databricks-sdk-py/issues/628

Thanks for the quick response! Maybe we can wait a bit longer and see if the issue above is addressed