search

databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
https://dbx.readthedocs.io
Other
439 stars 120 forks source link

feature: provide an option to deploy init_scripts as workspace files via dbx #832

Open domste opened 1 year ago

domste commented 1 year ago

Expected Behavior

Provide an option to upload (and use) cluster init-scripts to the workspace via deployment.yaml. Storing them in dbfs is deprecated (see https://docs.databricks.com/api/azure/workspace/clusters/create).

Current Behavior

It is not possible to automatically upload files referenced in deployment.conf using either file:// or file:fuse;// as a workspace file. Which on the other hand is suggested as the new best practice.

Also there is no hint that using cluster-scoped init-scripts is the way to go in the examples here

Steps to Reproduce (for bugs)

Context

We are heavily using job clusters with deployments via deployment.yaml and as init-scripts located in dbfs are not supported any more from 1st of September there should be a solution to tackle this problem using dbx .

Workspace announcement

DB clusters create API

Deprecation note on cluster-scoped init scripts in dbfs

https://learn.microsoft.com/en-us/azure/databricks/files/workspace-init-scripts

Your Environment

domste commented 1 year ago

I'd also extend that feature request to deploy init-scripts to containers (s3, adfs, ...).