What should happen to projects created using the old template?

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.

Other

202 stars 100 forks source link

What should happen to projects created using the old template? #36

Closed kiranvasudev closed 3 years ago

kiranvasudev commented 3 years ago

Just noticed that the entire structure of the template has been changed. Was wondering if you could tell me what should happen to all the projects built using the old template? Should they be migrated to follow the new template?

renardeinside commented 3 years ago

Hi @kiranvasudev ! In short, you're not forced to use the new version - you can just continue using the old one if you're happy with it.

However, if you prefer the structure of the old project, but the dbx functionality seems more suitable for your workloads - please feel free to use them separately:

get the dbx .whl file into an existing project structure
install it
configure an environment via dbx configure
place the deployment.json somewhere in the project folder
reference local files in the deployment.json
disable project rebuild and job definition if it's not needed via dbx deploy --no-rebuild --no-package flags

kiranvasudev commented 3 years ago

you can just continue using the old one if you're happy with it.

Unfortunately, it does not work anymore when a new project is created. Attaching the error below. If I am not wrong, I think it is something related to this commit a183c11f7bcd5b70385caba90edc8d3b60e12519.

Thank you for the instructions on how we can use dbx in the old structure ! Will definitely be useful.

renardeinside commented 3 years ago

Unfortunately, it does not work anymore when a new project is created. Attaching the error below. If I am not wrong, I think it is something related to this commit a183c11.

It's not about the project itself, but about the permissions on the default mlflow-directories - now they're restricted for direct access. You need to move the experiment artifact location out of /dbfs/databricks/mlflow-tracking because it's now a read-only directory from the user perspective (only mlflow processes can write and read from it).