search

databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.
https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html
Apache License 2.0
416 stars 141 forks source link

Databricks CLI v0.212.0 breaks MLOps Stacks bundle validation #137

Closed dangz90 closed 7 months ago

dangz90 commented 7 months ago

After upgrading to the latest version of the Databricks CLI (v0.212.0), we are experiencing issues with MLOps Stacks bundle validation. Specifically, attempting to validate a bundle now returns the following error:

Error: failed to load /databricks-resources/feature-tables-workflow-resource.yml: error unmarshaling JSON: json: cannot unmarshal object into Go struct field Root.permissions of type []resources.Permission

The error occurs in other workflow yaml files too where there are permissions set. This was not an issue in previous versions of the CLI, and we suspect that the update has caused some sort of compatibility problem.

Steps to Reproduce:

Expected Result: MLOps Stacks bundle should validate without issue.

Actual Result: The validation process fails with the error message described above.

Environment:

arpitjasa-db commented 7 months ago

Hi @dangz90 what version did you upgrade from? I believe this may be the issue where the CLI changed syntax for permissions to become a reserved word, so we had to update MLOps Stacks as well.

For example, you can see in this workflow we changed permissions to common_permissions. I'd recommend trying that out in your workflow YAML files to see if that fixes it!

dangz90 commented 7 months ago

Hi @arpitjasa-db, I upgraded the CLI from 0.209.0. I changed permissions to common_permissions as mentioned and it worked perfectly.

Thanks!

arpitjasa-db commented 7 months ago

No problem, glad to hear it worked!