search

databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.
https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html
Apache License 2.0
460 stars 156 forks source link

Add experiment name to terraform workflow input #53

Closed mingyu89 closed 1 year ago

mingyu89 commented 1 year ago

Add experiment name to terraform workflow input so that both staging and prod workspace can get the correct experiment name.

Test

Made the same changes to CUJ project, successfully run the training workflow with validation enabled.

Screen Shot 2023-02-17 at 10 41 28 PM Screen Shot 2023-02-17 at 10 41 40 PM
vladimirk-db commented 1 year ago

Do we need to make these changes in the integration tests as well (github actions) ?

mingyu89 commented 1 year ago

@vladimirk-db The terraform is validated by the tests(and caught the format issue). The notebook is not covered by unit test, as it doesn't have pre-filled parameters and can not run on local system.

vladimirk-db commented 1 year ago

@vladimirk-db The terraform is validated by the tests(and caught the format issue). The notebook is not covered by unit test, as it doesn't have pre-filled parameters and can not run on local system.

I meant these integration tests that run on Databricks: https://github.com/databricks/mlops-stack/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/.github/workflows/run-tests.yml

Hmm I see, validation is not there as a step (should it be there? so we automatically do validation on changes in staging).

mingyu89 commented 1 year ago

@vladimirk-db Current integration test only covers training. If model validation is in disabled state, it may not align with user's expectation if CI/CD failed due to model validation failure/compiling error.