Update MLOps stacks project layout - Githubissues

databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html

Apache License 2.0

460 stars 156 forks source link

Update MLOps stacks project layout #46

Closed mingyu89 closed 1 year ago

mingyu89 commented 1 year ago

Signed-off-by: Mingyu Li mingyu.li@databricks.com

Followed stacks layout proposal https://docs.google.com/document/d/1BTFOzxiVzCJ2uKN0f9N3id85pL32Gof6c0cD1ao7fNo/edit#

It's a big change. Let's focus on code correctness for now and fix/improve docs later.

Changes

Create private variable _parent_folder with value databricks-mlops-stacks
Use {{cookiecutter._parent_folder}} as parent folder and modify layout
fix tests and issues

All tests are passing

Sanity check of generating project and running Train notebook

Screen Shot 2023-02-10 at 1 41 21 PM

Screen Shot 2023-02-10 at 1 41 35 PM

new layout

databricks-mlops-stacks
- .azure
  - devops-pipelines
    - …
- .github
  - workflows
    - …
- mlops-stacks-config
  - .mlops-setup-scripts
    - cicd
      - …
    - terraform
      - …
    - README.md
  - terraform
    - prod
      - …
    - staging
      - …
    - README.md
- docs
  - images
    - ...
  - ...
- - deployment - notebooks - TriggerModelDeploy.py - feature-engineering - features - dropoff_features.py - pickup_features.py - notebooks - GenerateAndWriteFeatures.py - tests - dropoff_features_test.py - pickup_features_test.py - inference - notebooks - BatchInference.py - model-validation - notebooks - ModelValidation.py - training - data - sample.parquet - notebooks - Train.py - TrainWithFeatureStore.py - profiles - … - steps - … - tests - … - recipe.yaml
- README.md
- pytest.ini
- requirements.txt
- test-requirements.txt
- _params_testing_only.txt

mingyu89 commented 1 year ago

Closing this one. Created a new PR https://github.com/databricks/mlops-stack/pull/49