Update MLOps stacks project layout

[mingyu89]

Signed-off-by: Mingyu Li mingyu.li@databricks.com

stacks layout design doc https://docs.google.com/document/d/1BTFOzxiVzCJ2uKN0f9N3id85pL32Gof6c0cD1ao7fNo/edit#bookmark=id.gklrrat3qlf8

It's a big change. Let's focus on code correctness for now and fix/improve docs later.

Decisions

1. Create one more variable for the root folder.
2. We'll just ask user what's the root folder name. Add to instructions that if user intends to use polyrepo, they should keep the root folder name same as project name.
3. Put terraform folder inside the project
4. Don't create a separate folder for projects, just add them to the root folder.

Layout

non-project related folders

• projects
  • .azure
    • devops-pipelines
    • …
  • .github
    • workflows
    • …
  • .mlops-setup-scripts
    • cicd
      • …
    • terraform
      • …
    • README.md
  • docs
    • images
      • ...
    • ...
  • README.md
  • pytest.ini
  • requirements.txt
  • test-requirements.txt
  • _params_testing_only.txt

project related folders

• projects
  • my_project1
    • terraform/bundle yaml.
    • utils.py
    • training
    • feature_engineering
    • validation
    • deployment
      • batch_inference
      • model_deployment
    • monitoring
    • data
      • sample.parquet
    • tests
      • training
      • feature_engineering
  • my_project2
  • my_project3
  • utils

---

Test

All tests are passing

Sanity check of generating project

run Train notebook

run batch inference

[mingyu89]

Not ready for review

[mingyu89]

Ready for review

[mingyu89]

Ready for review again:) @vladimirk-db @zhe-db @arpitjasa-db

[mingyu89]

Tested running training notebook and batch inference in databricks workspace.

[mingyu89]

@arpitjasa-db I added you to my repo. It's the refactor2 branch. https://github.com/mingyu89/test-repo1/tree/refactor2 Please note that the root folder name is not visible because it's also the git root folder. cc @zhe-db @vladimirk-db

[mingyu89]

@vladimirk-db I feel it better too to have one less layer for users. I made the change and tested by generating project and run the training and inference on databricks.