Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Option to deploy single workflow per project #111

Open nachatz opened 3 months ago

nachatz commented 3 months ago

Is your feature request related to a problem? Please describe.

Currently, when deploying a given project in Brickflow, it bundles and deploys all workflows under the project. Ideally, for versioned individual workflows we'd like to have the option to deploy just 1 workflow at a time. Ex.

Projects:
...Project1
......workflow_a.py
......workflow_b.py
...Project2
......workflow_a.py
......workflow_b.py

brickflow projects deploy --project Project1 -e local deploys workflow_a and workflow_b, a nice addition would be to have the option to only deploy workflow_a or workflow_b.

    Describe the solution you'd like

brickflow projects deploy --project Project1 --workflow workflow_a -e local with --workflow as optional

Where the bundled path could then possibly include the workflow name

...user/project/workflow/env

    Describe alternatives you've considered The alternative is creating a new project per workflow. In our case where we want a ETL pipeline's stages very carefully bundled, versioned, and released this yields 3 new projects at minimum per pipeline (with 3+ pipelines this yields 9+ minimum total projects increasing codebase size & complexity).

pariksheet commented 1 month ago

This is indeed very handy feature. However, we should enable this feature only for local development.

for higher environments, it does not make sense to deploy only set of workflows and should be part of version control.