search

Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 39 forks source link

[FEATURE] Support development mode #81

Open stikkireddy opened 8 months ago

stikkireddy commented 8 months ago

local/dev mode will do the following:

https://docs.databricks.com/en/dev-tools/bundles/deployment-modes.html#development-mode

The prepending tag may break imports, etc.

  1. Prepends all resources that are not deployed as files or notebooks with the prefix [dev ${workspace.current_user.userName}] and tags each deployed job and pipeline with a dev Databricks tag.
  2. Marks all related deployed Delta Live Tables pipelines as development: true. See Use development mode to run pipeline updates.
  3. Enables the use of --compute-id in related calls to the bundle deploy command, which overrides any and all existing cluster definitions that are already specified in the related bundle configuration file. Instead of using --compute-id in related calls to the bundle deploy command, you can set the compute_id mapping here, or as a child mapping of the bundle mapping, to the ID of the cluster to use.
  4. Pauses all schedules and triggers on related deployed jobs.
  5. Enables concurrent runs on all related deployed jobs.