databricks / cli

Databricks CLI
Other
132 stars 50 forks source link

Add paths field to bundle sync configuration #1694

Closed pietern closed 1 month ago

pietern commented 1 month ago

Changes

This field allows a user to configure paths to synchronize to the workspace.

Allowed values are relative paths to files and directories anchored at the directory where the field is set. If one or more values traverse up the directory tree (to an ancestor of the bundle root directory), the CLI will dynamically determine the root path to use to ensure that the file tree structure remains intact.

For example, given a databricks.yml in my_bundle that includes:

sync:
  paths:
    - ../common
    - .

Then upon synchronization, the workspace will look like:

.
├── common
│   └── lib.py
└── my_bundle
    ├── databricks.yml
    └── notebook.py

If not set behavior remains identical.

Tests

pietern commented 1 month ago

1695 needs to merge before this one; I broke it out of this one to keep this one focused.

pietern commented 1 month ago

Need to take a look at:

pietern commented 1 month ago

After discussing the metadata computation with @shreyas-goenka, we decided to back out the last commit that changes metadata computation to be relative to the sync root because it turns out the relative job path is relative to the bundle root path in the Git section of the metadata (this is the relative path of the bundle root inside the Git repository).