databricks / cli

Databricks CLI
Other
132 stars 50 forks source link

PythonMutator: propagate source locations #1783

Open kanterov opened 1 week ago

kanterov commented 1 week ago

Changes

Add a mechanism to load Python source locations in the Python mutator. Previously, locations pointed to generated YAML. Now, they point to Python sources instead. Python process outputs "locations.json" containing locations of bundle paths, examples:

{"path": "resources.jobs.job_0", "file": "src/examples/job_0.py", "line": 3, "column": 5}
{"path": "resources.jobs.job_0.tasks[0].task_key", "file": "src/examples/job_0.py", "line": 10, "column": 5}
{"path": "resources.jobs.job_1", "file": "src/examples/job_1.py", "line": 5, "column": 7}

Such locations form a tree, and we assign locations of the closest ancestor to each dyn.Value based on its path. For example, resources.jobs.job_0.tasks[0].task_key is located at job_0.py:10:5 and resources.jobs.job_0.tasks[0].email_notifications is located at job_0.py:3:5, because we use the location of the job as the most precise approximation.

This feature is not yet enabled by default because PyDABs don't yet ignore unknown arguments, and this will be only supported with 0.6.0.

experimental:
  pydabs:
    enabled: true
    import: [...]
    load_locations: true

Note: for now, we don't update locations with relative paths, because it has a side effect in changing how these paths are resolved

Example

% databricks bundle validate

Warning: job_cluster_key abc is not defined
  at resources.jobs.examples.tasks[0].job_cluster_key
  in src/examples/example.py:10:1

Tests

Unit tests and manually

Stacked on top of https://github.com/databricks/cli/pull/1782