Deploying a workflow with the same name, but different owner, as a deleted workflow, fails

brookpatten commented 1 month ago

Deploy bundle/workflow workflow-a as user-a, owned by user-a
Delete workflow-a in the UI
Deploy bundle/workflow workflow-a as user-b, owned by user-b

Deployment fails with message

Error: cannot read job: User *** does not have Admin or Manage Run or Owner or View permissions on job ***************44

with databricks_job.workflow_a,
on bundle.tf.json line 374, in resource.databricks_job.workflow_a:
374:       },

Workflow is not visible in UI to change permissions

Expected Behavior

Job should deploy, or provide a clear description of the issue, or provide steps to fix, overwrite or undelete the old job

Actual Behavior

Once the job is deleted, there is no way to change the owner. The only way to fix this seems to be to make user_b an admin and re-deploy

OS and CLI version

linux, 0.225.0

Is this a regression?

Not that I know of.

Debug Logs

As above

andrewnester commented 1 month ago

@brookpatten even though you switch identities but still deploy the same bundle from the same machine, the local cache at .databricks folder in your bundle root is still kept. This cache contains a state from previous deployment which have a reference to the old (now deleted job) which new user don't have an access to.

So one workaround would be to remove .databricks folder before deploying with a new identity.

Note: removing .databricks folder and deploying with a new identity will also create a new copy of all other resources defined in DABs.

If your goal is to preserve the rest of resources when switch to new identity, you can "unbind" deleted job before deploying with a new identity. So the flow can look like this

Deploy bundle/workflow workflow-a as user-a, owned by user-a
Delete workflow-a in the UI
Run databricks bundle deployment unbind workflow-a as user-a (this will instruct DABs to remove job from local state)
Deploy bundle/workflow workflow-a as user-b, owned by user-b
Job should be successfully deployed with user-b as an owner

dinjazelena commented 2 weeks ago

Can u comment on what unbind means more detailed? Since this usecase happens to us a lot of times. And with Databricks Job there can be only single owner and no way to change owner of already deployed job but to delete it and also delete .databricks.

andrewnester commented 2 weeks ago

@dinjazelena unbind essentially means to instruct DABs to unlink resource in YAML configuration and remote resource in workspace so they are not connected and DABs won't manage (delete, update) this remote resource anymore

databricks / cli