search

databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
https://dbx.readthedocs.io
Other
437 stars 119 forks source link

dbx config should support `job_tasks` #818

Open ericfeunekes opened 1 year ago

ericfeunekes commented 1 year ago

Workflow tasks should support job_tasks to enable the latest behaviour in Databricks. Named properties will also need to support job://.

stephen-bowser commented 1 year ago

This would be very helpful for me too. Is this on the roadmap at all?

jford-ctm commented 1 year ago

This would be great if dbx supported the run_job_task type, and it should be able to perform the lookup by Job Name rather than supplying the job_id as per the API.

nathnx commented 12 months ago

This would be great if dbx supported the run_job_task type, and it should be able to perform the lookup by Job Name rather than supplying the job_id as per the API.

To add, this means the job_id value has to be initialised on the workflow prior to retrieval step as the id cannot be pre-defined. In other word, the child job needs to be deployed before the parent job.