search

databricks / bundle-examples

Examples of Databricks Asset Bundles
Other
57 stars 23 forks source link

Define all-purpose job clusters #19

Open ryanhubbell opened 5 months ago

ryanhubbell commented 5 months ago

Is there a way to define all-purpose job clusters? Looking at default_python_job, it seems that the clusters are defined per job. I don't see any way to reference the cluster from another job (at least using ${resources.jobs.job-name.job_clusters} doesn't work. Just using the cluster key doesn't work either.

munish7771 commented 1 month ago

@ryanhubbell were you able to find a workaround for this? If so, could you please share.

ryanhubbell commented 1 month ago

I was not able to do it using Databricks Asset Bundles. I instead defined it via Terraform. In fact, I'm moving all of our stuff over to just using Terraform. It supports a lot more than DAB does, and it doesn't use yaml (not my favorite).