databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
202 stars 100 forks source link

Create and manage persistent clusters #47

Closed Laudenlaruto closed 3 years ago

Laudenlaruto commented 3 years ago

Hello,

I was wondering if it was possible to define persistent cluster within the deployment.json file. Side by side with the jobs definition. With something like the following syntax:

{
    "default": {
        "clusters": [{
            "cluster_name": "test",
            ...   
            }
        }],
        "jobs": [{
            "name": "test",
            "existing_cluster_id": "test",
            ....
        }]
    }
}
renardeinside commented 3 years ago

Hi @Laudenlaruto ! This functionality is already provided by databricks-cli. In dbx we concentrate on CI and deployment, but the creation of a persistent cluster is more about infrastructure settings. Please use databricks CLI to create a persistent cluster, and then provide the cluster id into deployment.json.

Laudenlaruto commented 3 years ago

Ok thank you very much for the clarification.