Deploy Databricks jobs from source folder

[skalpin]

This task will update Databricks with jobs stored in a folder as JSON files. Optionally deleting jobs from databricks that do not exist in the folder. Jobs are matched by name.

For example jobs can be defined as

{
    "name": "demo",
    "new_cluster": {
        "spark_version": "6.3.x-scala2.11",
        "node_type_id": "Standard_DS3_v2",
        "spark_env_vars": {
            "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
        },
        "enable_elastic_disk": true,
        "num_workers": 8
    },
    "email_notifications": {},
    "timeout_seconds": 0,
    "schedule": {
        "quartz_cron_expression": "0 0 * * * ?",
        "timezone_id": "US/Pacific"
    },
    "notebook_task": {
        "notebook_path": "/Shared/do_nothing",
        "revision_timestamp": 0
    },
    "max_concurrent_runs": 1
}

[Vcoder89]

The first paragraph of this extension mentions "deployment of Databricks Jobs" but I've found no way of getting that done, I'm looking forward for this PR. Thanks!

[skalpin]

Unfortunately I think this repository is dead