Support triggering of Databricks Workflows

[paulgrainger85]

Feature description

Add the ability to trigger Databricks workflows via the plugin. Right now this can be achieved using the API

    - id: databricks_workflow
      type: io.kestra.plugin.core.http.Request
      uri: "{{secret('databricks-host')}}/api/2.0/jobs/run-now"
      method: POST
      headers:
        authorization: "Bearer {{secret('databricks-secret')}}"
      body: |
        {
           "job_id": "{{namespace.databricks_workflow_a_id}}"
        }

https://docs.databricks.com/en/jobs/jobs-2.0-api.html

[loicmathieu]

We already support:

• Create: https://docs.databricks.com/en/jobs/jobs-2.0-api.html#create
• Submit: https://docs.databricks.com/en/jobs/jobs-2.0-api.html#runs-submit

I'm not sure I understand the difference with RunNow but this can be added

[loicmathieu]

And in fact ou Create hob task, create the Job and runs it so I don't think we need a RunNow task, see https://kestra.io/plugins/plugin-databricks/tasks/job/io.kestra.plugin.databricks.job.createjob

[Ben8t]

Indeed, @paulgrainger85 is there a difference with Workflow maybe ? (I'll double check after afternoon meetings)

[paulgrainger85]

It's not clear how I would provide the job ID in this case. There is configuration for Spark job, Notebook, however Workflows require a job ID and its not clear how I could use this plugin

[paulgrainger85]

My understanding is that we are missing this

We support all the task types except the run_job_task

[loicmathieu]

OK, in this case it's not lot of work. @mgabelle do you want to have a look? I can give you some explanation if needed as I do the original tasks.

[Ben8t]

@loicmathieu yes @mgabelle is starting on this - thx for supporting him if needed 🙏