search

microsoft / azdo-databricks

A set of Build and Release tasks for Building, Deploying and Testing Databricks notebooks
MIT License
25 stars 30 forks source link

Execute Notebook task creates new job on every execution #20

Open mikanyg opened 4 years ago

mikanyg commented 4 years ago

When running notebooks using the Execute Notebook task a job is created for every execution. I would expect the operation to be create (or update existing) job.

Over time this will bloat the job list in databricks.

carlo-quinonez commented 4 years ago

Would a PR be accepted for this? This is an issue for us as well. My plan was to add a optional parameters to Execute Notebook for an optional job name (default "AzDO Execution") and job id.

If the job name is specified, it's used to create (or update) the job name that's displayed in Databricks.

If the jobid is specified, call use the databricks jobs reset instead of jobs create.