search

Azure / azure-databricks-operator

Kubernetes Operator for Databricks
MIT License
113 stars 48 forks source link

Updating manifest for djobs is not working as expected. #90

Open Azadehkhojandi opened 5 years ago

Azadehkhojandi commented 5 years ago

to reproduce: create a djob update the notebook path It doesn't update the notebook path but shows the successful message (it can be at Datbricks API level or Operator level)

amrasem commented 4 years ago

@Azadehkhojandi Can you please add me to this repo to be able to assign any bug to myself.

@idc101 there is a limitation at Databricks API level, some fields can't be applied to the active jobs. Reference from Databricks documentation.

The new settings of the job. These new settings replace the old settings entirely. Changes to the following fields are not applied to active runs: JobSettings.cluster_spec or JobSettings.task. Changes to the following fields are applied to active runs as well as future runs: JobSettings.timeout_second, JobSettings.email_notifications, or JobSettings.retry_policy. This field is required.

--

idc101 commented 4 years ago

If we try to update fields that are not supported, we should probably delete the job and recreate it.

amrasem commented 4 years ago

@idc101 I raised a pr to fix this bug, still under review.