[ISSUE] Can not update zone_id from non-auto to auto-az

tejas-angelone commented 5 months ago

Describe the issue

The zone_id for the job cluster isn't getting updated from the non-auto availability zone to auto AZ. Whereas, the update works for non-auto AZs only. So, once a cluster is created with non-auto AZ you can switch to auto.

Issue started from CLI v0.220.0

Steps to reproduce the behavior

Create a resource config as below and deploy the bundle

job_clusters:
- job_cluster_key: job_cluster
  new_cluster:
    spark_version: 13.3.x-scala2.12
    node_type_id: i3.xlarge
    aws_attributes:
      zone_id: ap-south-1a
    autoscale:
        min_workers: 1
        max_workers: 4

Update the zone_id: auto and deploy the bundle

Expected Behavior

zone_id must update to auto

Actual Behavior

The zone_id doesn't get updated from ap-south-1a (any non-auto AZ) to auto

OS and CLI version

Databricks CLI v0.220.0

Is this a regression?

Works in Databricks CLI v0.219.0

dmarcus30 commented 4 months ago

Having same issue here

ribugent commented 1 month ago

In our case we noticed this regression today, we could update the definitions and then update manually on the web, but in our case the number of job clusters is insane and we'll lose too much time.

Unfortunately downgrading to 0.219 in our case doesn't seem to work due to the usage of complex variables.

EDIT: I ended up modifying one by one during 20 minutes all job clusters after adjusting the configuration files.

databricks / cli