Inconsistent behavior in complex variable override

Describe the issue

clusters.yml

variables:
  cluster2:
    description: "team custom tags on clusters"
    type: complex
    default:
      spark_version: "14.2.x-scala2.11"
      node_type_id: "Standard_DS3_v2"
      num_workers: 4
      spark_conf:
        spark.speculation: false
        spark.databricks.delta.retentionDurationCheck.enabled: false

databricks.yml

bundle:
  name: complex-variables-multiple-files
include:
  - ./clusters.yml
resources:
  jobs:
    my_job:
      job_clusters:
        - job_cluster_key: key2
          new_cluster: ${var.cluster2}
targets:
  dev:
    variables:
      cluster2:
        type: complex
        default:
          spark_version: "14.2.x-scala2.11"
          node_type_id: "Standard_DS3_v2"
          num_workers: 4
          spark_conf:
            spark.speculation: false
            spark.databricks.delta.retentionDurationCheck.enabled: false

In cli version v0.227.1 and previous versions, databricks bundle validate -t dev --output json passes without warnings, and the variables get substituted in the jobs.

        "job_clusters": [
          {
            "job_cluster_key": "key2",
            "new_cluster": {
              "node_type_id": "Standard_DS3_v2",
              "num_workers": 4,
              "spark_conf": {
                "spark.databricks.delta.retentionDurationCheck.enabled": "false",
                "spark.speculation": "false"
              },
              "spark_version": "14.2.x-scala2.11"
            }
          }
        ],

In cli version v0.228.0, databricks bundle validate -t dev --output json returns 2 warnings:

Warning: unknown field: default
  at resources.jobs.my_job.job_clusters[0].new_cluster
  in databricks.yml:17:9

Warning: unknown field: type
  at resources.jobs.my_job.job_clusters[0].new_cluster
  in databricks.yml:16:9

and the variables do not get substituted in the jobs.

        "job_clusters": [
          {
            "job_cluster_key": "key2",
            "new_cluster": {}
          }
        ],

Configuration

Install v0.228.0 curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh Install v0.227.1 curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/v0.227.1/install.sh | sh

Steps to reproduce the behavior

Run databricks bundle validate -t dev --output json
See warnings and output

Expected Behavior

type attributes in complex variables should not be considered as a field

Actual Behavior

type attributes in complex variables is considered as a field, and all the fields within the variable are unable to parse.

OS and CLI version

v0.228.0

Is this a regression?

Yes. work for all previous version.

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

databricks / cli