`databricks warehouses set-workspace-warehouse-config` errors with `enable_serverless_compute is required`

antonsteenvoorden commented 3 months ago

Describe the issue

databricks warehouses set-workspace-warehouse-config fails when providing a JSON. The error is enable_serverless_compute is required even though this is part of the JSON on the root.

Steps to reproduce the behavior

create JSON file sql-warehouse-global-settings.json with contents

{
"security_policy":"DATA_ACCESS_CONTROL",
"enable_serverless_compute": false,
"data_access_config": [
{
  "key": "spark.sql.warehouse.dir",
  "value": "/mnt/ADLS/spark-warehouse"
}
],
"enabled_warehouse_types": [
{
  "enabled":true,
  "warehouse_type":"CLASSIC"
},
{
  "enabled":true,
  "warehouse_type":"PRO"
}
],
"sql_configuration_parameters": {}
}

Run databricks warehouses set-workspace-warehouse-config --json @./sql-warehouse-global-settings.json
See error

The same happens when:

Omitting enable_serverless_compute from the JSON
Adding an entry to enabled_warehouse_types with {"enabled":true, "warehouse_type":"SERVERLESS"}
Enabling serverless compute via the workspace settings in the databricks UI and retrying all of the above

Expected Behavior

To have set the warehouse configuration. Also according to the API documentation this is not part of the payload.

Actual Behavior

It appears the JSON is validated against something undocumented and causes an error. Additionally, this is also not returned in the json when calling databricks warehouses get-workspace-warehouse-config.

OS and CLI version

OS: MacOS Sonoma 14.4.1 Databricks-CLI: v0.222.0

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?

Debug Logs

14:56:17  INFO start pid=84298 version=0.222.0 args="/opt/homebrew/bin/databricks, warehouses, set-workspace-warehouse-config, --json, @./config/sql-warehouse-global-settings.json, --debug"
14:56:18 DEBUG PUT /api/2.0/sql/config/warehouses
> {
>   "data_access_config": [
>     {
>       "key": "spark.sql.warehouse.dir",
>       "value": "/mnt/ADLS/spark-warehouse"
>     },
>   ],
>   "enabled_warehouse_types": [
>     {
>       "enabled": true,
>       "warehouse_type": "CLASSIC"
>     },
>     {
>       "enabled": true,
>       "warehouse_type": "PRO"
>     }
>   ],
>   "security_policy": "DATA_ACCESS_CONTROL",
>   "sql_configuration_parameters": {}
> }
< HTTP/2.0 400 Bad Request
< {
<   "error_code": "INVALID_PARAMETER_VALUE",
<   "message": "enable_serverless_compute is required."
< } pid=84298 sdk=true
14:56:18 DEBUG non-retriable error: enable_serverless_compute is required. pid=84298 sdk=true
Error: enable_serverless_compute is required.
14:56:18 ERROR failed execution pid=84298 exit_code=1 error="enable_serverless_compute is required."

pietern commented 3 months ago

Thanks for reporting.

It is possible that this field hasn't been added to our API specs yet. I have flagged this with the relevant team.

In the mean time you can work around the issue by using the databricks api command and making a raw API call to /api/2.0/sql/config/warehouses with the same body.

andrewnester commented 2 months ago

This seems to be related to the issue in Go SDK when fales boolean values marked as omitempty are being dropped from request body as a result of unmarshalling

https://github.com/databricks/databricks-sdk-go/issues/1002

databricks / cli