databricks : Error: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API[ISSUE]

[gajendra-singh30]

I'm encountering an issue connecting Databricks to the Databricks CLI. Whenever I attempt to execute any command on Databricks, I receive the following error:

PS C:\Users> databricks catalogs list databricks : Error: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log: At line:1 char:1

• databricks catalogs list

• 
  + CategoryInfo          : NotSpecified: (Error: unexpect...s. Request log::String) [], RemoteException
  + FullyQualifiedErrorId : NativeCommandError

GET /api/2.1/unity-catalog/catalogs
> * Host: 
> * Accept: application/json
> * Authorization: REDACTED
> * User-Agent: cli/0.216.0 databricks-sdk-go/0.36.0 go/1.21.8 os/windows cmd/catalogs_list sdk-feature/pagination auth/pat
< HTTP/2.0 403 Forbidden
< * Content-Length: 75
< * Content-Type: application/json; charset=utf-8
< * Date: Wed, 27 Mar 2024 16:29:22 GMT
< * Server: databricks
< * Vary: Accept-Encoding
< * X-Databricks-Reason-Phrase: Unauthorized access to Org: 6777798819429555
< {
<   "error_code": 403,
<   "message": "Unauthorized access to Org: 6777798819429555"
< }

[m-kirsch]

I have the same issue.

[kwulffert23]

I'm getting the issue when triggering a PR from the CLI, it worked once and then it stopped working with the error: `Error: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

GET /api/2.0/preview/scim/v2/Me
> * Host: 
> * Accept: application/json
> * Authorization: REDACTED
> * User-Agent: cli/0.212.2 databricks-sdk-go/0.29.1 go/1.21.6 os/linux cmd/bundle_validate auth/pat cicd/github
< HTTP/2.0 403 Forbidden
< * Content-Length: 126
< * Content-Type: application/json; charset=utf-8
< * Date: Tue, 30 Apr [20](https://github.com/kwulffert23/kw_mlops/actions/runs/8896718188/job/24430146666#step:4:21)24 14:41:36 GMT
< * Server: databricks
< * Vary: Accept-Encoding
< * X-Databricks-Reason-Phrase: Source IP address: 40.71.163.68 is blocked by Databricks IP ACL for workspace: 1444828305810485
< {
<   "error_code": 403,
<   "message": "Source IP address: 40.71.163.68 is blocked by Databricks IP ACL for workspace: 1444828305810485"
< }

Error: Process completed with exit code 1.`

When it worked once:

All the workflows actions triggered by the PRs after the first one are failing with the same error:

[DJSaunders1997]

I'm getting the same issue, any news on a fix?

[dan-oraylis]

I am getting this issue when trying to create a secret scope in Databricks.

{ "backend_azure_keyvault": { "dns_name": "https://kv00000000001.vault.azure.net/", "resource_id": "/subscriptions/0000000000000000000000000000000000000000/resourceGroups/rg-name/providers... (46 more bytes)" }, "scope": "kv_default", "scope_backend_type": "AZURE_KEYVAULT" } < HTTP/2.0 403 Forbidden < { < "error_code": 403, < "message": "Invalid Authorization" < }: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 tf_req_id=8a85bf24-b0ff-16ea-ddd0-290bf0824f99 tf_resource_type=databricks_secret_scope @module=databricks tf_provider_addr=registry.terraform.io/databricks/databricks tf_rpc=ApplyResourceChange timestamp="2024-05-28T17:43:24.597+0200"
2024-05-28T17:43:24.602+0200 [DEBUG] provider.terraform-provider-databricks_v1.44.0.exe: non-retriable error: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

POST /api/2.0/secrets/scopes/create
* Host:
* Accept: application/json
* Authorization: REDACTED
* Content-Type: application/json
* Traceparent: 00-adb61c8dd4293befb945e5dddf439969-d069ce94551d550c-01
* User-Agent: databricks-tf-provider/1.44.0 databricks-sdk-go/0.40.1 go/1.21.9 os/windows terraform/1.6.6 resource/secret_scope auth/pat
{
  "backend_azure_keyvault": {
    "dns_name": "https://kv-care-core-dev-we.vault.azure.net/",
    "resource_id": "/subscriptions/0000000000000000000000000000000/resourceGroups/rg-name/providers/Microsoft.KeyVault/vaults/kv00000000001"
  },
  "scope": "kv_default",
  "scope_backend_type": "AZURE_KEYVAULT"
}
< HTTP/2.0 403 Forbidden
< * Access-Control-Allow-Headers: Authorization, X-Databricks-Azure-Workspace-Resource-Id, X-Databricks-Org-Id, Content-Type
< * Access-Control-Allow-Origin: *
< * Cache-Control: no-cache, no-store, must-revalidate
< * Content-Length: 52
< * Content-Type: application/json; charset=utf-8
< * Date: Tue, 28 May 2024 15:43:22 GMT
< * Expires: 0
< * Pragma: no-cache
< * Server: databricks
< * Vary: Accept-Encoding
< * X-Databricks-Reason-Phrase: Invalid Authorization
< {
<   "error_code": 403,
<   "message": "Invalid Authorization"
< }
```: tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=8a85bf24-b0ff-16ea-ddd0-290bf0824f99 @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/logger/logger.go:33 @module=databricks tf_resource_type=databricks_secret_scope tf_rpc=ApplyResourceChange timestamp="2024-05-28T17:43:24.598+0200"        
2024-05-28T17:43:24.603+0200 [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-05-28T17:43:24.610+0200 [ERROR] provider.terraform-provider-databricks_v1.44.0.exe: Response contains error diagnostic: tf_resource_type=databricks_secret_scope @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail=""
diagnostic_summary=
| cannot create secret scope: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
| ```
| POST /api/2.0/secrets/scopes/create
| > * Host:
| > * Accept: application/json
| > * Authorization: REDACTED
| > * Content-Type: application/json
| > * Traceparent: 00-adb61c8dd4293befb945e5dddf439969-d069ce94551d550c-01
| > * User-Agent: databricks-tf-provider/1.44.0 databricks-sdk-go/0.40.1 go/1.21.9 os/windows terraform/1.6.6 resource/secret_scope auth/pat
| > {
| >   "backend_azure_keyvault": {
| >     "dns_name": "https://kv0000000001/",
| >     "resource_id": "/subscriptions/00000000000000000000000000000000000/resourceGroups/rg-name/providers/Microsoft.KeyVault/vaults/kv00000000001"
| >   },
| >   "scope": "kv_default",
| >   "scope_backend_type": "AZURE_KEYVAULT"
| > }
| < HTTP/2.0 403 Forbidden
| < * Access-Control-Allow-Headers: Authorization, X-Databricks-Azure-Workspace-Resource-Id, X-Databricks-Org-Id, Content-Type
| < * Access-Control-Allow-Origin: *
| < * Cache-Control: no-cache, no-store, must-revalidate
| < * Content-Length: 52
| < * Content-Type: application/json; charset=utf-8
| < * Date: Tue, 28 May 2024 15:43:22 GMT
| < * Expires: 0
| < * Pragma: no-cache
| < * Server: databricks
| < * Vary: Accept-Encoding
| < * X-Databricks-Reason-Phrase: Invalid Authorization
| < {
| <   "error_code": 403,
| <   "message": "Invalid Authorization"
| < }
| ```

[pedrojunqueira]

same error here. Any luck?

`Error: failed to compute file content for {{.project_name}}/databricks.yml.tmpl. template: :35:31: executing "" at : error calling user_name: unexpected error handling request: json: cannot unmarshal number into Go struct field APIErrorBody.error_code of type string. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

GET /api/2.0/preview/scim/v2/Me
> * Host:
> * Accept: application/json
> * Authorization: REDACTED
> * Traceparent: 00-f7b14fbd0e1408e40c2f3bf8641ffdbe-3b77f17a3e2ec2bc-01
> * User-Agent: cli/0.220.0 databricks-sdk-go/0.41.0 go/1.21.10 os/linux cmd/bundle_init auth/pat
< HTTP/2.0 403 Forbidden
< * Content-Length: 52
< * Content-Type: application/json; charset=utf-8
< * Date: Wed, 05 Jun 2024 06:50:14 GMT
< * Server: databricks
< * Vary: Accept-Encoding
< * X-Databricks-Reason-Phrase: Invalid access token.
< {
<   "error_code": 403,
<   "message": "Invalid access token."`

[pedrojunqueira]

Solved the issue.

I went to the ~/.databrickscfg config file and updated my workspace token that was expired. Then it worked for me.

[privard]

Thanks @pedrojunqueira that solved it for me. Initially I tried putting the token in a local.tfvars file and initializing the databricks provider with it

provider "databricks" {
  alias = "workspace"
  host  = "https://dbc-XXXXXX-XXX.cloud.databricks.com"
  token = var.databricks_token
}

But terraform completely ignored it. Changing the token in my ~/.databrickscfg file did the trick. Does anyone know how to force terraform to ignore it and favor the variables?