[ISSUE] Issue with experimental exporter when exporting groups and users in Identity Federation enabled workspaces

[vadivelselvaraj]

When trying to export users and groups using the exporter command, it fails with the below error.

Configuration

export DATABRICKS_HOST=https://<workspace-name>.cloud.databricks.com/
export DATABRICKS_TOKEN=****

../admin/.terraform/providers/registry.terraform.io/databricks/databricks/1.6.5/darwin_amd64/terraform-provider-databricks_v1.6.5 exporter \
  -skip-interactive \
  -services=groups,users \
  -listing=groups,users

Output:

[INFO] Importing  module into /Users/vselvaraj/Gitlab/data-atlas-terraform/provider/databricks/rivian-atlas-prod-dump directory Databricks resources of groups,users services
[INFO] Using directly configured PAT authentication
[INFO] Configured pat auth: host=https://rivian-atlas-prod.cloud.databricks.com, token=***REDACTED***
[INFO] Caching groups in memory ...
[WARN] :523 - Get "https://<workspace-name>.cloud.databricks.com/api/2.0/preview/scim/v2/Groups?": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
[WARN] :523 - Get "https://<workspace-name>.cloud.databricks.com/api/2.0/preview/scim/v2/Groups?": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
[ERROR] databricks_group (groups service) listing failed: Get "https://<workspace-name>.cloud.databricks.com/api/2.0/preview/scim/v2/Groups?": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[ERROR] no resources to import
$

Important Factoids

• All these failures happened only with workspaces that have Identify Federation enabled. When I tried this with a workspace that didn't have this feature, the export worked.

[alexott]

It looks like there is a networking problem. Exporter works just fine with identity federated workspaces (tested just today), just threat them as local users

[vadivelselvaraj]

It looks like there is a networking problem. Exporter works just fine with identity federated workspaces (tested just today), just treat them as local users

Doesn't seem like a network problem. I tried this for 3 workspaces with IF enabled on 3 different days. Plus, the URL((https://.cloud.databricks.com/api/2.0/preview/scim/v2/Groups) opens up fine in my browser(with the Unauthorized error though)

[vadivelselvaraj]

@alexott Can you confirm if the DATABRICKS_HOST var was set to https://accounts.cloud.databricks.com when you ran the exporter successfully?

FYI, I tried that as well and it didn't work. Below were the errors.

$ ./export-resources.sh 
[INFO] Importing  module into /Users/vselvaraj/Gitlab/data-atlas-terraform/provider/databricks/databricks-account-dump directory Databricks resources of groups,users services
[INFO] Using directly configured PAT authentication
[INFO] Configured pat auth: host=https://accounts.cloud.databricks.com, token=***REDACTED***
[WARN] /api/2.0/preview/scim/v2/Me:401 - Databricks API (/api/2.0/preview/scim/v2/Me) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/aws-workspace 
[WARN] /api/2.0/preview/scim/v2/Me:401 - Databricks API (/api/2.0/preview/scim/v2/Me) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/aws-workspace 
[ERROR] Databricks API (/api/2.0/preview/scim/v2/Me) requires you to set `host` property (or DATABRICKS_HOST env variable) to result of `databricks_mws_workspaces.this.workspace_url`. This error may happen if you're using provider in both normal and multiworkspace mode. Please refactor your code into different modules. Runnable example that we use for integration testing can be found in this repository at https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/aws-workspace
$

[alexott]

Ok, we'll need to modify exporter to support a separate accounts SCIM API

[alexott]

@vadivelselvaraj can you test the latest versions - it mostly works now

[vadivelselvaraj]

I was able to export users and groups at the account level now using version 1.19.0.