[ISSUE] Issue with GCP databricks provider impersonation

[horkyada]

Configuration

locals {
  project_sa = "project-admin@my-project.iam.gserviceaccount.com"
}

provider "google" {
  impersonate_service_account = local.project_sa
}

provider "databricks" {
  host                   = "https://accounts.gcp.databricks.com"
  google_service_account = local.project_sa
}

Expected Behavior

Provider will use the "local.project_sa" service account for databricks resources creations.

Actual Behavior

Works as expected running locally (with authenticated as a user), but for some reason databricks provider does not impersonate the service account while running in CI (atlantis) using another service account. That atlantis service account has permissions to impersonate that service account ant it indeed happens for google resources (google provider).

I added local.project_sa email to databricks, but I also needed to add the atlantis service account in order to work. (That is the proof that the provider indeed uses the original atlantis service account and not the impersonated one).

It causes several problems e.g. lack of permissions etc.

Steps to Reproduce

terraform plan/apply in atlantis vs locally

Terraform and provider versions

• provider registry.terraform.io/databricks/databricks v1.27.0
• provider registry.terraform.io/hashicorp/google v4.84.0

Debug Output

Important Factoids

EDIT: databricks provider impersonation does not work correctly when running terraform on behalf of a service account mounted by GOOGLE_CREDENTIALS environment variable.

[mgyucht]

Thanks for reporting @horkyada! Can you include any debug logs when running from CI? Set TF_LOG=trace to get everything.

[horkyada]

Thanks.

Attached the output. Haven't found anything relevant though. The only thing is that google provider correctly impersonates that:

2023-10-05T08:06:31.919Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2023/10/05 08:06:31 [INFO] Terraform is configured with service account impersonation, original identity: ca-gcp-prod-atlantis-gcp@***.iam.gserviceaccount.com, impersonated identity: ca-gcp-dev-resman-dp-0@***.iam.gserviceaccount.com

The scenario is that I removed admin permissions for atlantis user (with altantis service account email) in databricks accounts console and ran a plan against already existing databricks resources (created manually before).

The main error is

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: cannot read mws networks: This API is disabled for users without account admin status. Contact your administrator for more information
│ 
│   with module.databricks_workspace.databricks_mws_networks.this,
│   on .terraform/modules/databricks_workspace/modules/workspace_gcp/databricks_workspace.tf line 1, in resource "databricks_mws_networks" "this":
│    1: resource "databricks_mws_networks" "this" {
│ 
╵

Which is expected given that the atlantis user is not admin any more. The problem is that it should not use the atlantis user, but the ca-gcp-dev-resman-dp-0 one.

debug_dbc.txt

[horkyada]

I am able to replicate it locally now. Our CICD is using GOOGLE_CREDENTIALS env var - when I set this env var with the service account json key databricks provider does not correctly impersonate the defined service account as a google_service_account in the provider definition. Google provider is working fine with that.

So to recap the issue: databricks provider impersonation does not work correctly when running terraform on behalf of a service account mounted by GOOGLE_CREDENTIALS environment variable.

Could you please look at it? It is a blocker for us.