[ISSUE] Provider issue databricks_mws_log_delivery returns error but creates untracked log_delivery

[ethompsy]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Databricks Provider issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Configuration

provider "databricks" {
    host = "https://accounts.cloud.databricks.com"
    username = local.username
    password = local.password
}

locals {
    username = jsondecode(data.aws_secretsmanager_secret_version.databricks_billing_credentials.secret_string)["username"]
    password = jsondecode(data.aws_secretsmanager_secret_version.databricks_billing_credentials.secret_string)["password"]
}

data "aws_secretsmanager_secret" "databricks_billing_credentials" {
  name = "databricks_billing_credentials"
}

data "aws_secretsmanager_secret_version" "databricks_billing_credentials" {
  secret_id = data.aws_secretsmanager_secret.databricks_billing_credentials.id
}

terraform {
    required_version = ">= 0.12"
    required_providers {
        aws = "~> 2.64"
        databricks = {
            source  = "databrickslabs/databricks"
            version = "0.5.7"
        }
    }
}

locals { 
    tags = var.context.tags
    billing_bucket = "sonatype-${var.context.name}-metastore-${var.context.environment}"
    databricks_account_id = jsondecode(data.aws_secretsmanager_secret_version.databricks_account_id.secret_string)["databricks_account_id"]
}

data "aws_secretsmanager_secret" "databricks_account_id" {
  name = "databricks_account_id"
}

data "aws_secretsmanager_secret_version" "databricks_account_id" {
  secret_id = data.aws_secretsmanager_secret.databricks_account_id.id
}

resource "aws_s3_bucket" "logdelivery" {
  bucket = "databricks-billing-${var.context.environment}"
  acl    = "private"
  versioning {
    enabled = false
  }
  force_destroy = true
  tags = merge(var.tags, {
    Name = "databricks-billing-${var.context.environment}"
  })
}

resource "aws_s3_bucket_public_access_block" "logdelivery" {
  bucket             = aws_s3_bucket.logdelivery.id
  ignore_public_acls = true
}

data "databricks_aws_assume_role_policy" "logdelivery" {
  external_id = local.databricks_account_id
  for_log_delivery = true
}

resource "aws_iam_role" "logdelivery" {
  name               = "${var.context.name}-billing-${var.context.environment}"
  description        = "(${var.context.name}-billing) UsageDelivery role"
  assume_role_policy = data.databricks_aws_assume_role_policy.logdelivery.json
  tags               = var.tags
}

data "databricks_aws_bucket_policy" "logdelivery" {
  full_access_role = aws_iam_role.logdelivery.arn
  bucket           = aws_s3_bucket.logdelivery.bucket
}

resource "aws_s3_bucket_policy" "logdelivery" {
  bucket = aws_s3_bucket.logdelivery.id
  policy = data.databricks_aws_bucket_policy.logdelivery.json
}

resource "databricks_mws_credentials" "log_writer" {
    account_id       = local.databricks_account_id
    credentials_name = "Usage Delivery"
    role_arn         = aws_iam_role.logdelivery.arn
}

resource "databricks_mws_storage_configurations" "log_bucket" {
    account_id                 = local.databricks_account_id
    storage_configuration_name = "Usage Logs"
    bucket_name                = aws_s3_bucket.logdelivery.bucket
}

resource "databricks_mws_log_delivery" "usage_logs" {
    account_id = local.databricks_account_id
    credentials_id = databricks_mws_credentials.log_writer.credentials_id
    storage_configuration_id = databricks_mws_storage_configurations.log_bucket.storage_configuration_id
    delivery_path_prefix = "billable-usage"
    config_name = "usage-data-${var.context.environment}"
    log_type = "BILLABLE_USAGE"
    output_format = "CSV"
    status = "DISABLED"
}

Expected Behavior

We expected the provider to create or not create the databricks_mws_log_delivery object and track the result.

Actual Behavior

When we ran the TF we got this error:

module.base.module.dbr_billing.databricks_mws_log_delivery.usage_logs: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.base.module.dbr_billing.databricks_mws_log_delivery.usage_logs,
│ provider
│ "module.base.module.dbr_billing.provider[\"registry.terraform.io/databrickslabs/databricks\"]"
│ produced an unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

But when we checked we found that the databricks_mws_log_delivery object was in fact created exactly as specified. So we ran it again to see if this was a hiccup and we got the same error but a second databricks_mws_log_delivery object was created.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Terraform and provider versions

$ terraform version
Terraform v1.1.6
on darwin_amd64
+ provider registry.terraform.io/databrickslabs/databricks v0.5.7
+ provider registry.terraform.io/hashicorp/aws v2.70.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Debug Output

https://gist.github.com/ethompsy/5d19834f1c0cb30a8fbb57870d6b3fc9

Important Factoids

Are there anything atypical about your accounts that we should know? Not that I am aware.

[nfx]

@ethompsy i don't see logs with /api/2.0/accounts/../log-delivery api call. could you try again and find those specifically?

also, i'm seeing a strange error - ReferenceTransformer: reference not found: "local.databricks_account_id" - jsondecode(data.aws_secretsmanager_secret_version.databricks_account_id.secret_string)["databricks_account_id"] seems to be empty 🤷 ... though, i see that you're making correct calls into account apis with account id c6186072-...

[ethompsy]

@nfx That snippet is pulling our Databricks account ID out of a secret in our AWS account. It isn't going to work for you, but you could swap in a string literal. (Just don't post our account number in Github ;) )

The provider blows up before creating the log. IDK how I can get the logs.

[nfx]

is it consistently blowing up?

[ethompsy]

Yes, on every run it will throw the error, but still create the databricks_mws_log_delivery object and leave it out of the TF State. This is essentially the worst possible thing it could do. We had to comment it out and create these manually (through the API) to keep moving.

[nkvuong]

@ethompsy

this happens because of status = "DISABLED"

Databricks only allows you to disable a log delivery configuration but not delete it. Hence the only way for Terraform to delete a log delivery configuration is to mark it as DISABLED. However the provider currently eagerly removed all log delivery configuration that has DISABLED status from the state file. I'll make the change so that it checks the HCL definition as well before removing the configuration.