vrej-abramian commented 1 year ago

Hello, terraform apply works fine and deploys all resources, but on running terraform destroy fails to destroy the resource and stuck on Still destroying... as you may see in the following:

module.mongodb.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTBkODkxZjUzZjEzODM2YWM1ZmM3-c...pZA==:NjUzOTBiMjdkY2M3ZmMwZWZjNWQ3ZmNh, 2m20s elapsed]

Terraform CLI and Terraform MongoDB Atlas Provider Version

# terraform version
Terraform v1.5.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.14.0
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/mongodb/mongodbatlas v1.12.2
##################################################
# Tested by terraform cli `v1.6.2` as well, same results.

Terraform Configuration File

locals {
  mongo_organizationId = "<SUBSTITUDE_WITH_YOUR_ORGANIZATIONID>"
  aws_region           = "us-east-1"

  general_tag = {
    Environment = "test"
    Terraform   = "true"
  }
}
### The `cluster.tf` file content
resource "mongodbatlas_project" "mongo_project" {
  name   = test-proj
  org_id = local.mongo_organizationId
}

resource "mongodbatlas_cluster" "cluster1" {
  // General
  name       = var.mongo_clusterName
  project_id = mongodbatlas_project.mongo_project.id

  // Operations
  mongo_db_major_version  = var.engine_version
  cloud_backup = var.provider_backup_enabled

  // Provider Settings "block"
  provider_name                  = "AWS"
  provider_instance_size_name    = var.mongo_size
  provider_region_name           = var.mongo_provider_region
  auto_scaling_disk_gb_enabled   = true
  provider_volume_type           = "STANDARD"
  termination_protection_enabled = var.deletion_protection

  #depends_on = [mongodbatlas_network_container.dgcdbcontainer]

  lifecycle {
    ignore_changes = [paused]
  }
}

resource "mongodbatlas_cloud_backup_schedule" "cluster1" {
  count = var.custom_backup_policy ? 1 : 0

  project_id   = mongodbatlas_cluster.cluster1.project_id
  cluster_name = mongodbatlas_cluster.cluster1.name

  reference_hour_of_day    = var.mcbs.reference_hour_of_day
  reference_minute_of_hour = var.mcbs.reference_minute_of_hour
  restore_window_days      = var.mcbs.restore_window_days

  auto_export_enabled = var.auto_export_enabled ? var.auto_export_enabled : false
  use_org_and_group_names_in_export_prefix = var.use_org_and_group_names_in_export_prefix

  dynamic "export" {
    for_each = var.auto_export_enabled ? {fakeObj = "yes"} : {}
    content {
      export_bucket_id = mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[count.index].export_bucket_id
      frequency_type   = var.export_frequency_type
    }
  }

  // This will now add the desired policy items to the existing mongodbatlas_cloud_backup_schedule resource
  dynamic "policy_item_hourly" {
    for_each = contains(keys(var.mcbs_policy_items), "policy_item_hourly") ? {fakeObj = "yes"} : {}

    content {
      frequency_interval  = var.mcbs_policy_items.policy_item_hourly.frequency_interval
      retention_unit      = var.mcbs_policy_items.policy_item_hourly.retention_unit
      retention_value     = var.mcbs_policy_items.policy_item_hourly.retention_value
    }
  }

  dynamic "policy_item_daily" {
    for_each = contains(keys(var.mcbs_policy_items), "policy_item_daily") ? {fakeObj = "yes"} : {}

    content {
      frequency_interval  = var.mcbs_policy_items.policy_item_daily.frequency_interval
      retention_unit      = var.mcbs_policy_items.policy_item_daily.retention_unit
      retention_value     = var.mcbs_policy_items.policy_item_daily.retention_value
    }
  }

  dynamic "policy_item_weekly" {
    for_each = contains(keys(var.mcbs_policy_items), "policy_item_weekly") ? {fakeObj = "yes"} : {}

    content {
      frequency_interval  = var.mcbs_policy_items.policy_item_weekly.frequency_interval
      retention_unit      = var.mcbs_policy_items.policy_item_weekly.retention_unit
      retention_value     = var.mcbs_policy_items.policy_item_weekly.retention_value
    }
  }

  dynamic "policy_item_monthly" {
    for_each = contains(keys(var.mcbs_policy_items), "policy_item_monthly") ? {fakeObj = "yes"} : {}

    content {
      frequency_interval  = var.mcbs_policy_items.policy_item_monthly.frequency_interval
      retention_unit      = var.mcbs_policy_items.policy_item_monthly.retention_unit
      retention_value     = var.mcbs_policy_items.policy_item_monthly.retention_value
    }
  }

}

# IAM Role for MongoDB Atlas
resource "aws_iam_role" "mongodb_atlas" {
  count = var.auto_export_enabled ? 1 : 0

  name = "mongodb_atlas_integration-${mongodbatlas_cluster.cluster1.name}"

  assume_role_policy = jsonencode({
    "Version":"2012-10-17",
    "Statement":[
      {
        "Effect":"Allow",
        "Principal":{
          "AWS": mongodbatlas_cloud_provider_access_setup.setup_only[count.index].aws_config.*.atlas_aws_account_arn
        },
        "Action":"sts:AssumeRole",
        "Condition":{
          "StringEquals":{
            "sts:ExternalId": mongodbatlas_cloud_provider_access_setup.setup_only[count.index].aws_config.*.atlas_assumed_role_external_id
          }
        }
      }
    ]
  })
}

# IAM Policy for MongoDB Atlas
resource "aws_iam_policy" "mongodb_atlas_policy" {
  count = var.auto_export_enabled ? 1 : 0

  name = "mongodb_atlas_integration-${mongodbatlas_cluster.cluster1.name}"

  description = "Policy for MongoDB Atlas integration of ${mongodbatlas_cluster.cluster1.name} cluster"

  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "s3:GetBucketLocation",
        "Resource": "arn:aws:s3:::${var.snapshot_export_bucket_name}"
      },
      {
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::${var.snapshot_export_bucket_name}/*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:*",
        "Resource": [
          "arn:aws:s3:::${var.snapshot_export_bucket_name}",
          "arn:aws:s3:::${var.snapshot_export_bucket_name}/*"
        ]
      }
    ]
  })
}

# Attach MongoDB Atlas Policy to Role
resource "aws_iam_role_policy_attachment" "mongodb_atlas_policy_attach" {
  count = var.auto_export_enabled ? 1 : 0
  role       = aws_iam_role.mongodb_atlas[count.index].name
  policy_arn = aws_iam_policy.mongodb_atlas_policy[count.index].arn
}

resource "mongodbatlas_cloud_provider_access_setup" "setup_only" {
  count = var.auto_export_enabled ? 1 : 0
  project_id = mongodbatlas_cluster.cluster1.project_id
  provider_name = "AWS"   // WARNING: Changing the `provider_name` will result in destruction of the existing resource and the creation of a new resource.
}

resource "mongodbatlas_cloud_provider_access_authorization" "auth_role" {
  count = var.auto_export_enabled ? 1 : 0
  project_id =  mongodbatlas_cluster.cluster1.project_id
  role_id    =  mongodbatlas_cloud_provider_access_setup.setup_only[count.index].role_id

  aws {
    iam_assumed_role_arn = aws_iam_role.mongodb_atlas[count.index].arn
  }
}

resource "aws_s3_bucket" "mongodb_atlas_export_bucket" {
  count = var.auto_export_enabled ? 1 : 0

  bucket = var.snapshot_export_bucket_name
  force_destroy = true

  tags = {
    Name        = "Dedicated bucket for Mongo Backups"
    Environment = "Mongodb Atlas"
  }
}

resource "mongodbatlas_cloud_backup_snapshot_export_bucket" "cluster1" {
  count = var.auto_export_enabled ? 1 : 0
  project_id  = mongodbatlas_cluster.cluster1.project_id
  iam_role_id = mongodbatlas_cloud_provider_access_authorization.auth_role[count.index].role_id
  ##bucket_name = var.snapshot_export_bucket_name
  bucket_name = aws_s3_bucket.mongodb_atlas_export_bucket[count.index].id
  cloud_provider = "AWS"  // This is hardcoded since currently is the only supported provider by MongoAtlas.
}
####################################################################################

### The `variables.tf` file content
# Provide the sensitive parameters via your `terraform.tfvars` file
variable "mongo_public_key" {}
variable "mongo_private_key" {}
variable "mongo_password" {}
### ### ###

variable "provider_backup_enabled" {
  type        = bool
  default     = true
}

# To configure a custom relevant trigger time with RDS backups.
variable "custom_backup_policy" {
  type = bool
  default = true
}

# `mcbs` is abbrevation for `mongodbatlas_cloud_backup_schedule`
variable "mcbs" {
  type = object({
    reference_hour_of_day     = number
    reference_minute_of_hour  = number
    restore_window_days       = number
  })

  default = {
    reference_hour_of_day     = 16
    reference_minute_of_hour  = 10
    restore_window_days       = 1
  }
}

variable "mcbs_policy_items" {
  type = map(object({
    frequency_interval = number
    retention_unit     = string
    retention_value    = number
  }))

  default = {
    /*policy_item_hourly = {
      frequency_interval = 1          #accepted values = 1, 2, 4, 6, 8, 12 -> every n hours
      retention_unit     = "days"
      retention_value    = 1
    },*/
    policy_item_daily = {
      frequency_interval = 1          #accepted values = 1 -> every 1 day
      retention_unit     = "days"
      retention_value    = 2
    },
    policy_item_weekly = {
      frequency_interval = 4          # accepted values = 1 to 7 -> every 1=Monday,2=Tuesday,3=Wednesday,4=Thursday,5=Friday,6=Saturday,7=Sunday day of the week
      retention_unit     = "weeks"
      retention_value    = 3
    },
    policy_item_monthly = {
      frequency_interval = 40        # accepted values = 1 to 28 -> 1 to 28 every nth day of the month
                                     # accepted values = 40 -> every last day of the month
      retention_unit     = "months"
      retention_value    = 4
    }
  }
}

# Enable Auto Exporting of backups to AWS S3 Bucket
variable "auto_export_enabled" {
  type = bool
  default = true
}

variable "use_org_and_group_names_in_export_prefix" {
  type = bool
  default = true
}

variable "export_frequency_type" {
  type = string
  default = "daily"
}

variable "snapshot_export_bucket_name" {
  type = string
  default = "vrej-test-bucket"
}

variable "deletion_protection" {
  default = false
}

variable "is_production" {
  default = false
}

Steps to Reproduce

Run the following and wait for all resources to be created.

terraform apply --var-file terrafor.tfvars

Run the following to destroy resources and you will observe that it will stuck!

terraform destroy --var-file terrafor.tfvars

Expected Behavior

Destroying all the created resources successfully.

Actual Behavior

Fails to destroy the mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0] resource and stuck on Still destroying... as you may see in the following:

module.mongoatlas.mongodbatlas_cloud_backup_schedule.cluster1[0]: Destroying... [id=Y2x1c3Rlcl9uYW1l:dnJlai10ZXN0-cHJvamVjdF9pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi]
module.mongoatlas.mongodbatlas_auditing.audit: Destroying... [id=65393a81941a2e42316ea82b]
module.mongoatlas.mongodbatlas_project_ip_access_list.iplists[2]: Destroying... [id=ZW50cnk=:MTMxLjIyNi4zNC4xMzgvMzI=-cHJvamVjdF9pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi]
module.mongoatlas.mongodbatlas_database_user.user: Destroying... [id=YXV0aF9kYXRhYmFzZV9uYW1l:YWRtaW4=-cHJvamVjdF9pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi-dXNlcm5hbWU=:dnJlag==]
module.mongoatlas.mongodbatlas_project_ip_access_list.iplists[1]: Destroying... [id=ZW50cnk=:NjQuMjI3LjEyMy40Ni8zMg==-cHJvamVjdF9pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi]
module.mongoatlas.mongodbatlas_project_ip_access_list.iplists[0]: Destroying... [id=ZW50cnk=:MjEyLjU5LjY0LjY5LzMy-cHJvamVjdF9pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi]
module.mongoatlas.mongodbatlas_auditing.audit: Destruction complete after 1s
module.mongoatlas.mongodbatlas_cloud_backup_schedule.cluster1[0]: Destruction complete after 1s
module.mongoatlas.mongodbatlas_database_user.user: Destruction complete after 1s
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Destroying...  # < -- Starts here
[id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-cHJvamVjdF9pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi]
module.mongoatlas.mongodbatlas_project_ip_access_list.iplists[0]: Destruction complete after 2s
module.mongoatlas.mongodbatlas_project_ip_access_list.iplists[1]: Destruction complete after 2s
module.mongoatlas.mongodbatlas_project_ip_access_list.iplists[2]: Destruction complete after 3s
module.mongoatlas.aws_iam_role_policy_attachment.mongodb_atlas_policy_attach[0]: Destroying... [id=mongodb_atlas_integration-vrej-test-20231025160552706500000001]
module.mongoatlas.aws_iam_role_policy_attachment.mongodb_atlas_policy_attach[0]: Destruction complete after 2s
module.mongoatlas.aws_iam_policy.mongodb_atlas_policy[0]: Destroying... [id=arn:aws:iam::786064378110:policy/mongodb_atlas_integration-vrej-test]
module.mongoatlas.aws_iam_policy.mongodb_atlas_policy[0]: Destruction complete after 0s
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 10s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 20s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 30s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 40s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 50s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 1m0s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 1m10s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 1m20s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 1m30s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 1m40s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 1m50s elapsed]
module.mongoatlas.mongodbatlas_cloud_backup_snapshot_export_bucket.cluster1[0]: Still destroying... [id=aWQ=:NjUzOTNjZWM3NWRjNTgwOWUzZGI2Njc4-c...pZA==:NjUzOTNhODE5NDFhMmU0MjMxNmVhODJi, 2m0s elapsed]
.
.
.

Debug Output

Not available.

Crash Output

Not available.

Additional Context

I have attempted to remove all the count = ... and [count.index] implementations from the code, but no luck.

The only workaround I could use to destroy was going to MongoDB Atlas UI, Edit config of the relevant cluster, and disable the Cloud Backup so that within a few seconds (or on the next terraform destroy ) it succeeds to destroy the stuck resource as well as all other remaining ones.

References

github-actions[bot] commented 1 year ago

Thanks for opening this issue! Please make sure you've followed our guidelines when opening the issue. In short, to help us reproduce the issue we need:

Terraform configuration file used to reproduce the issue
Terraform log files from the run where the issue occurred
Terraform Atlas provider version used to reproduce the issue
Terraform version used to reproduce the issue
Confirmation if Terraform OSS, Terraform Cloud, or Terraform Enterprise deployment

The ticket INTMDB-1228 was created for internal tracking.

vrej-abramian commented 1 year ago

Update

I have found out that the only workaround for successful termination is to:

Update the mongodbatlas_cloud_backup_schedule resource code snippet and set auto_export_enabled to false like below:

resource "mongodbatlas_cloud_backup_schedule" "cluster1" {
count = var.custom_backup_policy ? 1 : 0

project_id   = mongodbatlas_cluster.cluster1.project_id
cluster_name = mongodbatlas_cluster.cluster1.name

reference_hour_of_day    = var.mcbs.reference_hour_of_day
reference_minute_of_hour = var.mcbs.reference_minute_of_hour
restore_window_days      = var.mcbs.restore_window_days

#auto_export_enabled = var.auto_export_enabled ? var.auto_export_enabled : false
auto_export_enabled = false
...

Run terraform apply to implement the change.
And only then running terraform destroy will destroy all resources smoothly.

This is not documented in https://registry.terraform.io/providers/mongodb/mongodbatlas/latest/docs/resources/cloud_backup_schedule or https://github.com/mongodb/terraform-provider-mongodbatlas/tree/master/examples/atlas-cloud-backup-schedule. Also, it is a blocker for automated pipelines that run terraform apply, run some tests, and then terraform destroy.

Any solutions/fixes on this would be appreciated.

Zuhairahmed commented 8 months ago

Thanks since you're unblocked going to resolve this issue, feel free to open another item incase you should need anything. thanks again!

mongodb / terraform-provider-mongodbatlas

`mongodbatlas_cloud_backup_snapshot_export_bucket` resource stuck on `Still distroying...` #1569