mongodbatlas_encryption_at_rest key rotation impossible to perform with Azure KeyVault

[gvilarino]

When using customer keys for encryption at rest it is possible to rotate encryption keys with no downtime through the MongoDB atlas portal by just specifying a different (and accessible) key ID (it even works by changing any credentials, even using a different KeyVault).

The problem is when using this module's mongodbatlas_encryption_at_rest resource: having a complete project + cluster + encryption set of resources, like so:

resource "mongodbatlas_project" "project" {
  name   = "test-project"
  org_id = "some-org-id"
}

resource "mongodbatlas_cluster" "cluster" {
  project_id = mongodbatlas_project.project.id
  name       = "test-cluster"
  encryption_at_rest_provider = "AZURE"
  provider_name               = "AZURE"

  # Other settings...
}

resource "mongodbatlas_encryption_at_rest" "encryption" {
  project_id = mongodbatlas_project.project.id

  # Disabled AWS KMS...

  azure_key_vault = {
    enabled             = true
    #...
    key_identifier      = "SOME-KEY-IDENTIFIER"
    #...
  }

  # Disabled GCE KMS...
}

Now, rotating the encryption key should just be changing the key_identifier value and applying the change. I.e.:

# no changes here...

resource "mongodbatlas_encryption_at_rest" "encryption" {
  project_id = mongodbatlas_project.project.id

  # Disabled AWS KMS...

  azure_key_vault = {
    enabled             = true
    #...
    key_identifier      = "OTHER-KEY-IDENTIFIER"
    #...
  }

  # Disabled GCE KMS...
}

However, the plan for this is not an unpdate-in-place, rather a recreation of the whole encryption configuration.

$ terraform plan

Terraform will perform the following actions:

  # mongodbatlas_encryption_at_rest.encryption must be replaced
-/+ resource "mongodbatlas_encryption_at_rest" "encryption" {
    ...

      ~ azure_key_vault  = { # forces replacement
            "azure_environment"   = "AZURE"
            "client_id"           = "38448f6a-0d05-4100-a883-23fc9bc5d1fa"
            "enabled"             = "true"
          ~ "key_identifier"      = "SOME-KEY-IDENTIFIER" -> "OTHER-KEY-IDENTIFIER"
          ...
      }
      ~ id               = "some-id" -> (known after apply)

...

If these encryption settings are used by a cluster, and if attempting to apply, you get the following error:

Error: error deleting a encryptionAtRest (some-id): PATCH https://cloud.mongodb.com/api/atlas/v1.0/groups/some-id/encryptionAtRest: 409 (request "Conflict") Cannot disable Encryption at Rest on the group because it is still enabled on one or more clusters in the group.

This makes it pragmatically impossible to perform an encryption key rotation from this module without first disabling encryption at rest entirely.

[themantissa]

@gvilarino thank you for the excellent detail. We'll have the team take a look at this and the other issue you reported.

[rvdh]

Unfortunately #128 does not seem to fix this issue. It's happening for the GCP provider as well.

This is running with the 0.4.0 provider plugin:

 # module.mongodbatlas.mongodbatlas_encryption_at_rest.encryption must be replaced
-/+ resource "mongodbatlas_encryption_at_rest" "encryption" {
      - aws_kms          = {} -> null
      - azure_key_vault  = {} -> null
      ~ google_cloud_kms = { # forces replacement
            "enabled"                 = "true"
          ~ "key_version_resource_id" = "projects/xxxx-111111/locations/europe-west1/keyRings/testing-keyring/cryptoKeys/testing-crypto_key/cryptoKeyVersions/9" -> "projects/xxxx-111111/locations/europe-west1/keyRings/testing-keyring/cryptoKeys/testing-crypto_key/cryptoKeyVersions/10"
            "service_account_key"     = jsonencode(
                {
                    auth_provider_x509_cert_url = "https://www.googleapis.com/oauth2/v1/certs"
                    auth_uri                    = "https://accounts.google.com/o/oauth2/auth"
                    client_email                = "mongodb-atlas@xxxx-111111.iam.gserviceaccount.com"
                    client_id                   = "11111111111111111111"
                    client_x509_cert_url        = "https://www.googleapis.com/robot/v1/metadata/x509/mongodb-atlas%40xxxx-111111.iam.gserviceaccount.com"
                    private_key                 = <<~EOT
                        -----BEGIN PRIVATE KEY-----
                        xxx
                        -----END PRIVATE KEY-----
                    EOT
                    private_key_id              = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
                    project_id                  = "xxxx-111111"
                    token_uri                   = "https://oauth2.googleapis.com/token"
                    type                        = "service_account"
                }
            )
        }
      ~ id               = "aaaaaaaaaaaaaaaaaaaaaaaa" -> (known after apply)
        project_id       = "aaaaaaaaaaaaaaaaaaaaaaaa"
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Warning: Resource targeting is in effect

You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.

The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.mongodbatlas.mongodbatlas_encryption_at_rest.encryption: Destroying... [id=aaaaaaaaaaaaaaaaaaaaaaaaa]

Warning: Applied changes may be incomplete

The plan was created with the -target option in effect, so some changes
requested in the configuration may have been ignored and the output values may
not be fully updated. Run the following command to verify that no other
changes are pending:
    terraform plan

Note that the -target option is not suitable for routine use, and is provided
only for exceptional situations such as recovering from errors or mistakes, or
when Terraform specifically suggests to use it as part of an error message.

Error: error removing encryption at rest (aaaaaaaaaaaaaaaaaaaaaaaa): PATCH https://cloud.mongodb.com/api/atlas/v1.0/groups/aaaaaaaaaaaaaaaaaaaaaaaa/encryptionAtRest: 409 (request "Conflict") Cannot disable Encryption at Rest on the group because it is still enabled on one or more clusters in the group.

cc @PacoDw @marinsalinas @themantissa

[themantissa]

@gvilarino PR #128 should have fixed this for Azure. Wanted to confirm it did. @rvdh we are working to solve this for GCP as well w/ #212, just fyi.

[gvilarino]

Thanks @themantissa for the heads-up. I'll give it a try as soon as I can.