mongodbatlas_database_user lifecycle ignore_changes is ignored

[Benjamin-Vogt]

Description

In our environment we need to rotate secrets regularly and as we try to achieve a high availability we don't want downtimes. To do so I'm creating a new user, provide it to our application (cluster) and delete the old one from the db once the app instances have all come back up.

I can't find a way to achieve this with mongodbatlas_database_user, so I've resorted to doing it manually. The issue I'm running into is:

1. I want to create the resources (including user) via terraform
2. I want to rotate the user/password manually (I mean I don't want to, I just couldn't find a better way)
3. I want to still be able to manage the resources via terraform

If I'm rotating the user manually, terraform tries to create it every time I run apply. It is able to (as long as the user isn't destroyed and recreated manually), but the application fails to connect instantly.

Terraform CLI and Terraform MongoDB Atlas Provider Version

Terraform v0.15.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.42.0
+ provider registry.terraform.io/mongodb/mongodbatlas v0.9.1

Steps to Reproduce

Expected Behavior

I want to exclude mongodbatlas_database_user from changes made via terraform apply.

Actual Behavior

I can't get the resource to comply to

  lifecycle {
    ignore_changes = all
  }

Debug Output

Terraform will perform the following actions:

  # module.database.mongodbatlas_database_user.elasticbeanstalk_db_user will be updated in-place
  ~ resource "mongodbatlas_database_user" "elasticbeanstalk_db_user" {
        id                 = "xxxxxxxx"
      # Warning: this attribute value will be marked as sensitive and will not
      # display in UI output after applying this change. The value is unchanged.
      ~ password           = (sensitive value)
      # Warning: this attribute value will be marked as sensitive and will not
      # display in UI output after applying this change. The value is unchanged.
      ~ username           = (sensitive)
        # (5 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[themantissa]

@Benjamin-Vogt I'm having a bit of challenge in following your description here. Logs would be a huge help but without them I'll need some more details/clarification.

So you are saying that you create the user, then rotate user/password manually - do you mean you are creating the user in Terraform and then changing the password in the UI or via the API directly and then Terraform is picking up on a change after? I'm not quite sure what you are doing so I'm not quite sure if I fully understand the follow up potential issues noted

Finally, we do have an integration we developed to include support for MongoDB Atlas database users with Hashicorp Vault: https://www.vaultproject.io/docs/secrets/databases/mongodbatlas for just this kind of use case as Terraform is not an ideal tool for secret management. I'd recommend considering Vault for this use case.

cc @nikhil-mongo

[Benjamin-Vogt]

Hi @themantissa. Sorry about that, I struggled to find the right wording. I'll try to explain it better.

• I'm creating the DB and user via Terraform
• The application is created via Terraform as well, user and password are stored in AWS secrets manager
• Once a month we need to rotate the credentials. As the app should stay online, I need to create a new user and password, store it to secrets manager and use this in our application (restart required)
• I can delete the old password/user

The issue arises solely due to the time gap between rotating the user and restarting the app, even if we had both in one terraform state. This is working fine so far as long as I don't reapply the DB (user). Terraform tries to recreate the user as It's not aware of the manual change. My Idea was to use the ignore_changes flag to simply prevent any changes to cause a recreation/update.

I can create users on the fly with a random_password resource and a timestamped username for example, but this will always replace the user the application uses at the moment, locking the app out.

Maybe I'm not understanding the resources fully.

[themantissa]

Hi @Benjamin-Vogt no worries at all. Written communication is always a bit challenging. Hopefully we can work this out.

So I don't have enough here to be sure I'm able to repro what you are doing so not sure why ignore_changes is not working in the way you expect. Is this a correct understanding - ? is where I'm not sure:

• You are creating a database user via Terraform
• You create an application via Terraform and store the username and password for the database user in AWS SM for the app to use.
• Then when you need to rotate the credentials you create a new user and password in the UI or Terraform(?) and put that in AWS SM for the app to use?
• You then delete the old password with manually in the UI or with Terraform (?)

If Terraform is trying to recreate the user because it's not aware of the manual change are you refreshing the Terraform state? Is there a reason you can't create a 2nd user with Terraform and then remove the first user? The issue here I believe is this is getting more towards config management than I@C and Terraform does not always excel in the former use.

One other thing to note is that the database user does take time to apply in Atlas as it has to be applied to all clusters in the project. You can check the status programmatically with this endpoint: https://docs.atlas.mongodb.com/reference/api/clusters-check-operation-status/

[Benjamin-Vogt]

Hi @themantissa. Thank you for your patience.

• You are creating a database user via Terraform
  • Correct. This is the initial setup.
• You create an application via Terraform and store the username and password for the database user in AWS SM for the app to use.
  • Correct, though the user and password have been stored in SM by the DB script, the application is just fetching it from there.
• Then when you need to rotate the credentials you create a new user and password in the UI or Terraform(?)
  • In the MongoDB Atlas UI
• and put that in AWS SM for the app to use?
  • Yes, followed by a restart of the application (in batches to prevent downtime). New app is started, health is checked and afterwards the old one is terminated.
• You then delete the old password with manually in the UI or with Terraform (?)
  • In the MongoDB Atlas UI

I tried not to do my manual changes (just changed the user name and password in SM), but then Terraform simply replaces the DB user and locks the application out.

By creating a 2nd user you mean create two by default and toggle the usage in the app? I could taint and recreate the user ressource not in use and then switch it in my beanstalk instance, afterwards remove the old one. I'm not sure if this wouldn't mean the same amount of manual work with the added danger of tainting the wrong one.

[nikhil-mongo]

@Benjamin-Vogt I understand that these changes have to be applied manually because the database user creation does takes time and as a good practice if something is being managed through automation, any changes manually would certainly make the situation worse.

I have a workaround for you, please see if that is feasible but as @themantissa mentioned terraform is IaC and not secret management tool, so Vault will be the better option to deal with these changes.

• Create a user initially using terraform and get it added to AWS Secrets Manager.
• In the next month, instead of creating the user manually in Atlas UI, create another resource block for the second user, create the user with terraform apply and update the user to AWS Secrets Manager.
• Now, when you have restarted all the application nodes successfully with the new credentials, you can delete the old user resource block and run terraform apply to remove it from Atlas without making any changes manually.

If you do not wish to delete the resource block from the configuration, you can simply run

terraform destroy -target RESOURCE_TYPE.NAME

This is useful for when you intend on keeping that resource code but want to temporarily destroy it and use it in next month for rotation.

[Benjamin-Vogt]

Hi @nikhil-mongo. Thanks, I guess this would be the best way (without utilizing Vault). Maybe instead of deleting/creating the configuration blocks using the count prop?

[nikhil-mongo]

@Benjamin-Vogt I am glad you found it useful. I would not suggest using count prop because it is helpful in configuring where you are comfortable with immediate deletion or creation of the resource. But in your case you are looking for the application to restart before deletion therefore count prop may hinder with the workflow.