[Bug]: "Terraform detected the following changes made outside" for changes made inside.

[halberom]

Terraform Core Version

1.6.2

AWS Provider Version

5.45.0

Affected Resource(s)

• aws_instance and aws_ec2_instance_state
• aws_instance and aws_ebs_volume
• aws_security_group and aws_vpc_security_group_ingress_rule or aws_vpc_security_group_egress_rule
• aws_iam_role if attaching managed policies via aws_iam_role_policy_attachment

that I'm aware of so far.

Expected Behavior

Terraform should only display changes under Terraform detected the following changes made outside of Terraform since the last "terraform apply"... when someone or something changes resources managed by Terraform outside of Terraform.

Also

Entities and their metadata should be listed only once in state or all matching entities should be updated at the same time if a change occurs.

Actual Behavior

Terraform shows Terraform detected the following changes made outside of Terraform since the last "terraform apply"... for changes applied by Terraform.

Probably because entities are saved in state inline and within standalone resources, but only the standalone resource gets updated during an applied change. Subsequent plans show "Terraform detected the following changes..." because the inline data has not been updated.

example combinations I've found so far include

• aws_instance and aws_ec2_instance_state
• aws_instance and aws_ebs_volume
• aws_security_group and aws_vpc_security_group_ingress_rule or aws_vpc_security_group_egress_rule
• aws_iam_role if attaching managed policies via aws_iam_role_policy_attachment

Relevant Error/Panic Output Snippet

The following output is after changing instance state from stopped to running.

plan output:

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:
  # aws_instance.this has changed
  ~ resource "aws_instance" "this" {
        id                                   = REDACTED
      ~ instance_state                       = "running" -> "stopped"

excerpt from state file:

  {
      "mode": "managed",
      "type": "aws_ec2_instance_state",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          ...
          "attributes": {
            ...
            "state": "running",
  ...
 {
      "mode": "managed",
      "type": "aws_instance",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
            ...
            "instance_state": "stopped",

Terraform Configuration Files

https://gist.github.com/halberom/4a4aaabda189df511d32926986968268

Steps to Reproduce

See the example gist, or you can verify/test with any of the combinations (and probably others) mentioned above.

• use aws_instance to create an ec2 instance
• use aws_ec2_instance_state to set the state to "running"
• apply
• run plan, note no changes
• update aws_ec2_instance_state to set the state to "stopped"
• apply
• run plan, Terraform detected the following changes, aws_instance state has changed outside of terraform.
• review the state file, you should see two different instance running states as pasted above in the example output.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

related to https://github.com/hashicorp/terraform-provider-aws/issues/22317 and https://github.com/hashicorp/terraform-provider-aws/issues/26161

I also raised https://discuss.hashicorp.com/t/query-re-entities-listed-twice-in-state-when-they-can-be-inline-but-are-created-as-individual-resources/64794 to ask about this.

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[halberom]

I think https://github.com/hashicorp/terraform-provider-aws/issues/22317 and https://github.com/hashicorp/terraform-provider-aws/issues/26161 are related, you could argue this is a duplicate. But they only mention specific cases, whereas this bug seems to be much broader.

[aristosvo]

@halberom Just out of curiosity, does it help when putting these arguments in lifecycle.ignore_changes?

[halberom]

@aristosvo I was pretty sure it wouldn't, but double checked to be sure and no luck.

Things like instance_state in aws_instance is a returned attribute, not a parameter. So while it can be added without error, I'm not sure it'd do anything regardless.

Also ignore_changes is intended for use when external systems/users might make changes to a managed resource - to stop it from overriding/reverting back to state. That's not the case here - Terraform isn't trying to apply/revert changes, it has in fact already made them.

My current workaround is to use the following logic in my Makefile

apply:
    terraform apply -auto-approve ${PLAN_OUTPUT_DIR}/${PLAN_OUTPUT_FILE}
    @terraform apply -auto-approve -refresh-only >/dev/null

which successfully forces a state refresh silently. adds a few seconds to runtime but I can live with it.

[aristosvo]

I didn't expect that to work as well, but wasn't sure 👍🏽

It feels like this is a Terraform Core issue, not something a provider will be able to resolve. This issue seems to cover it, but only offers your already used suggestion.

[halberom]

Yes quite possible. I was assuming that the provider code determined what state would be recorded for a given resource, given it's what takes the parameters, makes the api calls, and receives the responses.

[mkwtsec]

Experiencing this as well

aws_iam_role if attaching managed policies via aws_iam_role_policy_attachment terraform 1.9.3 and AWS provider 5.49

concerning because Apply with succeed without error, but omits the attachment of many if not all of the aws_iam_role_policy_attachment resources.

running a subsequent apply will show

Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:
  # module.x.module.lambda-role-example.aws_iam_role.this has changed
  ~ resource "aws_iam_role" "this" {
        id                    = "Example"
      ~ managed_policy_arns   = [
          + "arn:aws:iam::00000000000:policy/example1",
          + "arn:aws:iam::00000000000:policy/example2",
          + "arn:aws:iam::00000000000:policy/example3",
          + "arn:aws:iam::00000000000:policy/example4",
          + "arn:aws:iam::00000000000:policy/example5",
        ]
        name                  = "Example"
      + tags                  = {}
        # (11 unchanged attributes hidden)
    }

[alexander-nolan]

I have also just come across this bug when using the azurerm_network_security_group alongside the azurerm_network_security_rule resource. When defining rules within the security rule resource, these changes are detected on the NSG resource as a change outside of terraform. I have manually defined the security_rule attribute on the NSG resource, which correctly removes the security rules on the first plan/apply, but they re-appear as changes outside of terraform on the next plan.

The only solution I can think of is to define the rules inline on the network_security_group resource and remove the network_security_rule resource entirely, which unfortunately will require quite a significant code rewrite.