[Bug]: Ip address of instance was not updated. TF reports that EC2 will be updated in place, but it does not.

[EugenKon]

Terraform Core Version

v1.8.5

AWS Provider Version

v5.55.0

Affected Resource(s)

• aws_instance

Expected Behavior

The instance is not updated in place, because its type was changed and reboot will occur.

Actual Behavior

The instance is rebooted and its IP was changed

Relevant Error/Panic Output Snippet

Another example of wrong state after terraform plan/apply:

Terraform will perform the following actions:

  # module.private-cloud.aws_instance.server[0] will be updated in-place
  ~ resource "aws_instance" "server" {
        id                                   = "i-XX"
      ~ instance_type                        = "t3.micro" -> "t2.micro"
        tags                                 = {
            "ConsulAutoJoin" = "nomad-auto-join"
            "Name"           = "nomad-server-0"
            "NomadType"      = "server"
        }
        # (39 unchanged attributes hidden)

        # (8 unchanged blocks hidden)
    }

  # module.private-cloud.aws_instance.www[0] will be updated in-place
  ~ resource "aws_instance" "www" {
        id                                   = "i-XX"
      ~ instance_type                        = "m5.xlarge" -> "m4.xlarge"
        tags                                 = {
            "ConsulAutoJoin" = "nomad-auto-join"
            "Name"           = "nomad-client-0"
            "NomadType"      = "client"
        }
        # (39 unchanged attributes hidden)

        # (9 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.
Saved the plan to: ../../state.d/terraform.plan

And after apply:

ConnectionInfo = <<EOT

Client EC2 instances: x223
Server EC2 instances: x.66

The Nomad UI can be accessed at http://x66:4646/ui
The Consul UI can be accessed at http://x.66:8500/ui
with the token: x

SSH into client: ssh -i state.d/nomad-www.pem ubuntu@x.223
SSH into server: ssh -i state.d/nomad-www.pem ubuntu@x.66

EOT
consul_server = "http://x.66"
consul_token = <sensitive>

ecrRegistryUrl = "x.dkr.ecr.us-west-2.amazonaws.com"
iguides_bucket_name = "nomad-portal-iguides"
lambdaNames = {
  "autodraft" = "nomad_autodraft"
}
private_ip = "x245"
project_name = "nomad"
public_ip = "x223"
secret_key = <sensitive>
user_bucket_name = "nomad-portal-users"

If I plan with refresh-only option then changes are visible:

Changes to Outputs:
  ~ ConnectionInfo      = <<-EOT
      - Client EC2 instances: x.223
      + Client EC2 instances: x.151
      - Server EC2 instances: x.66
      + Server EC2 instances: x50

      - The Nomad UI can be accessed at http://5x.66:4646/ui
      + The Nomad UI can be accessed at http://x50:4646/ui
      - The Consul UI can be accessed at http://5x.66:8500/ui
      + The Consul UI can be accessed at http://x.50:8500/ui
        with the token: xx

      - SSH into client: ssh -i state.d/nomad-www.pem ubuntu@x.223
      + SSH into client: ssh -i state.d/nomad-www.pem ubuntu@x.151
      - SSH into server: ssh -i state.d/nomad-www.pem ubuntu@x.66
      + SSH into server: ssh -i state.d/nomad-www.pem ubuntu@x.50
    EOT
  ~ consul_server       = "http://x.66" -> "http://x.50"
  ~ public_ip           = "x.223" -> "x.151"

You can apply this plan to save these new output values to the Terraform state, without
changing any real infrastructure.

Terraform Configuration Files

resource "aws_instance" "server" {
  instance_type           = local.server_instance_type
  ami                     = "ami-xxx"
  availability_zone       = "${var.aws_region}a"
  key_name                = aws_key_pair.ssh.key_name
  disable_api_termination = false

  root_block_device {
    volume_type           = "gp3"
    volume_size           = 8
    delete_on_termination = true
  }

  subnet_id                   = aws_subnet.main_a.id
  associate_public_ip_address = true
  iam_instance_profile        = aws_iam_instance_profile.www.name
}

Steps to Reproduce

1. Create an instance
2. Change its type from t3 -> t2
3. Apply the changes.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @EugenKon 👋 Thank you for taking the time to raise this! Are you able to supply a full configuration that can be used to reproduce the issue? I notice from the logging that you provided that you're using modules and outputs -- the way those things are configured can have an impact.

[EugenKon]

@justinretzolk I can, but I can not send it to your email address, because my email address is blocked in some way =(

[justinretzolk]

Hmm that's strange @EugenKon -- I'm not sure what email you're trying to send to, but usually folks will encrypt the logs and then upload in a comment here. That said, I reviewed the initial ask again and noticed something that I'd not caught last time.

The instance is not updated in place, because its type was changed and reboot will occur.

In this case, what Terraform means by "update in place" is that Terraform itself will not be destroying and then re-creating the resource as it does for resources that are not able to be updated (note that the instance ID stays the same, for example). That AWS triggers the instance to reboot after the update isn't something that Terraform has any insight into. We looked into this on a previous issue, where a couple of workaround options were presented as well.

Since this is currently behaving as expected, I'm going to close this issue. If you experience unexpected behavior in the future, please do continue to let us know!

[github-actions[bot]]

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.