A way to hide certain expected changes from the "refresh" report ("Objects have changed outside of Terraform")

petkaantonov commented 3 years ago

After upgrading to 0.15.4 terraform reports changes that are ignored. It is exactly like commented here: https://github.com/hashicorp/terraform/issues/28776#issuecomment-846547594

Terraform Version

Terraform v0.15.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.42.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Terraform Configuration Files


resource "aws_batch_compute_environment" "batch_compute" {
  lifecycle {
    ignore_changes = [compute_resources[0].desired_vcpus]
  }

...

  compute_resources {
...
  }
}

resource "aws_db_instance" "postgres_db" {
  ...

  lifecycle {
    prevent_destroy = true
    ignore_changes = [latest_restorable_time]
  }
}

Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.db.aws_db_instance.postgres_db has been changed
  ~ resource "aws_db_instance" "postgres_db" {
        id                                    = "db"
      ~ latest_restorable_time                = "2021-05-25T10:24:14Z" -> "2021-05-25T10:29:14Z"
        name                                  = "db"
        tags                                  = {
            "Name" = "DatabaseServer"
        }
        # (47 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # module.batch_processor_dot_backend.aws_batch_compute_environment.batch_compute has been changed
  ~ resource "aws_batch_compute_environment" "batch_compute" {
        id                       = "batch-compute"
        tags                     = {}
        # (9 unchanged attributes hidden)

      ~ compute_resources {
          ~ desired_vcpus      = 0 -> 2
            tags               = {}
            # (9 unchanged attributes hidden)
        }
    }

Expected Behavior

No changes should be reported because they are listed in ignored changes.

Actual Behavior

Changes are reported.

Steps to Reproduce

Change any resource outside of terraform and see that terraform apply reports changed even when they should be ignored.

Additional Context

References

apparentlymart commented 3 years ago

Hi @petkaantonov! Thanks for opening this feature request.

The current behavior is reflecting the long-standing behavior that Terraform does still detect and incorporate remote objects into the state, but then while producing a plan it ignores differences between the configuration and the state. The difference reported in your example is a difference between the prior state and the current remote object, and ignore_changes has never affected that situation but that fact was less visible before because Terraform just silently updated the state rather than reporting it.

While it might seem immaterial whether Terraform updates the state or not here, it can result in a change in behavior of your configuration if any other expressions in the module refer to that value. To be specific, if you had any reference to aws_batch_compute_environment.batch_compute.compute_resources[0].desired_vcpus elsewhere in your module then they would return 2 rather than 0 after detecting this change, and so that is what Terraform is reporting here, in case that ends up being useful context for understanding which actions Terraform proposes (or doesn't propose) in the plan.

I assume that in your case this doesn't really matter much, because you don't have any references to aws_batch_compute_environment.batch_compute.compute_resources[0].desired_vcpus elsewhere in your module and so you don't actually care what's reported in the state. One potential improvement we could consider for this mechanism is for Terraform to try to only report changes to resources that have other expressions referring to them, since changes to a value that nothing refers to can't possibly affect the behavior of the configuration.

However, such a rule is easier to say than to implement because what we've done here is just expose in the UI some long-standing Terraform behavior that was previously invisible, and so changing that behavior at this late state will likely require a lot of research to make sure that the changes don't break use-cases we're not currently aware of. The current output is an honest and correct account of Terraform's behavior, and so I think we need to consider here whether the right thing to do is change Terraform's behavior (which, for something this fundamental, would be challenging to do at this late stage in Terraform's life) or to change the UI to fudge the details a little so that it leaves hidden some details that surface inconvenient truths that don't actually affect configuration behavior.

We won't be able to make any significant changes in this regard in the near future, because the scope of this project was just to be more explicit about what Terraform was already doing rather than to change how Terraform behaves, but we'll use this issue to represent the need and consider what we might change in future releases.

Thanks again!

petkaantonov commented 3 years ago

I see. In that case, would it be possible to add a separate way to ignore these? e.g. ignore_drifts or something like that? Wouldn't that be easy to implement?

apparentlymart commented 3 years ago

The refresh operation doesn't currently really consider the resource configuration at all -- it's job is to synchronize the existing state with remote objects -- so making it react to anything new in the resource configuration is not a trivial design decision.

We also know from our experiences with ignore_changes that this idea of partially ignoring changes to parts of objects is generally not as simple as it first appears: there are lots of resource types where two or more attribute values are connected to each other in some way, and so ignoring one without ignoring the other can make the result inconsistent. For current ignore_changes that sort of inconsistency isn't a massive problem aside from being a bit confusing, because during planning we're just proposing some changes and not actually changing anything, but doing that during refresh might well lead to a situation where the object in the state is no longer a valid input to the provider that's managing it, which would likely lead to errors or crashes where a provider has to deal with broken input in a location where it never did before.

petkaantonov commented 3 years ago

Could the refresh operation be made configurable in some other way? like an .ignorefile There is probably going to be 3rd party solution if it's not added anyway.

Just some way to ignore these is important, otherwise the output is not as useful because it becomes just manually ignored noise and people become conditioned to ignore it regardless of what it says.

apparentlymart commented 3 years ago

I don't currently have any ready-to-implement designs to address this feedback. We'll need to do some more research and design work in order to address it. Until then, indeed if your system routinely makes changes to objects outside of Terraform as part of its regular function then the feature in its current state will not be so useful for you,

Hopefully it can still answer the sorts of questions it is aiming to solve, though: if Terraform proposes making a change that doesn't seem justified by a corresponding change in configuration, you can refer to the note about detected changes to see if any of the changes it detected are the explanation. If the actions Terraform is proposing match what you were expecting anyway, then the changes detected are just some extra information that you can safely ignore.

rhgreene commented 3 years ago

In situations where the user is forced to add an ignore - to stop Terraform from chasing it's own tail - ie with the use of an aws_autoscaling_attachment (where it is documented that you must use an ignore in corresponding autoscaling groups https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_attachment), I think this behaviour constitutes a bug. The drift output states that "Objects have changed outside of Terraform" when in reality the changes are made within terraform and have had to be ignored for reasons that might be justified as a bug in their own right.

Choosing to output drift before taking into account ignores thus creates potentially significant noise when using Terrafom within it's documented bounds.

koalalorenzo commented 3 years ago

I can relate to this issue a lot, our daily work now has a new pain: filtering out the noise of ignored changes. "muting" the output would help A LOT! :)

anyway, thank you guys for the good work, keep up with it! 👍

bschaeffer commented 3 years ago

This feels like a pretty invasive change. I second what @koalalorenzo said.

The main problem with this change is that we lean on our large team of engineers to inspect their own changes carefully on a per-team basis and apply those changes without constant infra approval. Even when diffs are small, though, we can sometimes misunderstand the plan which can lead to some unfortunate consequences. Not everyone understands the inner workings of terraform, or even the best practices, so there is always going to be mistakes. This isn't terraform's fault, we all need to learn the tool a little bit better.

Now, though, we have to sift through an entire block of output and tell our team to just ignore it up to a point. Its just going to lead to less clear plan output that people put less effort into reviewing. I think its going to erode trust, too, even though I suspect the opposite is the goal.

For instance, take a google_container_node_pool resource. If its auto-scaling, every single time you apply the project you get a new node_count attribute diff that is wholly un-actionable. If somebody sees that every single time, but I tell them to ignore it because it should absolutely be ignored, whose to say when they get a -/+ change on that resources (maybe by accident or lack of understanding) they aren't going to assume its also ignorable.

TBH, I have ignored many attributes in terraform because we may change them outside of terraform and have come to rely on the fact that it really doesn't and shouldn't care about the changes to attributes you don't specify. IDK. Ranting a bit.

I have to be totally honest I see no point in opting everyone into this for every resource automatically. I don't understand the problem that its solving and unless there is a way for it to be completely filtered out I would love to see this change reverted in the next release.

(I also don't think the solution is ignore_attributes.)

sofam commented 3 years ago

I am very surprised about this change and it adds a ton of noise to the already not exactly clear and concise plans. The extra verbosity adds a lot of cognitive load which risks at best to make people glance over the plans and at worst make them ignore them completely.

I really, really don't understand the reasoning here.

Again, other than that, thank you for a great product over the years!

bschaeffer commented 3 years ago

I can get a little worked up and forget that a lot of people put in a lot of work to get terraform where it is today. I don't want to belittle that. People like @apparentlymart (who helped us in the forums when we were starting to write our own internal terraform provider plugins) are awesome. Terraform is awesome. Obviously my job is infinitely easier because of it. ❤️

As for this feature, it was in the CHANGELOG and I did glance over it, so its on me that I upgraded to 0.15.4. I just wanted to register my feeedback that I really wish this was separate from the plan, or that it was opt-in when running plans/applies, or that it was even opt in at a module or resource level (a lifecycle { show_drift = true } attribute?).

geoff-reason commented 3 years ago

Just adding another resource type that now always appears in my terraform output - aws_efs_file_system.

(from https://github.com/hashicorp/terraform/issues/28845 which I will mark as a duplicate). EFS file systems are supposed to grow and change, as many other types of resource they are explicitly designed that way.

With no way to ignore, the plan/apply output is now so noisy I've gone back to terraform 0.15.3 as wading through tens of modified resources on every single plan/apply just to catch the one that might be important is not a workable pattern for me.

lra commented 3 years ago

Since terraform 0.15.4, I get confused messages from my team every single day. I can't imagine the impact this change had globally.

Please add an option to disable displaying those by default and go back to the previous behavior, with maybe an explicit way to get more details about the changes only when needed (e.g a debug mode).

This is really confusing right now, and someone needs to understand the internals of terraform to appreciate the difference between a drift and remote changes.

joeaawad commented 3 years ago

One thing that surprised me a lot about this change is that the Note: Objects have changed outside of Terraform still shows up if you do terraform plan -input=false -output=tfplan >/dev/null && terraform show tfplan. I had hoped that this note would be output as part of the Refreshing state ... messages and as such would be hidden, but it turns out that the note is part of the saved plan file. If this was changed so that the note is output with the Refreshing state ... messages instead of being saved inside of the plan, that would make a big difference for the ability of users to silence that note and anyone who was already silencing the refreshing state messages would automatically get this new note silenced as well.

As near as I can tell, the only way to currently silence the warning is to run terraform refresh >/dev/null && terraform plan -input=false -refresh=false, which has several downsides that @jbardin and @apparentlymart pointed out in https://github.com/hashicorp/terraform/issues/27214.

chloeruka commented 3 years ago

This is really confusing right now, and someone needs to understand the internals of terraform to appreciate the difference between a drift and remote changes.

Agreed, this bit us earlier. Another example is that AWS autoscaling groups will always have "external" changes to attributes such as desired_count from normal scaling behaviour so the refresh-only state plan will show drift. I was going to report this to the AWS provider, but it sounds like they might not yet have a way to fix this. If there are underlying architecture changes that need to be carefully considered, I would certainly prefer to suppress these drift notifications until they can be made more useful.

As the below is an ignored attribute, the resource is not affected by Terraform and the message amounts to noise: Example:

  # aws_autoscaling_group.example-scaling-group has been changed
  ~ resource "aws_autoscaling_group" "example-scaling-group" {
      ~ desired_capacity          = 50 -> 36
        id                        = "example-scaling-group"
        name                      = "example-scaling-group"
        # (20 unchanged attributes hidden)

A person reviewing this plan has to already know the attribute is ignored on that resource or read "below the line" in order to confirm the resource is not affected. Another way of looking at this is as a formatting issue, since the change notification looks very similar to the actual plan, and it took me and the rest of my team a while to realise it's a separate section.

ryanking commented 3 years ago

Another example - okta_app_oauth supports users which can also be configured by a separate resource. If you use the separate resource or some out-of-band way to manage the users, you will always see a change here.

loozhengyuan commented 3 years ago

I think it is fine if Terraform now reports drifts explicitly but it will be great if we could have a way to hide it because we might not be interested in changes happening outside of Terraform in certain use cases.

One way we can implement this is to add a --ignore-drifts flag to the relevant terraform xxx subcommands. This is the approach I'm most in favour with because it allows this to be decided at runtime and to be varied flexibly across different use cases (i.e. running locally vs running in CI/CD platforms, etc.). It also provides users a simple way to revert to the old representation of the output.

$ terraform plan --ignore-drifts
$ terraform apply --ignore-drifts

In addition, we could also implement this at a resource/module level using the ignore_drifts field. This may be helpful for some use cases where we might be interested drifts in general but not all fields.

resource "aws_instance" "example" {
  # ...

  lifecycle {
    ignore_drifts = [
      etag, # Ignore drifts for specific attributes
    ]
  }
}

resource "aws_instance" "example" {
  # ...

  lifecycle {
    ignore_drifts = all # Ignore drifts for all attributes
  }
}

cbfield commented 3 years ago

+1 to this issue.

Part of our workflow involves creating EMR clusters in AWS, including running a few steps to modify some software configurations. After the cluster is created, our data scientists run hundreds more steps throughout the lifetime of the cluster. Now every single step in every cluster shows up in every Terraform plan, despite the lifecycle ignore.

We use Atlantis to execute our Terraform configurations, but cannot do so now thanks to this. The plans produced are too long to appear in Gitlab comments, and as such we cannot see any of the plans produced by Atlantis.

kpkrishnamoorthy commented 3 years ago

To add to the discussion here, I'm also seeing that (for example) the Principals list in a role always shows up as "has been changed outside of Terraform", even if I attempt to manually update the principals list via the AWS web console.

          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ Service = [
                              + "lambda.amazonaws.com",
                              + "ec2.amazonaws.com",
                                "states.us-east-1.amazonaws.com",
                                # (1 unchanged element hidden)
                                "cloudwatch.amazonaws.com",
                              - "ec2.amazonaws.com",
                              - "lambda.amazonaws.com",
                            ]
                        }
                        # (3 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )

Each time I edit it via the web console to reflect what Terraform thinks it ought to be, AWS saves the list in a random order (not alphabetical), so Terraform always thinks that there's been a change. Even when I force an apply, or taint and import the resource, this warning persists.

Outside of this warning, the plan results in the message "No changes. Your infrastructure matches the configuration.".

This, to me, doesn't make sense because if Terraform thinks that there is drift, an apply should be able to rectify that drift. This does not appear to be the case, which makes the whole drift warning kind of pointless and noisy.

nfrappart commented 3 years ago

Hi guys, as everyone else, this change is quite impacting for us. On azure, it appears that every apply resulting in changes in other resources will be seen as a drift. So far, I encountered to cases:

creating a new subnet using the azurerm_subnet resource: The plan/apply goes well, but on the next plan, even if there are no changes, Terraform informs me that my VNet has changed, because it wants to reflect the newly added subnet in the VNet attributes. But this change was actually made by Terraform, in the previous run.
creating a virtual machine, as the MAC Address and VM id are known after deployment, Terraform detects changes on the NIC on the next plan, because now, the NIC has indeed a MAC and a VM id.

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.Azadds_mgmt_VM.azurerm_network_interface.TerraVM-nic0 has been changed
  ~ resource "azurerm_network_interface" "TerraVM-nic0" {
        id                            = "/subscriptions/<subscription_id>/resourceGroups/myRG/providers/Microsoft.Network/networkInterfaces/myvm-nic0"
      + mac_address                   = "00-xx-xx-xx-xx-xx"
        name                          = "myvm-nic0"
      + tags                          = {}
      + virtual_machine_id            = "/subscriptions/<subscription_id>/resourceGroups/myRG/providers/Microsoft.Compute/virtualMachines/myvm"
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

I think we can all agree the Terraform team is doing a tremendous job at bringing the product where it is today, but this particular change although attempting to bring more visibility, brings, in my opinion, a lot of confusion.

I think I will roll back to 0.15.3 until some improvements are made about this feature.

dimisjim commented 3 years ago

hey @kpkrishnamoorthy

I have submitted an issue about this in the provider tracker https://github.com/hashicorp/terraform-provider-aws/issues/19727

But it would be definitely cool if core terraform allowed for an opt-in option to hide the output of this new feature until provider devs fix it on their side

acrewdson commented 3 years ago

I think we can all agree the Terraform team is doing a tremendous job at bringing the product where it is today, but this particular change although attempting to bring more visibility, brings, in my opinion, a lot of confusion.

Just want to second this.

koalalorenzo commented 3 years ago

Do we have anybody working on it? If so, do we have an ETA for this? It has been an annoying issue that confuses a lot of people. Our Atlantis comments on PRs are way too big for no reason, and it is super frustrating!

🙏 Please let us know how to help somehow, and pressure to get this shipped in the next minor/patch release!!! 🙏

jbscare commented 3 years ago

We're affected by this too, and it seems clearly like a bug. We've got things in our config where we've told Terraform to ignore changes to those things; now it ignores the changes in that it won't try to control those things, but it no longer ignores them when reporting on them.

If you've told Terraform to ignore changes to something, it should ignore those changes silently.

If that isn't universally agreeable, perhaps a general option like ignore_changes_silently or something?

fbreckle commented 3 years ago

I actually propose to make this entire drift detection feature OPT IN instead of opt out for the time being. As this thread shows, its causing lots of issues and confusion for many people, with different kinds of usecases.

I DO see the benefit of the feature in general, but putting this change into a release sure backfired. I'd wager if they just remove it in 1.0.1 not many would actually miss it.

JordanP commented 3 years ago

@fbreckle it seems you are making assumptions about how Terraform is used everywhere, and what the community as a whole thinks about this feature.

I myself find this feature useful, I sure want to know when my Infra has changed under the hood and what exactly changed outside of Terraform. We depend heavily on Terraform and this new feature gives us visibility and increases the trust we have in this tool.

It has some downsides (like for us we are using Kubernetes with node autoscaler and node auto upgrade so TF reports that the node count and the node version has changed quite often) but I am pretty sure that with the required changes in the providers/resources, we can find an elegant solution without reverting the whole thing.

Thanks Terraform for being opinionated !

take-five commented 3 years ago

I think it's clear that opinions are split, and while a part of the community is happy about this change, the other part doesn't have a way to opt out.

For us it's a blocker and prevents us from upgrading to 0.15 (1.0).

jbscare commented 3 years ago

@JordanP : We definitely want to know about drift in the resources that we've told Terraform to manage. What we don't want to know about is drift in resources that we've told Terraform to ignore.

Do you -- does anyone -- have a use case for "our config says that Terraform should ignore this resource, but we want to hear about changes to it anyway"?

There was this mentioned early on:

While it might seem immaterial whether Terraform updates the state or not here, it can result in a change in behavior of your configuration if any other expressions in the module refer to that value. To be specific, if you had any reference to aws_batch_compute_environment.batch_compute.compute_resources[0].desired_vcpus elsewhere in your module then they would return 2 rather than 0 after detecting this change, and so that is what Terraform is reporting here, in case that ends up being useful context for understanding which actions Terraform proposes (or doesn't propose) in the plan.

(https://github.com/hashicorp/terraform/issues/28803#issuecomment-848049967)

That seems like a weird situation, where you'd have references to values that you've told Terraform to ignore the value of.

dimisjim commented 3 years ago

@jbscare

That seems like a weird situation, where you'd have references to values that you've told Terraform to ignore the value of.

This should be processed as a plan-time explicit warning, in my opinion.

The only reason I can think of, when someone would need to explicitly use a reference to a resource attribute that its changes are being ignored explicitly, is at initial resources creation.

But even that is the reason, then it wouldn't make sense for terraform to continuously notify users about "changes being made outside of terraform" in this new section. A warning message would be adequate and less prone to misinterpretation.

simlu commented 3 years ago

Quick chime in: We use TF to set up and manage our AWS accounts and detect any drift. Our configs and accounts have worked for years without any major changes. Now we are suddenly getting warnings without any details (just empty) and I still have no idea why.

I read a bit, but still don't really understand what is happening. Will have to spend more time, but ideally things would be a bit more obvious (why are we getting a warning, what are our next steps, etc)

Love the tool btw and this is not a rant, just an FYI 😉

ryanking commented 3 years ago

I am pretty sure that with the required changes in the providers/resources, we can find an elegant solution without reverting the whole thing.

I don't think this is true @JordanP. The way this is implemented, there is now way for providers to work around this.

dimisjim commented 3 years ago

@ryanking

hmm this is quite the antithesis of what @jbardin has claimed recently: https://github.com/hashicorp/terraform/issues/28911#issuecomment-857647716

If there is really no way for providers to work with this tf-core feature in mind, then we should definitely have a disable flag asap

cfbao commented 3 years ago

@ryanking

hmm this is quite the antithesis of what @jbardin has claimed recently: #28911 (comment)

If there is really no way for providers to work with this tf-core feature in mind, then we should definitely have a disable flag asap

that comment is mostly about how list diffs are handled. if provider mess up the ordering, then yeah it's on the provider. but this issue is about not showing ignored changes in drift detection, which is mostly an orthogonal problem, and providers have no control over this.

robomon1 commented 3 years ago

This falls under the "too much information" category to me. This is debug output at best. Set the TF_LOG=DEBUG and then you can get this output. Here is a good example of why it doesn't work for me:

  ~ resource "aws_security_group" "chat" {
        id                     = "sg-0f46dd9f058d12345"
        name                   = "prod-chat-8"
        tags                   = {
            "Environment" = "prod"
            "Name"        = "prod-chat-8"
            "Terraform"   = "yes"
        }
      + tags_all               = {
          + "Environment" = "prod"
          + "Name"        = "prod-chat-8"
          + "Terraform"   = "yes"
        }
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

The tags_all is an "attribute". It is not an "argument". It is computed. I won't be setting it - ever. Even if I apply this if I add/change/delete a tag it will also change the tags_all attribute and I will have to apply it again to get rid of this message. And there are many, many more of these examples.

gerardgorrion commented 3 years ago

I have some similar error:

panic: Error reading level state: strconv.ParseInt: parsing "84850493440": value out of range

This is the attribute output by terrafom of efs system. There're some wy to evict this error without delete efs system from tfstate?

Thanks

nodesocket commented 3 years ago

I also just ran into this upgrading from 0.14.11. My feedback would be NOT a command line flag I have to provide to ignore changes outside of Terraform but a new attribute that behaves like lifecycle { ignore_changes = [] }.

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.fusionauth.aws_db_instance.rds has been changed
  ~ resource "aws_db_instance" "rds" {
        id                                    = "fusionauth"
      ~ latest_restorable_time                = "2021-07-06T16:42:02Z" -> "2021-07-06T16:57:02Z"
        name                                  = "fusionauth"
        tags                                  = {
            "Env"  = "stage"
            "Name" = "fusionauth"
        }
        # (52 unchanged attributes hidden)
    }
  # module.postgres.aws_db_instance.rds has been changed
  ~ resource "aws_db_instance" "rds" {
        id                                    = "postgres"
      ~ latest_restorable_time                = "2021-07-06T16:39:08Z" -> "2021-07-06T16:59:09Z"
        name                                  = "postgres"
        tags                                  = {
            "Env"  = "stage"
            "Name" = "postgres"
        }
        # (52 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions
to undo or respond to these changes.

tullydwyer commented 3 years ago

We need a way to output only the actual changes Terraform intends to perform. Currently 100% of the output from "Objects have changed outside of Terraform" is noise for us (unavoidable and intended changes, or ignored fields).

Does anyone know how to output only Terraforms planned changes?

I plan on testing running a terraform apply -refresh-only -auto-approve before each plan to reduce noise, however changes can still happen between.

Edit: don't us the command above not useable with certain changes: https://github.com/hashicorp/terraform/issues/28939

lra commented 3 years ago

Does anyone know how to output only Terraforms planned changes?

No, there is no way, hence this GitHub issue. A flag or an env var would be welcome.

tullydwyer commented 3 years ago

Update: we have tested running terraform apply -refresh-only -auto-approve before each plan, it works for us.

We were currently scrolling through ~390 lines of detected change output that weren't actual differences down to 0.

Update 2: The above command is a hack which has the potential to cause issues. See Indigenuity's comment below.

Only run the above if you have the ability to recover your statefile from any issues. Only run if you have state file versioning.

Update 3: don't us the command above not useable with certain changes: https://github.com/hashicorp/terraform/issues/28939

dmaier-FN commented 3 years ago

It strikes me as a step backwards that we now have to run an -auto-approve job just so the output of plan is intelligible. @robomon1 hit the nail on the head: it may not be debug output by strictest definition but it's certainly debug output in utility.

While there have been past cases where I would have loved this view into the internals of terraform refresh I don't need it 99% of the time and most of our users don't need it ever. Especially when some changes are so laughably trivial:

  ~ resource "google_bigquery_table" "table" {
        id                  = <sensitive>
      ~ last_modified_time  = 1625689042601 -> 1625696261956
      ~ num_bytes           = 3889196 -> 3943304
      ~ num_rows            = 14390 -> 14590
      # (13 unchanged attributes hidden)

So, like some others, our solution has been to downgrade to 15.3. I echo those others as well when I say that Terraform has changed the way I work. Thanks for your tireless work on it.

koalalorenzo commented 3 years ago

Is anybody working on this? 😅

EDIT: just noticed that this got a "enhancement" tag, while IMHO from a UX point of view this is a MAJOR f*ck up and not an enhancement! 😓 😢 We are still suffering from it and it is getting super frustrating. I wish somebody would consider this for the next release or update us on the status. What can we do?

nodesocket commented 3 years ago

Also, looking for a fix to hide this related to RDS instances and ~ latest_restorable_time = "2021-07-20T21:21:23Z" -> "2021-07-20T21:26:23Z"

ghost commented 3 years ago

Update: we have tested running terraform apply -refresh-only -auto-approve before each plan, it works for us.

We were currently scrolling through ~390 lines of detected change output that weren't actual differences down to 0.

It is workaround for some cases. I have some AWS IAM resources, changes still appear even after I ran this command.

I really don't like this in update. They should make it print only when user want it (via flag), let terraform show plan in way people expect. Nice feature but bad implement

ma44in commented 3 years ago

We also use atlantis and our mouse wheel is burning now, due to scrolling down in merge request to get to the interesting terraform part. :smile: We use terraform with VMware and face this issue with a custom attribute, which get updated daily by a backup software.

~ custom_attributes = {
          ~ "999" = "Backup Server=..., EndTime=2021-07-20T19:44:00Z"
        }

Like others said, it would be really great to have some sort of switch to hide "ignore_changes-changes". Nevertheless terraform is such a great piece of software. Thank you!!! :slightly_smiling_face:

Tazminia commented 3 years ago

Hi,

Update: we have tested running terraform apply -refresh-only -auto-approve before each plan, it works for us.

We were currently scrolling through ~390 lines of detected change output that weren't actual differences down to 0.

If you refer to the terraform documentation, you can see that this command is deprecated.

The old output was more readable before introducing this "feature". Would have been better if we could opt-in (or not) for the new behaviour.

Indigenuity commented 3 years ago

Wait, wait, are people actually using -auto-approve as part of their workarounds for this? Seems rather dangerous! Even the documentation linked by @Tazminia warns explicitly about one risk:

Automatically applying the effect of a refresh is risky, because if you have misconfigured credentials for one or more providers then the provider may be misled into thinking that all of the managed objects have been deleted, and thus remove all of the tracked objects without any confirmation prompt.

And furthermore, these state updates aren't always clean. I had an aws_security_group resource with 30 aws_security_group_rule resources, and this refresh seemed to think that the rules had been changed externally to all be in-line rules. So suddenly terraform is thinking it needs to create 30 more resources again, since all the rules were moved to be in-line. Using -auto-approve in scenarios like this is a great way to blow up your infra!

If this feature is pushing people to risky workarounds, I'd suggest that it's more than just a nuisance and should be treated as a serious bug. While the output of this feature doesn't suggest -auto-approve, it does imply that -refresh-only is always harmless, which is untrue.

ghost commented 3 years ago

If you refer to the terraform documentation, you can see that this command is deprecated.

@Tazminia The command terraform refresh is deprecated, but they clearly state that it is an alias to terraform apply -refresh-only -auto-approve, which they WANT people to know about and understand because they introduced the -refresh-only flag in 0.15.4; these new flags are important to interacting with this new output.

It's definitely been a bit of a transition to 0.15.4+, and we would also like the option to show/hide this output, but overall so happy for the Terraform team to hit 1.0+! Thanks for all your hard work!

dmitrybelyakov commented 3 years ago

This is extremely annoying and frustrating for the team.

tullydwyer commented 3 years ago

Wait, wait, are people actually using -auto-approve as part of their workarounds for this? Seems rather dangerous! Even the documentation linked by @Tazminia warns explicitly about one risk:

Automatically applying the effect of a refresh is risky, because if you have misconfigured credentials for one or more providers then the provider may be misled into thinking that all of the managed objects have been deleted, and thus remove all of the tracked objects without any confirmation prompt.

And furthermore, these state updates aren't always clean. I had an aws_security_group resource with 30 aws_security_group_rule resources, and this refresh seemed to think that the rules had been changed externally to all be in-line rules. So suddenly terraform is thinking it needs to create 30 more resources again, since all the rules were moved to be in-line. Using -auto-approve in scenarios like this is a great way to blow up your infra!

If this feature is pushing people to risky workarounds, I'd suggest that it's more than just a nuisance and should be treated as a serious bug. While the output of this feature doesn't suggest -auto-approve, it does imply that -refresh-only is always harmless, which is untrue.

This is true. It is a hack which has the potential to cause issues. I will update my comment.

The only other alternative we can see is getting the team to scroll through and inspect hundreds (soon to be thousands) of lines of expected changes/changes that aren't even changes every single plan run (just for 1 pipeline). A lot of our resources are affected with this issue.

Update: don't us the command above not useable with certain changes: #28939

elkbullwinkle commented 3 years ago

Wouldn't having a flag on apply to show\hide these outside changes be a relatively simple solution?

Having all this extra output does make it harder to process what would actually change.

razor-x commented 3 years ago

A few thoughts from the POV of a Terraform Cloud user.

A flag would be nice along with a corresponding option for the workspace.
This output could be separated from the normal plan output in the Terraform Cloud UI.
Taking it one step further, it would be amazing if the planned changes could be put in their own log window. The plan output is generally very noisy and makes it much harder to review the plan than it could be:
- First it prints the init steps.
- Then it prints all the resources from the state refresh.
- Then it prints this new report
- Then, finally, if there are no errors, it prints the actual thing we want to look at and always need to review very carefully: the planned changes.

hashicorp / terraform