concise mode for `terraform plan`

[spkane]

Feature request

Add a concise mode to terraform plan.

Sometimes simple things like updating a large cloud-init template, can make the output of terraform plan hard to read through.

It would be very nice to have something like a --concise switch that would only show the change header lines and the final summary like:

-/+ module.ec2_master.template_file.cloudinit_master.3
~ module.ec2_agent.aws_elb.ext_http_proxy_lb

Plan: 1 to add, 1 to change, 1 to destroy.

[mitchellh]

Tagged as enhancement. For future: I'd say the flag should be -short.

[spkane]

Basically something close to this output:

tf plan -no-color | grep -E '^[[:punct:]]|Plan'

[stephenchu]

I know that PR #10649 is already out there, but I would also love to eliminate the following general explanation/description text from plan.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

I sense that the functionality I'm after should not be part of -short tho, so maybe -quiet.

[apparentlymart]

Given that there are lots of parts of the terraform plan output that people are likely to want to opt out of separately, perhaps it behooves us to use more descriptive names, at the expense of them being longer, so that their meaning is clearer (vs. something non-obvious like terraform plan -short -quiet):

• --show-attrs=false to hide the resource attributes from the diff, getting the behavior @spkane wanted
• --show-usage-help=false to hide the paragraphs of text intended to help explain Terraform's workflow, as @stephenchu wants

Both would default to true, preserving today's behavior.

[josephholsten]

Well, shoot. I tried rebasing #10649 today and command/plan.go has been significantly changed to use backend.

Currently, the actual checks for the show flag have to be made in backend/local/backend_plan.go. Unfortunately, it's not entirely clear how to get the show (or show-attrs & show-usage-help) from command/plan.go:(*PlanCommand).Run() over to backend/local/backend_plan.go:(*Local).opPlan() to make those checks.

Some nearby calls currently:

• b.CLI.Output(): github.com/mitchellh/cli.Ui.Output(). This isn't that common of a problem to solve at that level.
• b.Colorize(): colorstring.Colorize I guess we could add a filtering function, but then every print will require many filter passes. It already hurts now: b.CLI.Output(b.Colorize().Color(fmt.Sprintf(. And we don't need to track state like a colorizer.
• b.ContextOpts: *terraform.ContextOpts`: haha, no.
• ctx: context.Context always has Value(). ugh.
• op: backend.Operation: we could add ShowAttrs bool and ShowUsageHelp bool, I guess?

It's too bad we don't have a JSON format for plans, because then we could just throw jq at it.

[josephholsten]

oh, and to pitch into the flag sprawl, another display flag proposal for terraform plan -diff is at https://github.com/hashicorp/terraform/pull/11081 Kinda sounds like --show-usage-help=false.

[apparentlymart]

Hi all,

There have been a few different requests for variants of the plan output recently. I wanted to respond to that, but since there isn't a single central place I guess I will respond here for now:

At the current point in Terraform's development, we're being rather cautious about adding lots of options -- particularly CLI options, but also flags set in configuration -- because they significantly increase the testing surface and make it hard to do the internal refactoring we're still doing a lot of as we figure out the details of Terraform's architecture. We've found in earlier work that the non-default cases inevitably get short shrift when big changes are made, or that testing and supporting them ends up being a significant part of the development process.

Therefore I'm going to ask for some patience on features for UI and workflow customization, such as this request. In the long run we'd like to make Terraform's output flexible to serve lots of different cases, but that'll be easier to accomplish as the core architecture starts to stabilize and we can afford to make more assumptions.

Thanks for the great discussion here. This will be helpful as we review use-cases in future, when we get to the point of refining the UI details in this area.

[de-husk]

Hey, I think this viewpoint makes perfect sense. At this point, the amount of complexity added supporting these types of features would probably outweigh any benefits they bring. I'm going to close my PR.

[cheptsov]

One of the things I struggle with is that when doing terraform plan, it outputs the actual content of s3 objects being modifies. With pretty large binary files being uploaded, the output takes MBs and then is very hard to read and sometimes even not possible at all.

[rverma-nikiai]

@apparentlymart It's being a long time, wondering if we can start accepting pull requests like this. I think we are well aware of the requirement for this kind of pull requests. Terraform being in a position to support automation, this kind of defies the purpose. If we can't understand what terraform is actually doing cause of all such noise, it becomes very challenging to handle huge projects.

Created another issue related to this but we can track all progress here. https://github.com/hashicorp/terraform/issues/20960

[ketzacoatl]

@rverma-nikiai https://github.com/coinbase/terraform-landscape ought to hold you over in the meantime ;)

[rverma-nikiai]

@ketzacoatl I doubt it serves the purpose. I am more interested in hiding too verbose information in terraform output. This looks can make terraform messages more human readable.

[pijemcolu]

I'm a fan of: terraform plan | grep -E '#'

[kunickiaj]

Found this one works well for me: terraform plan | grep -E '^\s*[#~+-]'

[eWilliams35]

I had to add the -no-color flag -- terraform plan -no-color | grep -E '^.*[#~+-] .*'

[chattr]

This is a nice hybrid of above suggestions: terraform plan -no-color | grep -E '(^.*[#~+-] .*|^[[:punct:]]|Plan)'

[Tbohunek]

Given that there are lots of parts of the terraform plan output that people are likely to want to opt out of separately, perhaps it behooves us to use more descriptive names, at the expense of them being longer, so that their meaning is clearer (vs. something non-obvious like terraform plan -short -quiet):

• --show-attrs=false to hide the resource attributes from the diff, getting the behavior @spkane wanted
• --show-usage-help=false to hide the paragraphs of text intended to help explain Terraform's workflow, as @stephenchu wants

Both would default to true, preserving today's behavior.

For every "long" parameter, please also include equivalent shortcut or allow saving this into an env variable. Just imagine typing those two above parameters every time.

[Tbohunek]

When changing plan output, maybe you could also include this? https://github.com/hashicorp/terraform/issues/24233 To have number of replaced resources at the end of the output.

[byarbrough]

Totally hear what @apparentlymart is saying. When it is time to have the discussion, it would be great to have a short option that can be ingested into CI/CD pipelines.

As an example, GitLab 13.0 will embed a "create, update, delete" summary in merge requests !26830. This makes it easy for reviewers to see what changes will be made to infrastructure if they approve the request.

However, in order to make this work, the current solution is to pipe show -json output into a nasty jq query. This would certainly be simpler with some sort of summarized output from show, and I think improving interface with CI/CD pipelines is among one of the best reasons to have this enhancement.

Edit: here is the just published GitLab documentation: https://docs.gitlab.com/ee/user/infrastructure/#output-terraform-plan-information-into-a-merge-request

[sidcarter]

over 2 and a half years since @apparentlymart 's original response. Is this something that is being considered now, that it's 0.13 and this would be really helpful in terms of automation.

[mwarkentin]

@sidcarter good timing! https://github.com/hashicorp/terraform/releases/tag/v0.14.0-alpha20200910

[severity1]

with terraform cloud and its tiny output review box it's a bit too much to scroll 2000 lines of output when inspecting the plan output for a single change that spans multiple resources - like tags, I know it's just tag changes in this example but you still will want to inspect all the changes because there might be other changes outside your actual Pull Request.

EDIT: Nice to know that Concise diff will be default from 0.14.x onwards. Looking forward to this version!

[jbg]

It's not just the readability: with remote execution and large resource content, the vast majority of terraform plan runtime is spent just writing out MiBs of diff. Sometimes it takes so long that the plan finishes quickly, but then execution times out just because of the time taken writing out the diff. The concise diff format doesn't really help if you're adding new complex resources.

A mode where the content of resources are not printed would be super welcome.

[tculp]

I put together this jq query to summarize the object changes:

tfsum() {
  # Slurp to combine each separate json object into a single array, raw output for plain strings
  #
  # First map filters out the planned changes and builds a smaller object from the addr and action
  # Group by the actions, to put them in create, delete, read, and update arrays
  # Second map defines the action as a variable, removes the action from the object, deletes the action
  #   from each object, pulls the addr string out of the objects, and builds a string for each action group.
  #
  # Finally, run the output through another array pull to turn the array of strings into just strings.

  terraform plan -json | \
    jq --slurp --raw-output '
      map(
        select(.type == "planned_change") |
          {
            addr: .change.resource.addr,
            action: .change.action
          }
      ) |
      group_by(.action) |
      map(
        .[0].action as $action |
        del (.[].action) |
        map(.addr) |
        "\($action) (\(. | length)):\n    \(map(.) | join("\n    "))\n"
      ) |
      .[]
    '
}

It produces an output like

create (7):
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_security_group.this[0]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_iam_role.this[0]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_launch_template.this[0]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1-2"].aws_eks_node_group.this[0]

delete (7):
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_security_group.this[0]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_iam_role.this[0]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_launch_template.this[0]
    module.eks.module.eks_managed_node_group["nodegroup-basic-v1"].aws_eks_node_group.this[0]

read (1):
    module.eks.data.tls_certificate.this[0]

update (2):
    module.eks.aws_eks_cluster.this[0]
    module.eks.aws_iam_openid_connect_provider.oidc_provider[0]

[FranAguiar]

Any update on this?? Could be really useful

[apparentlymart]

Hi all,

In the many years since the original discussion on this issue, we implemented a machine-readable version of the plan output to enable anyone to build whatever kind of UI they want to visualize a Terraform plan at a level of detail that suits their needs.

You can make use of that by first saving the plan to a file and then asking Terraform to present the details from that file in the JSON output formats:

terraform plan -out=tfplan
terraform show -json tfplan

The built-in rendering of the plan is the one that we find gives the best compromise right now of being explicit about what will happen for most users, but this integration point means that anyone running Terraform in any sort of automation (whether that be full remote execution or just a wrapping script locally) can make different compromises if they wish.

Given this, I don't expect that we will invest in any more built in mechanisms to change details of this output for the foreseeable future, because that has high costs as I was describing earlier and there doesn't seem to be a clear consensus on any single concise form that would suit everyone. Allowing integration with Terraform to customize the output for yourself is therefore the best current compromise.

[Tbohunek]

I'm surprised with the silent acceptance. Instead of one central effort, at least 122 people now need to invest the "high costs" estimated by @apparentlymart - not efficient to say the least.

It would be nice if the community could create something that Terraform could then easily integrate. I already contributed to Terraform myself, but I'm not at a coding level to write something like this.

What saddens me the most is that this isn't the first time Terraform rejects highly-requested changes, and it aligns well with Terraform becoming a corporate. First :triangular_flag_on_post: is up.

[YevheniiPokhvalii]

Also, it would be convenient to see only created, updated in-place, or deleted resources (and their diffs as well). Like:

terraform plan --short --show=created
terraform plan --short --show=updated
terraform plan --short --show=deleted

[Dmitry1987]

btw is this implemented now in terraform 15?

[Dmitry1987]

Hi all,

In the many years since the original discussion on this issue, we implemented a machine-readable version of the plan output to enable anyone to build whatever kind of UI they want to visualize a Terraform plan at a level of detail that suits their needs.

You can make use of that by first saving the plan to a file and then asking Terraform to present the details from that file in the JSON output formats:

terraform plan -out=tfplan
terraform show -json tfplan

The built-in rendering of the plan is the one that we find gives the best compromise right now of being explicit about what will happen for most users, but this integration point means that anyone running Terraform in any sort of automation (whether that be full remote execution or just a wrapping script locally) can make different compromises if they wish.

Given this, I don't expect that we will invest in any more built in mechanisms to change details of this output for the foreseeable future, because that has high costs as I was describing earlier and there doesn't seem to be a clear consensus on any single concise form that would suit everyone. Allowing integration with Terraform to customize the output for yourself is therefore the best current compromise.

I tried terraform show -json tfplan and it takes a long time, does terraform reach out to aws and scan everything from scratch instead of just parsing the tfplan file at this stage?

[apparentlymart]

The terraform show command doesn't run any Terraform Core code (that is: it doesn't do any graph walks) and so it cannot take actions against remote APIs such as refreshing. Whatever slow action it's taking must therefore presumably be in either decoding the saved plan file or in the step of starting up the providers just to load their schema information, which is needed so that Terraform CLI can correctly interpret the resource-type-specific data stored in the plan.

If you're finding terraform show to be overly slow then I'd suggest opening a new issue for that so we can have a separate discussion about the details of the plan you're working with and hopefully understand whether there's an opportunity for optimization there.

[nitrocode]

Seems promising to add to a pipeline as a temporary solution until this is available.

https://github.com/dineshba/tf-summarize

[containerscrew]

Similar tool like tf-summarize https://github.com/containerscrew/tftools

[Dmitry1987]

are these verbosity levels configurable in the enterprise version of terraform or the open source and the enterprise are the same on client side?

[crw]

@Dmitry1987 Same client, although you might notice that the output in Terraform Cloud is more like what the original poster was requesting. The CLI output is the same, however.

[kodermike]

terraform plan -out=tfplan
terraform show -json tfplan

This work around doesn't work if you have secrets in terraform cloud, so it fails as a workaround to the real issue/challenge

╷
│ Error: Saving a generated plan is currently not supported
│
│ Terraform Cloud does not support saving the generated execution plan locally at this time.
╵

[mbarr-hrp]

It would be very helpful to be able to pipe the json output to stdout instead of having to save the binary plan, and then convert the saved plan file to json.

Yes, it's possible to just a temp file, if you're running entirely locally, but this seems like overkill vs just.. output the json after a plan.

[tgigli]

Currently using tf-summarize which works like a charm, with a tree view which is something that I wanted so hard.