Output canonical representation of configuration

[sparr]

Current Terraform Version

Terraform v0.12.0-dev

Use-cases

Enable linters, configuration enforcement tools, and configuration manipulation scripts to operate with only an HCL/JSON parser, rather than needing to reimplement all of the logic involved in module inputs and outputs, variable interpolation, etc.

Attempted Solutions

I have tried too many combinations to count, of ways to try to combine terraform plan and state outputs, with hcl and json and plan output parsers and manipulators.

Proposal

A new terraform tool that loads a configuration and then writes out HCL and/or JSON in a canonical format, with all of the necessary modules included and interpolation done.

For example, consider the following two files:

root/main.tf:

module "foo" {
    source = "../foo"
    bar = "123"
}

foo/main.tf:

variable "bar" {}
resource "sometype" "somename" {
    attr = "${var.bar}"
}

This new tool would render a single output file as follows:

resource "sometype" "somename" {
    attr = "123"
}

References

[apparentlymart]

Hi @sparr! This is an interesting idea; thanks for sharing it.

I think the closest thing we have to this in Terraform 0.12 is the terraform show -json command, which can take a saved plan file and render it out in a JSON format, including all of the planned new resource arguments. The plan is necessarily incomplete because certain values cannot be known until after the change is applied, but it includes the subset of the values that are known at plan time.

Converting that into Terraform language syntax instead is an interesting idea. It's not really possible to do that directly with the language as it exists today, because the transform you gave in your example would change the meaning of the configuration... the resource module.foo.sometype.somename would become just sometype.somename in the root module, which would be understood by Terraform as deleting the child module and would be problematic if two different modules had the same type/name combination, because they would collide. We'd therefore need to generate something that isn't really Terraform language at all, but some other Terraform-like HCL format; at that point, I'm not sure that would really be any better than just generating some simpler JSON output.

I'd be curious to hear more about what you tried already with plan files, because in principle saved plan files should contain the maximum possible information by virtue of being an on-disk serialization of Terraform's in-memory model of plans. If there was something you weren't able to do by reading the plan file then it's likely that Terraform wouldn't be able to generate it in HCL format either without some changes to its internal models, at which point the plan file would then likely include the same information anyway.

[sparr]

There is no way to produce a resource named module.foo.sometype.somename in a single file? That seems like a weakness of the format that deserves separate attention.

The problem with a plan file is that it represents changes to be made, while the configuration represents a target state, and the target state is what I am concerned with.

For very specific context, my use case here is to use https://github.com/stelligent/config-lint/ which currently only handles one .tf file at a time and thus doesn't work for files that refer to module inputs or resources from other files. My other angle of attack is to try to extract all of the configuration and state loading logic out of terraform itself, or perhaps from another tool if I can find one that does it all, and patch it into config-lint.

I did explore some paths in trying to combine a state file with a plan file to produce a description of a target state, including converting each to other formats using third party tools, but was not successful.

[apparentlymart]

The plan output produced by terraform show -json describes the plan in terms of old and new objects, so the new objects taken together are the same as the target state.

Since you may have previously been working with Terraform 0.11 plans, note that the plan model is considerably different in Terraform 0.12 and is now represented by a prior state and planned new state pair, not individual changes per attribute. When Terraform 0.12 renders a plan for human approval, it performs the attribute-level diff just in time for rendering, but the internal model is always entire objects before and after.

The JSON representation of plans for Terraform 0.12 is described in Plan Representation, and I think what you are looking for can be found in the planned_values property of that output. There is some additional detail available in resource_changes, and if you look at the after property of each change you will see the same whole new objects present in planned_values.

The Terraform Enterprise policy enforcement feature is implemented in terms of this same output, and that is essentially the same use-case you are describing here.

(Terraform 0.12.0 is currently in its release candidate phase, so there isn't a stable release yet but we expect one in a few weeks unless the release candidate identifies a significant blocker.)

[sparr]

Is there a way to turn that plan output back into a configuration, such that that configuration could be run against, say, a different AWS account or production environment?

[apparentlymart]

The JSON plan output alone doesn't have enough information to do that, but the binary plan file format (as of v0.12) has the whole configuration embedded inside it, so in principle you can extract the exact source files that created it and reconstruct the original source.