Output by default

[cemenson]

Hi there,

I've been using terraform for managing AWS infrastructure, and have seperated the folder structure as below:

• environments
  • prod
    • vpc
      • main.tf (contains terraform AWS VPC resources)
      • config.tf (contains provider / backend configuration - we use remote state in s3)
    • ec2
      • main.tf
      • config.tf
    • rds
  • dev
• modules

Each subfolder (eg vpc, ec2) has it's own remote state file in S3. We structure it this way to minimise the risk of unintentional changes / state file screw ups.

Unfortunately, it become a pain to cross reference resources with this structure. If, for example, we configure security groups under "vpc", and an instance needs to reference them in "ec2", the only way to manage that is either manually creating a data source in the "ec2" folder, or outputting the security groups and having a terraform_remote_state call.

As we have a lot of security groups, it is unclean & inefficient to have to create / remember outputs for all of them.

I propose having resources output by default, without explicit specification. That way all resources can be cross referenced easier and terraform files are more manageable.

For sensitive outputs, there can be a manual output that specifies the "sensitive" flag.

[cemenson]

As an example, this is what one of the config.tf files would look like:

provider "aws" {
  region                  = "eu-west-1"
  shared_credentials_file = "%USERPROFILE%/.aws/credentials"
}

terraform {
  backend "s3" {
    bucket         = "bucketNameHere"
    dynamodb_table = "terraform-state-lock"
    key            = "ec2/ec2.tfstate"
    region         = "eu-west-1"
    encrypt        = true
  }
}

Note the terraform.key would change based on aws service - e.g "vpc/vpc.tfstate" for VPC resources.

[cemenson]

Alternatively, or perhaps in conjuntion with - it would be extremely helpful if terraform were able to reference resources outside of their own folder. That would probably be even better than the initial feature request as it may help with terraform understanding dependencies.

[apparentlymart]

Hi @cemenson! Thanks for sharing this use-case.

I've seen other users achieve this by making the vpc configuration only responsible for configuring the main network objects (VPCs and subnets) and then allowing each other configuration to create the security groups it needs locally, thus keeping them close to where they are used (so that they can be very specialized to a particular need). This also avoids the need for a large amount of data shared between configurations as you've seen.

The indirection of modules returning data via outputs is an intentional language design decision to keep modules insulated from one another's implementation details. The Terraform language follows an "explicit is better than implicit" design principle, where we do not generally optimize for keeping files "small" and instead optimize for it being very clear how different objects are related. We usually make the analogy of a module being roughly equivalent to a function in a programming language: it takes inputs and produces outputs, and the caller does not generally have access to the local variables and temporary objects created by the function's implementation. Just as with functions, part of the job of designing a complex system with Terraform is choosing the best places to decompose the system while considering separation of concerns and other common software design principles.

However, we also know that certain use-cases tend to end up creating very systematic configuration with repeated constructs that are not convenient for users to edit directly. In that situation, we encourage the use of code generation to write the repetitive blocks, and then commit the result of that code generation (usually alongside the program that created it) into your version control system. In your case, the code generator might produce both the aws_security_group resource blocks and a single output that exports their ids as a map for convenient consumption in callers:

output "security_group_ids" {
  value = {
    example1 = "${aws_security_group.example1.id}"
    example2 = "${aws_security_group.example2.id}"
    example3 = "${aws_security_group.example3.id}"
  }
}

Then the caller (either directly as a child module or indirectly via terraform_remote_state) can access the keys of this map to get the security groups they need.

The forthcoming 0.12 release of Terraform will make it possible to export the whole resource object for each security group, which may make things easier in your case:

# For Terraform 0.12 and above only
output "security_groups" {
  value = {
    example1 = aws_security_group.example1
    example2 = aws_security_group.example2
    example3 = aws_security_group.example3
  }
}

With all of this said, since not exposing the individual resources is an intentional design decision and so not one we are likely to change in the near future. However, we will keep this use-case in mind and see if it can be wholly or partially addressed by other language changes in future while preserving the language design goals.

Thanks again for sharing this use-case!

[cemenson]

Hi @apparentlymart!

Thanks for the advice. I'll take than on board moving forward.

Just 1 question - can you elaborate on what you mean with regards to "code generation"? Do you mean using a seperate tool altogther to produce the outputs?

If so, I did try something similar to what you said by actually using terraform's local-exec provisioner in a null resource. What I tried was a follows:

Created a seperate file named "z_setOutput.tf" ("z" prefix so it executes LAST) which contained something as such:

resource "null_resource" "setOutput" {
  provisioner "local-exec" {
    command = <<EOF
    $resources = terraform state list
    echo 'output "securityGroups" {' >> output.tf
    foreach ($r in $resources) {
        $line = ('${"'+ $r +'.name}" = ${'+ $r +'.id}"')
        echo $line >> output.tf
    }
    echo '}' >> output.tf
    terraform apply
    terraform taint null_resource.setOutput
    EOF
    interpreter = ["PowerShell"]
  }
}

But I ran into a few errors and eventually gave up. The above code isn't exactly right, but it's off the top of my head what I can remember. It would need to have some quotation marks and dollar signs escaped, and maybe some other things...but you get the idea.

Do you think that maybe, as opposed to using something seperate from terraform, something like the above could be incorporated into terraform as a new provisioner or something?

I know some golang, and would love to help but don't really have any idea where to start (I've never contributed to a project before).

The only other problem with the above approach (even if it did work) is that it requires the user to be running windows with powershell...as well as being very hacky.

[apparentlymart]

Hi @cemenson!

When it comes to code generation, I mean to run a program outside of Terraform to generate the Terraform configuration files which you then check in to version control and apply. Think of it as a tool to save you the trouble of typing it out manually: you still want the fully-expanded thing checked in to your version control so you and your colleagues can easily read it, but by generating it with a script instead of with your keyboard you avoid the risk of the different copies growing out of sync over time.

Ultimately the goal here is to ensure that there's a readable, straightforward copy of what was applied in your version control history for future reference. Generating it during the Terraform run would not achieve that goal, because you will have already committed the change to version control before the code runs. (and indeed, as you found, it won't work because the parser already ran long before Terraform takes any actions that have side-effects).

You can write this helper program in whatever language you are comfortable with or that is convenient for your team to use.

[cemenson]

Hi @apparentlymart!

OK thank you. In my personal (and novice) opinion, I think that incorporating a feature that is able to generate the outputs (as the code generation could) into the existing terraform CLI would be sensible, as it would reduce dependencies on other tools and be "cleaner". But maybe I'm not seeing the bigger picture.

Thanks for your all your help!