feature: enable shared repository among multiple terraform projects

[rblumen-desk]

Use case:

• We are on Terraform Enterprise aka Atlas
• We have multiple git repos containing .tf files using the github integration to kick off TF jobs
• We have implemented a custom TF provider in Golang that multiple TF projects use
• The golang provider needs access to a set of shared files (non TF files)
• The set of shared files will live in their own git repo
• The golang provider needs a way to clone the shared file repo and locate the files in the Atlas environment

I opened a case with Hashi Support on this. The case number is 3295. The agent pointed out to me that you almost have this feature.

The solution was to create a module in the TF repos that use a TF module to import the shared files. The source attribute on the module would be provided to point to the git repo where the shared files live. TF will clone the git repo and place it under .terraform/.

The issue is how do resources created with the custom provider locate the shared source files that were cloned from git? The solution that was proposed in the support case was to use the fact the module.path is visible inside the module, and export it as an output of the module. Then the other resources that need it can import it using module.output.path.

The process would be slightly simplified if you made .path available as a default export or property of modules.

[apparentlymart]

Hi @rblumen-desk!

I'm first going to restate what I think is the problem you are trying to solve, in case I missed it: it sounds like you have a set of data files that you want to share across mutiple Terraform configurations from a separate, shared git repository. It also sounds like you've been advised by the customer success team to wrap those files in a Terraform module and you're looking for clarification on how best to do that.

In which case, my recommendation would be the following:

Place the files in question in your git repo, and then in the root of the repository make a single Terraform configuration file module.tf with the following contents:

output "path" {
  value = "${path.module}"
}

You can then reference your git repository as a child module in another configuration, and get access to that exported path name like this:

module "shared_files" {
  source = "git://...etc"
}

# Just a null resource as an example...
resource "null_resource" "example" {
  triggers = {
    file_contents = "${file("${module.shared_files.path}/example.txt"}"
  }
}

If the files in this repository have some interesting structure to them you could choose to have multiple outputs from this module exporting specific files or directories in the module, thus avoiding the need for the calling configuration to construct paths as I did in this example, but the above is the bare minimum Terraform configuration to export that path.

I hope that helps!

[rblumen-desk]

Hi @apparentlymart. The agent who handled case 3295 explained to me how to do this and provided an example so I am not looking for clarification about how to do that.

The purpose of the ticket was to suggest a feature enhancement that would make this slightly easier. I am thinking that this is a common use case so rather than providing a pattern with a boilerplate, provide language support for this.

My suggestion for the new feature was to expose .path as a default export or property of modules. The idea being to avoid the need to export .path as an output of the module. Then other TF resources could refer to ${module.x.path} with one less piece of code.

[apparentlymart]

Thanks for the clarification, @rblumen-desk.

Philosophically the design approach with Terraform is that modules are treated by their parents as "black boxes" and expose results only via explicit output blocks. This allows a module to present an explicit interface and thus reduce coupling between modules.

In this case, by adding the output for path.module as I suggested the module is explicitly giving its caller permission to access its files. I think we would not want to make this implicit since in larger, more complex systems this could create hidden coupling between modules where re-organizing the filesystem layout in one could inadvertently break another.

However, your use-case is an interesting one because what you want to share isn't really a Terraform module at all but rather just a set of files that would, ideally, be entirely opaque to Terraform. I could see us having a resource that would let you retrieve a remote repository just as pure data, thus avoiding the need for the shared module to contain any Terraform configuration files at all.

Here's how that might hypothetically look:

resource "local_dir_from_remote" "example" {
  source      = "git://..."
  destination = "${path.module}/shared-files"
}

However, this would not be 100% equivalent since the files would in this case be retrieved at apply time rather than at init time, so I'm going to tag this as "thinking" so we can let the implications of this soak for a little while. For now, the approach I gave above (which apparently echoed the support team) is the recommended method.

Thanks again!

[rblumen-desk]

@apparentlymart. Your discussion is going in the direction of what I really want, which is a way to share a set of opaque files among TF projects, from a git repo. If you could provide support for that as a first class resource that would be exactly my use case. For what I am doing, pulling the resources at apply time is probably better than init time.