Support for storing plan files in backends

[ljfranklin]

We use Concourse CI plus this Terraform Resource to automate Terraform. I'm trying to switch from a custom S3 client for storing state and plan files to the built-in Terraform backends. Storing state files is working fine, but I'm having trouble dealing with plan files.

As described in Plan and Apply on different machines, we need a way to run terraform plan -out=tfplan in one CI job, persist the plan file somewhere, the run terraform apply tfplan in another CI job. The user already needs to provide a backend config to store the statefile, it would be great to use the same backend to store the plan file. Is there a supported way to store the plan file via the CLI? If not, could the remote-state package be used to store a plan file?

[nbering]

I've seen your Concourse Resource Type before and was actually looking at possibly using it myself if our company selects Concourse (we're currently looking at all solutions before settling). Though I must confessed that I haven't gotten much further than reading the README.

I've also felt that the plan-and-apply workflow could use a fresh look with collaborative environments in mind.

In the mean-time, I would think for Concourse (and most other CI tools), that using build artifacts would be the simplest solution.

[ljfranklin]

In the mean-time, I would think for Concourse (and most other CI tools), that using build artifacts would be the simplest solution.

In Concourse, you have to use Resources (e.g. s3-resource) to move artifacts between jobs. Although you could configure the Terraform Resource with your backend creds and also configure a separate S3 Resource or similar with the same creds to pass the plan file, it's not ideal. I'd like to have the Terraform Resource be self-contained and allow you to generate a plan in one job and manually trigger a second job after review to apply that plan.

[apparentlymart]

Hi @ljfranklin! Thanks for this suggestion.

When backends were added in 0.9, the initial focus was on parity with the existing idea of a remote state backend, with the added concept of state environments, which have since been renamed to workspaces.

That initial work also put in place the foundations of a longer-term goal: remote operations. While not yet implemented, the intent of remote operations is to allow the operations that access remote APIs (refresh, plan, apply) to optionally be run remotely themselves, with the intent that then the backend would be responsible for providing credentials to the providers, storing plans, etc. The goal here is to avoid the need to distribute a variety of different credentials to everyone who might apply a particular configuration, and instead just distribute credentials for the backend itself.

The initial design sketches for this change the workflow slightly to accommodate remotely-stored plans:

# -save replaces -out=tfplan to call for the plan to be saved remotely
$ terraform plan -save
...

This plan was saved with id: abc123

To perform exactly these actions, run the following command to apply:
    terraform apply "abc123"

The idea here is that the backend produces some sort of opaque id representing the plan (which could be a filename, a key into a database table, or anything else) which then serves the same purpose as the plan filename.

All of what I described above is a lot more than what you asked for here, of course. I bring it up only because the generalizations that would be required to support remote plan/apply might also create the right abstractions to support remote plan storage even if execution is local. However, there are a few details here that don't quite line up and require more thought:

• A backend like s3 wouldn't be able to automatically allocate an id for a plan, so it would still be necessary for a user to provide a plan name on the command line. Perhaps for backends like this -out=... would still be used but would change so that the interpretation of its value is defined by the backend, with the s3 backend using it as part of an object key when remote plans are enabled.
• The current workflow doesn't have any step where a plan gets automatically deleted. Currently, users do this just by deleting the local disk file. Keeping an archive of historical plans is useful in some cases, so maybe this is not a big deal. (A backend that executes operations remotely would have this issue too, of course.)
• Although an automation tool like yours can, in principle, arrange to coordinate plan storage names so that they don't collide between runs, teams that aren't using orchestration would need some way to coordinate so they don't clobber each other's plan names, which could have some very bad consequences if e.g. User A creates a safe plan, User B then overwrites it with a destroy plan, and then User A accidentally applies that new plan.

So with all of that said: the root idea here seems like a good one, but I think there are some design details to figure out before we could do it, and we'd need to make sure whatever we come up with also works consistently with the more-advanced "remote operations" concept so we don't end up with three separate workflows.

I think we'll need to let this one soak for a little while, and potentially just include it as an additional use-case for remote operations since it seems like they have a bunch of work in common. Remote operations is not something we're working on immediately, but it's something we'd like to do somewhat soon to make Terraform easier to use in more complex environments.

[ljfranklin]

@apparentlymart thanks for the detailed explanation! I think the terraform plan -save flow would work just fine for me. Would need a -json flag to be able to grab the plan ID in an automated way, but otherwise looks great.

Give a shout if y'all need help updating backends to make it compatible with this flow, but I understand if things are still in the design phase for now.

[nbering]

For reference, here's another detailed use-case where the Build Kite team is using their own service w/ Terraform to manage the service's infrastructure. They can also fall back to running commands locally should something go very badly.

https://building.buildkite.com/terraform-techniques-with-buildkite-81c0d6619f24

[marco-m]

Hello any news ?

[eedwards-sk]

I could really use this feature for the same reasons. I'm also using concourse and found the same issues when trying to use terraform as a custom resource.

I designed a bit of a workaround for now, rather than trying to make a concourse resource, I made a series of concourse tasks, which allow you to generate plan artifacts, and then store them with your concourse resource type of choice.

https://github.com/Snapkitchen/concourse-terraform

Ideally though, I wouldn't have to manage plan files at all, and could let terraform do it, similar to how well the current remote backends work.

[nbering]

This marketing popup on Terraform.io suggests remote plans is a thing... but I don't see anything in the changelog....

I may be mis-understanding the wording, though?

[apparentlymart]

"Remote Plan & Apply" for the purpose of that webinar is talking about the remote operations features I was describing earlier. That is, it's talking about running plan operations remotely, rather than running the plan operation locally and storing the plan remotely.

[julienfr112]

What is the status on that ? any work around ?