Feature Request: Provider restrictions for plan/apply

[jaygorrell]

I'd like to suggest the ability (stolen from @denniswebb) to run a plan/apply against only specified provider in a project -- this could likely be implemented as an attribute of the provider blocks, giving lots of flexibility and control to the user through the usual input methods without introducing new user-facing systems.

For example:

provider "aws" {
  region  = "us-east-1"
  enabled = "${var.provider_aws_enabled}" 
}

provider "datadog" {
  api_key = "${var.datadog_api_key}"
  app_key = "${var.datadog_app_key}"
  enabled = "${var.provider_datadog_enabled}"
}

With this feature, larger projects can be broken up for speed purposes or more granular control over what gets applied. In one of our cases, we have 100's of Datadog alarms (2-3 per aws resource) and this really slows down Terraform runs and hammers the Datadog API -- these don't change often so the ability to not run that provider when rolling out infrastructure changes would be a huge speed benefit.

[grubernaut]

Hey @jaygorrell, thanks for the issue!

Just brainstorming here, but since we have -target would it be beneficial to be able to specify a single provider via -target as well as specific resources? Or maybe a -provider-target

This would allow a user to say terraform apply -provider-target=datadog or terraform apply -provider-target=aws instead of adding more configuration items to every provider's config structure.

[jaygorrell]

That would be great as well but as a enterprise user, this leads into a separate request -- we would need the ability to specify the -target or -provider-target from an Atlas run. The dialog that pops up for a manual run would be a perfect place for this but I'm getting out of scope now. :)

Supporting both approaches so enterprise TF can add support for -target would be ideal, though. That's needed even for the existing -target support.

[apparentlymart]

It sounds like the issue here is that refreshing dozens of resources takes a long time. I can definitely relate to that from having used Terraform to manage user accounts and git repositories.

I wonder if it would help to have a way to selectively disable just the refresh walk for certain resources. I used to work around similar problems with -refresh=false but that is tricky when using automation like Terraform Enterprise.

[Dominik-K]

I like the -provider-target flag from @grubernaut, too.

My use-case:

I'm peering VPC's from different accounts. To restrict Terraform's power to account A, I normally use aws-vault like this:

aws-vault exec account-A_admin -- terraform apply

To setup the peering and routing on account B, it would be nice to run

aws-vault exec account-B_admin -- terraform apply -provider-target=aws.account-B

aws-vault explicitly show the permissions scope, the provider's assume_role does it implicitly. That's why I'm prefering aws-vault for that.

[apparentlymart]

As part of the current work on improving the configuration language (coming in the next major release) we've done some refactoring of how -target works in preparation for some improvements to it in later releases. While we'll not be able to get to this just yet (the Terraform team at HashiCorp is still focused on configuration language improvements for the moment), I do like the idea of supporting targeting by provider, and I think our refactored handling of targeting should make it a little simpler to support something like this in a future release:

terraform apply -target=provider.aws

This is essentially the same idea as the separate -provider-target option, but just adjusted to extend the existing "target" syntax (which already supports whole modules, resources, and individual resource instances) to include providers too.

The internal model here is a function that can take a target address like this along with a particular resource instance and determine whether the resource instance is "contained within" the target. Since Terraform knows which single provider configuration is associated with each resource, it should be able to answer that question here, though it may require a little more juggling to ensure that the "associate resources with providers" step happens before the "remove untargeted graph nodes" step, which I think isn't true today.

We'll investigate and prototype some more once the dust has settled on the configuration language changes.

---

Some related longer-term changes in this area include #7388 which is about ways to optimize how we call into APIs that support some sort of batch request mechanism, and #4149 which is about (amongst other things) Terraform realizing that it's not yet able to configure a particular provider and to effectively automatically exclude it from targeting until dependent resources are created.

Provider-based targeting could potentially be a good shorter-term workaround for both of these things until we're able to get into the larger changes there, though we'll see during prototyping if that's actually true in practice. :thinking: