Ability to pass providers to modules in for_each

mightyguava commented 4 years ago

Use-cases

I'd like to be able to provision the same set of resources in multiple regions a for_each on a module. However, looping over providers (which are tied to regions) is currently not supported.

We deploy most of our infra in 2 regions in an active-passive configuration. So being able to instantiate both regions using the same module block would be a huuuge win. It's also our primary use case for for_each on modules being implemented in https://github.com/hashicorp/terraform/issues/17519.

Proposal

Proposed syntax from @jakebiesinger-onduo

provider "google" {
  alias = "goog-us-east1"
  region = "us-east1"
}
provider "google" {
  alias = "goog-us-west1"
  region = "us-west1"
}
locals {
  regions = toset(['us-east1', 'us-west1'])
  providers = {
    us-east1 = google.goog-us-east1
    us-west1 = google.goog-us-west1
  }
}
module "vpc" {
 for_each = local.regions
  providers = {
    google = local.providers[each.key]
  }
  ...
}

Another option would be to de-couple the region from providers, and allow the region to be passed in to individual resources that are region aware. As far as I know, both AWS and GCP credentials at least are global.

References

count and for_each for modules https://github.com/hashicorp/terraform/issues/17519
looping over providers https://github.com/terraform-providers/terraform-provider-bigip/issues/247
Multiple regions with same Provider https://github.com/hashicorp/terraform/issues/451
Reddit: https://www.reddit.com/r/Terraform/comments/cwp0d4/terraform_multi_region_question_can_i_just_use/

hashibot commented 4 years ago

Hello! :robot:

This issue seems to be covering the same problem or request as #9448, so we're going to close it just to consolidate the discussion over there. Thanks!

jspiro commented 4 years ago

Uh. These requests are in no way similar. Bad bot.

mightyguava commented 4 years ago

Yeah, this is not the same as #9448 at all. @pselle @apparentlymart could you help with reopening this issue?

pkolyvas commented 4 years ago

Hey there @mightyguava & @jspiro,

I'm going to re-open the issue as I agree that the concerns are not the same.

I did rename it for clarity; to distinguish this request from instantiating providers with for_each.

s1mark commented 4 years ago

@mightyguava I ran into the same abstraction issue with the azurerm provider. My goal was to automate multiple azure subscriptions and keep the code DRY as possible. Since I have to use Service Principals for auth with the azurerm provider, each subscription requires a separate provider declaration. I have ended up using terragrunt's generate function (https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#generate)

.
├── dev
│   └── terragrunt.hcl
├── modules
│   └── my_module
│       └── main.tf
├── prod
│   └── terragrunt.hcl
├── stage
│   └── terragrunt.hcl
└── variables.tf

./dev/terragrunt.hcl:

terraform {
  source = "${get_parent_terragrunt_dir()}/../"
}

# will generate content to ./providers.tf
generate "providers" {
  path      = "providers.tf"
  if_exists = "overwrite"
  contents = <<EOF
provider "azurerm" {
  # my main provider
  version         = "~> 2.6"
  subscription_id = "11111111-2222-3333-4444-555555555555"
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
  features {}
}

provider "azurerm" {
  alias           = "my_alias_provider"
  version         = "~> 2.6"
  subscription_id = "66666666-7777-8888-9999-000000000000"
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
  features {}
}
EOF
}

# will generate content to ./main.tf
generate "main" {
  path      = "main.tf"
  if_exists = "overwrite"
  contents = <<EOF
module "my_main_provider" {
  source = "./modules/my_module"
}

module "my_alias_provider" {
  source    = "./modules/my_module"
  providers = {
    azurerm = azurerm.my_alias_provider
  }
}
EOF
}

./variables.tf:

# passing these from cli or exporting to TF_VAR
# Note that both of my subscriptions use the same SP for auth and 
# both in the same tenant, so the difference is only the subscription_id
variable "client_id" {}
variable "client_secret" {}
variable "tenant_id" {}

The ./modules/my_module/main.tf contains the desired code without any provider block declaration (passing down provider declaration from root to child module)

Maybe this is not fully covering your scenario, but it provides some flexibility over different environment settings

vivanov-dp commented 4 years ago

First of all, thanks for the great work adding iteration and depends_on for modules - both are going to be really useful and I wished for them so many times back during 0.11 days when we were building the majority of our config.

In addition to each.key, I'd expect to be able to freely use maps with for_each and have each.<property> be a provider. This would require the ability to assign a provider "instance" to a local or list/map members. For example:

locals {
  modules_vars = {
    instance_1 = {
      var1 = ...
      var2 = ...
      provider = aws.euw1            // ERROR
    }
    instance_2 = {
      var1 = ...
      var2 = ...
      provider = aws.cnnw1           // ERROR
    }
  }
}

module "something" {
  source = "./module_something"
  for_each = local.modules_vars

  providers = { aws = each.provider }           // ERROR
  var1 = each.var1
  var2 = each.var2
}

This was the first thing I tried to do when I learned that 0.13 has for_each for modules, which brought me to #17519 and eventually - here. Since we maintain infrastructure in multiple AWS regions and availability zones around the world, most of the modules in our configuration require passing a provider along with at least a few other variables.

gbataille commented 4 years ago

While not strictly the same as #9448 I think they might be solved together.

First, like @vivanov-dp said, thanks for adding the for_each support for modules. I had been expecting it for a long time.

However I had not realized that provider configuration in modules was deprecated.

Here is my use case:

I use terraform to manage the list of AWS accounts I have in my organization
When I create a new account (from a variable list), I then want to provision it with a few common standard resources.

What I have now is that all those standard resources are in a module. I instanciate the module once per sub account and I pass the IAM role to the module. The module then opens a provider connection to the right account and the right role (different for each module instance).

This still works in 0.13. However, when I tried to migrate to "for_each" to instanciated all the modules for all the sub-account in a single module block, I hit the issue that providers inside modules are not supported anymore.

And since I can't construct a dynamic list of providers, I think I'm stuck.

Am I right to think there is currently no workaround for my use case?

Should I then split account creation and account "basic provisionning" in 2 different terraform projects?

thanks

nikolay commented 4 years ago

I personally think that inline provider declaration, which honors the module for_each or count is the cleanest solution:

module "some_module" {
  source = "./some-module"
  for_each = local.modules_elements

  provider "provider1" {
    ...
  }

  provider "provider2" {
    ...
  }

  var1 = each.var1
  var2 = each.var2
}

Ideally, this would support dynamic for providers as well.

Another option is to add for_each for provider as well along these lines:

provider "aws" {
  for_each = var.regions
  region = each.value
}

resource "some_type" "some_id" {
  provider = aws["us-east-1"]
}

vivanov-dp commented 4 years ago

@nikolay That can do the job, but why creating new providers for each module invocation ? Even if it is "for free" in terms of performance, which I don't really know, there are a bunch of properties to configure the provider and this approach would require to put them all into local.modules_elements and list them all in each provider declaration in each module invocation. You can't really declare an AWS provider just by setting the region. It requires a profile, or access_key&secret_key too and it is very likely that the assume_role would also be set. It has 15+ properties and many of them become useful as the architecture grows.

nikolay commented 4 years ago

@vivanov-dp This was pseudocode just to illustrate my point, which was that the logic of how the provider should be initialized could be encapsulated in the module. I can't think of a situation where the instantiation of a provider would be an expensive operation. Also, providers with assume_role have session information, which may not make sense to be reused across different modules, but will happen due to natural laziness if we have to create too many aliases.

vivanov-dp commented 4 years ago

@nikolay What I understand is that you propose to have this:

locals {
  modules_vars = {
    instance_1 = {
      var1 = ...
      var2 = ...
      region = ...
      profile = ...
      role_arn = ...
    }
    instance_2 = {
      var1 = ...
      var2 = ...
      region = ...
      profile = ...
      role_arn = ...
    }
  }
}

module "some_module" {
  source = "./some-module"
  for_each = local.modules_vars

  provider "aws" {
      region  = each.region
      profile = each.profile
      assume_role {
          role_arn = each.role_arn
      }
  }

  var1 = each.var1
  var2 = each.var2
}

instead of:

provider "aws" {
    alias   = "euw1"
    region  = "eu-west-1"
    profile = var.aws_west_profile
    assume_role {
        role_arn = "arn:${var.aws_partition}:iam::${var.aws_account_id}:role/TerraformRole"
    }
}
provider "aws" {
    alias   = "cnnw1"
    region  = "cn-northwest-1"
    profile = var.aws_cn_profile
    assume_role {
        role_arn = "arn:${var.aws_cn_partition}:iam::${var.aws_cn_account_id}:role/TerraformRole"
    }
}

locals {
  modules_vars = {
    instance_1 = {
      var1 = ...
      var2 = ...
      provider = aws.euw1
    }
    instance_2 = {
      var1 = ...
      var2 = ...
      provider = aws.cnnw1
    }
  }
}

module "something" {
  source = "./module_something"
  for_each = local.modules_vars

  providers = { aws = each.provider }
  var1 = each.var1
  var2 = each.var2
}

But then I have one set of providers for everything else and one set of the same properties just for the modules. So which one is the source of truth ? Unless I declare my providers by using the same sets of properties - so I have to create a new abstraction - the set of providers properties and use that in all places.

As I said - it can do the job, I think it looks nice and is not a bad idea, but IMO it involves more changes to the existing configuration than if we could just use the already defined providers - which in my case are in an external file, often 1 or 2 directories up the hierarchy and propagated down via a script that generates main.tf & variables.tf.

jon-walton commented 4 years ago

A use case for me would be to configure a dynamic provider based on output from a module using for_each

such as creating multiple kubernetes clusters (foo) and optionally applying resources (bar)

module "foo" {
  source = "./foo"
  for_each = var.foo_things

  var1 = each.key
  var2 = each.values.something
}

module "bar" {
  source = "./bar"
  for_each = { for k, v in var.bar_things : k => v if v.add_bar_to_foo == true }

  provider "some_provider" {
    config1 = module.foo[each.values.foo_thing].output1
    config2 = module.foo[each.values.foo_thing].output2
    config3 = module.foo[each.values.foo_thing].output3
  }

  var1 = each.key
  var2 = each.values.something
}

nikolay commented 4 years ago

@vivanov-dp The ideal approach is to have identical code and only data, which varies between environment and clusters within the environment. Right now, almost everything has for_each/count except providers.

nikolay commented 4 years ago

@jon-walton Your example is identical to mine, but I illustrated if the module needs more than a single provider.

jon-walton commented 4 years ago

@jon-walton Your example is identical to mine, but I illustrated if the module needs more than a single provider.

My example illustrates the provider config being supplied to a module is set by the output of another module which also uses for_each

vivanov-dp commented 4 years ago

@nikolay Sure, having for_each for providers sounds logical and natural and I fully support it, I believe it deserves its own feature request

nikolay commented 4 years ago

@jon-walton Fair enough, we need dynamic providers - one way or another. Right now providers and outputs are the only two static resources in Terraform.

apparentlymart commented 4 years ago

Hi all! Thanks for the interesting discussion here.

It feels to me that both this issue and #9448 are covering the same underlying use-case, which I would describe as: the ability to dynamically declare and use zero or more provider configurations based on data determined at runtime.

These various proposals all have in common a single underlying design constraint: unlike most other concepts in Terraform, provider configurations must be available for operations on resources that belong to them, which includes planning, updating, and eventually destroying. This means that a provider configuration must be available at the same time a new resource is added to the configuration, must have a stable name that can be tracked between runs in the Terraform state, and they must continue to be available until every resource instance belonging to them has been destroyed and/or removed from the state.

It is due to that design constraint that provider configurations remain separated from all other concepts in the restrictions placed on them in the configuration. Design work so far seems to suggest that there are some paths forward to making provider configuration associations (that is, the association of resources to provider configurations) more dynamic, but the requirement that each provider configuration be defined by a static provider block in the root module seems necessary to ensure that the provider block can remain in the configuration long enough to destroy existing resource instances associated with it, which happens after they are removed from the configuration.

One design we've considered (though this is not necessarily the final design we'd move forward with) is to make provider configurations a special kind of value in the language, which can be passed by reference through expressions in a similar sense that other values can. For example:

variable "networks" {
  type = map(
    object({
      cidr_block   = string
      aws_provider = providerconfig(aws)
    })
  )
}

resource "aws_vpc" "example" {
  for_each = var.networks
  provider = each.value.aws_provider

  cidr_block = each.value.cidr_block
}

The aws_provider attribute here is showing a hypothetical syntax for declaring that an attribute requires a configuration for the aws provider, with that reference then usable in provider arguments in resource and data blocks where static references would be required today. That syntax is intended to replace the current "proxy provider configuration" special-case syntax, by allowing provider configurations to pass through variables instead. However, this design does have the disadvantage of requiring explicit provider configuration passing, whereas today child modules can potentially inherit non-aliased provider configurations automatically in simple cases.

However, the calling module would still be required to declare the provider configurations statically with provider blocks, perhaps like this:

provider "aws" {
  alias = "usw2"

  region = "us-west-2"
}

provider "aws" {
  alias = "use2"

  region = "us-east-2"
}

module "example" {
  source = "./modules/example"

  networks = {
    usw2 = {
      cidr_block   = "10.1.0.0/16"
      aws_provider = provider.aws.usw2
    }
    use2 = {
      cidr_block   = "10.2.0.0/16"
      aws_provider = provider.aws.use2
    }
  }
}

Notice that the two provider configurations must still have separate static identities, which can be saved in the state to remember which resource belongs to which. But this new capability of sending dynamic references for provider configurations still allows writing a shared module that can be generic in the number of provider configurations it works with; only the root module is required to retain a static set of provider configurations.

There is also some possibility here of allowing count and for_each in provider blocks to permit provider addresses like provider.aws["use2"] (where use2 is the each.key), but this is more problematic because it creates another opportunity to "trap" yourself in an invalid situation: if you use the same value in for_each for both a resource configuration and its associated provider configuration, removing an item from that map would cause the resource instance and the provider configuration to be removed at the same time, which violates the constraint that the provider configuration must live long enough to destroy the instance recorded in the state. Given how common it is to get into that trap with provider blocks inside child modules today (which is why we've been recommending against that since Terraform 0.11), we're reluctant to introduce another feature that has a similar trap. For that reason, I predict that for_each and count for provider configurations (as proposed in #9448) won't make it through a more detailed design pass for this family of features.

I've shared the above mainly to just show some initial design work that happened for this family of features. However, I do have to be honest and share some unfortunate news: the focus of our work is now shifting towards stabilizing Terraform's current featureset (with minor modifications where necessary) in preparation for a Terraform 1.0, and a mechanism like the one I described above would be too disruptive to Terraform's internal design to arrive before that point.

The practical upshot of this is that further work on this feature couldn't begin until at least after Terraform 1.0 is released. Being realistic about what other work likely confronts us even after the 1.0 release, I'm going to hazard a guess that it will be at least a year before we'd be able to begin detailed design and implementation work for features in this family.

I understand that this is not happy news: I want this feature at least as much as you all do, but with finite resources and conflicting priorities we must unfortunately make some hard tradeoffs. I strongly believe that there is a technical design to address the use-cases discussed here, but I also want to be candid with you all about the timeline so that you can set your expectations accordingly.

nikolay commented 4 years ago

@apparentlymart Having providerconfig(aws) is a bit limiting as you can't pass the dynamic index from a TFC variable or terraform.tfvars.json file. The easiest and probably quickest to implement it just to allow something like provider.aws[var.provider_alias] - you still have static providers, just dynamic references to them.

cregkly commented 4 years ago

I refer to the blog announcement for TF 0.13 with this block of code:

variable "project_id" {
  type = string
}

variable "regions" {
  type = map(object({
    region            = string
    network           = string
    subnetwork        = string
    ip_range_pods     = string
    ip_range_services = string
  }))
}

module "kubernetes_cluster" {
  source   = "terraform-google-modules/kubernetes-engine/google"
  for_each = var.regions

  project_id        = var.project_id
  name              = each.key
  region            = each.value.region
  network           = each.value.network
  subnetwork        = each.value.subnetwork
  ip_range_pods     = each.value.ip_range_pods
  ip_range_services = each.value.ip_range_services
}

This implies we can do for_each over a region...

nikolay commented 4 years ago

@cregkly Yes, but we're talking about providers here, not modules.

vivanov-dp commented 4 years ago

@cregkly This example is with Google cloud - the provider instance is not constrained within the region with Google, so you don't need multiple provider instances to use different regions - resources have 'region' properties themselves

cregkly commented 4 years ago

@cregkly This example is with Google cloud - the provider instance is not constrained within the region with Google, so you don't need multiple provider instances to use different regions - resources have 'region' properties themselves

And I quote the original post:

I'd like to be able to provision the same set of resources in multiple regions a for_each on a module. However, looping over providers (which are tied to regions) is currently not supported.

And then they gave a google cloud example...

cregkly commented 4 years ago

@cregkly Yes, but we're talking about providers here, not modules.

Ability to pass providers to modules in for_each

cregkly commented 4 years ago

@apparentlymart Can you guys put a better example up on the blog post about TF 13 then? It uses the example of for_each over regions with google cloud. Naturally it is the first thing I wanted to try out with in AWS, then it turns out it can't be done.

At the very least link to the something that explains why this works with Google Cloud and not others like AWS.

I appreciate you insights and transparency on the development to version 1.

apparentlymart commented 4 years ago

I think the person who wrote that blog post was motivated to find an existing registry module with a relatively simple interface so that the module's own complexity wouldn't overwhelm the article with module-specific complexity. The point of it is just to be a generic (but working) example of what the syntax looks like for marketing purposes, not to be documentation. In general I'd suggest thinking of HashiCorp blog posts as being more "notification that the thing exists" than "guide/example on how to use the thing".

The HashiCorp education team wrote a long-form guide on for_each which discusses these things in more detail.

pkolyvas commented 4 years ago

I updated the blog post a while ago, but I am waiting for another team to push the changes live. It looks like our blogging platform was updated between the release of 0.13 and today.

The replaced example is designed to signal the for_each feature without misleading users to believing they can copy paste code and use it as is.

I apologize for the delay in getting this remediated.

Update: I went back to check and the blog post has been updated.

timmjd commented 4 years ago

Our use-case is the multi account setup where we deploy stuff like IAM roles for monitoring permission to all accounts and do have a centrally Grafana that does collect these data.

Looks like currently there is no way to handle this without an addon like terragrunt?

The following would be an example on how this could be handled if you require the provider to stay on root level. But this also requires to have the for_each available on providers.

# A list of AWS accounts that also might come from an external source (json / yaml)
locals {
  accounts = {
    "4711"    = { something = "foo" }
    "0815"    = { something = "bar" }
    ...
  }
}

# Generate a AWS STS token via Vault, each role is mapped to a different AWS account
data "vault_aws_access_credentials" "sts" {
  for_each    = local.accounts

  role        = each.key

  backend     = "aws"
  type        = "sts"
}

# Create a provider for each account by pasting in the STS tokens
provider "aws" {
  for_each    = local.accounts # MISING FEATURE

  region      = "eu-central-1"
  alias       = each.key

  access_key  = data.vault_aws_access_credentials.sts[each.key].access_key
  secret_key  = data.vault_aws_access_credentials.sts[each.key].secret_key
  token       = data.vault_aws_access_credentials.sts[each.key].security_token
}

# Paste the provider down to the the account module
module "account" {
  for_each    = local.accounts

  something   = each.value.something

  providers "aws" {
    aws       = aws[each.key]
  }
}

rjudin commented 4 years ago

We have the same use case as https://github.com/hashicorp/terraform/issues/24476#issuecomment-709070083 for AWS account bootstrap (has to iterate by each provider)

module "account" {
  for_each    = local.accounts

  something   = each.value.something

  providers "aws" {
    aws       = aws[each.key]
  }
}

m4ce commented 4 years ago

same problem here - it's quite a limitation and it makes for_each next to useless ..

philipp-ranft commented 4 years ago

@timmjd @rjudin @m4ce We had similar issues and we have come up with a combination of local_file (Documentation) and templatefile() (Documentation) to work around this issue:

resource "local_file" "per_account_generated" {
  content = templatefile(format("%s/per-account-generated.tpl", path.module), {
    accounts = local.accounts
  })
  filename = format("%s/per-account-generated.tf", path.module)
}

Contents of per-account-generated.tpl:

%{ for name, account in accounts ~}
module "${name}" {
  source = "./setup_per_account"

  account_id   = "${account.id}"
  account_name = "${name}"
  tags         = local.tags
}

%{ endfor ~}

This will generate Terraform code using Terraform. Therefore, if you add a new account you need to run terraform apply twice. The first run will update the per-account-generated.tf and the second run will create the resources. You may also need to do a terraform init in-between if you are using modules as we do.

Maybe this helps you in the meantime. 😃

rjudin commented 4 years ago

thank you, @Philipp-Navis for extraordinary solution ;-] I believe the Terraform team able to make it happen during one-shot by provider iteration. 'Apply terraform twice' is against idempotence principle which declarative IaaC follows.

s1mark commented 4 years ago

Since @apparentlymart noted that the terraform team will not implement this feature until they stabilize and reach v1.0.0 I thought to give another shot at this with the help from terragrunt. Earlier I have shared that terragrunt provides an abstraction layer eazing the problem a bit https://github.com/hashicorp/terraform/issues/24476#issuecomment-619450972

With the hint from the terragrunt community (thanks @lorengordon ) I have managed to dynamically generate the provider blocks for my dynamically generated modules in an iterative manner. At first I declared locals in the terragrunt.hcl

locals {
  modules = {
    "provider_01" = {
      "team_a" = "Contributor"
      "team_b" = "Contributor"
    }
    "provider_02" = {
      "team_a" = "Contributor"
      "team_b" = "Contributor"
    }
  }
}

and then using the generate function of terragrunt and heredoc syntax to iteratively create the proper content. Breaking down the process of this: The heredoc created string is picked up by the generate function which creates the main.tf file before terragrunt executes the terraform apply command.

generate "main" {
  path      = "main.tf"
  if_exists = "overwrite"
  contents  = <<EOF
%{ for sub, role in local.modules ~}
module "${sub}" {
  source    = "./module"
  role_map  = {
%{ for k, v in role ~}
    "${k}" = "${v}"
%{ endfor ~}
  }
  providers = {
    azurerm = azurerm.${sub}
  }
}

%{ endfor ~}
EOF
}

As you can see this is an example for the azure provider that invokes a module which manages IAM on the corresponding subscription defined in the providers. It is one step process and the only extra is the terragrunt wrapper (fyi @Philipp-Navis @rjudin @timmjd @milldr )

milldr commented 4 years ago

same issue here. I have essentially the same workaround as @Philipp-Navis except I generated the template with python + jinja2. Figured I'd share it here if @rjudin or anyone wanted a way to avoid "apply terraform twice" (even though it's still a 2 step process)

First I have a template such as roles.tf.j2 with something such as:

{% for alias in accounts %}
resource "aws_iam_role" "{{alias}}" {
  provider = aws.{{alias}}
...
}
{% endfor %}

then you can run something like this with python to fill it out:

def main():
    """ Render the Jinja2 template file
    """
    terraform_directory = os.path.abspath("terraform")
    template_directory = os.path.abspath("templates")

    role_template_path = os.path.join(template_directory, "roles.tf.j2")
    role_tf_path = os.path.join(terraform_directory, "roles.tf")

    # Get the template file content
    with open(role_template_path, "r") as terraform_template:
        templated_file_content = terraform_template.read()
    template = Template(templated_file_content)

    # Apply jinja and generate file
    role_tf = template.render(accounts=get_accounts())
    warning_header = (
        "# Warning: automatically generated file\n"
        + "# Please edit template/roles.tf.j2 and use the script make_roles.py\n"
    )
    with open(role_tf_path, "w") as terraform_file:
        terraform_file.write(warning_header)
        terraform_file.write(role_tf)

favoretti commented 4 years ago

What makes this very sad is the fact that aside of dynamically configuring main providers - 3rd party providers become very hard to use in modules. Our main use-case is DataBricks, where we have a module for_each loop to spin up the workspaces in azure (azurerm_databricks_workspace) and then need to bubblegum-tiewrap provisioning of internal workspace things by scripts and pipelines, rather than using databricks provider.

Essentially this means we either need to drop the use of the module and create dozen of files per workspace or abandon hopes of using the provider for now. I think it's pretty much the case for quite a few other 3rd party providers.

nikolay commented 4 years ago

I hate to say but Terraform is IaC where C stands for "configuration", not "code". Although I understand the technical challenges, to upgrade from "configuration" to "code", we really need some more dynamic abilities, which includes dynamic providers.

lorengordon commented 4 years ago

Sounds like a use case for cdktf...

nikolay commented 4 years ago

@lorengordon It's a nice excuse not to keep Terraform up with the demand. Even now we can do Jinja2 templating of .tf files, but many of us choose not to. I personally have been fascinated by AsCode experiment, but I still want to be able to do all these things with just pure Terraform. It's not a huge challenge - Terraform just needs to borrow certain things from Terragrunt and others from Pulumi and cdktf. Also, with all of its weirdness, lazy evaluation in Jsonnet makes a lot of sense and could be very expressive, too. I think the big issue of HasiCorp is that they wanted to invent TOML with HCL v1 and with v2 they admitted that they've made a mistake as nobody loves doing "${var.name}" when they can just use var.name instead. They also admitted their mistake again with Sentinel, which doesn't use HCL v1 or v2, but another symptom of the "not invented here" syndrome. To their defense, I am not sure if Rego predates Sentinel.

The issue with Jinja2 and the rest is that it requires a compilation phase, which is not something that works well with the present generation of Terraform Cloud. The same applies to cdktf, but Terraform Cloud will most definitely support it... one day (soon?).

I don't think any DevOps engineer will be happy to code infrastructure in TypeScript... when they just learned Go after the industry moved from Python to Go for systems programming.

lorengordon commented 4 years ago

@nikolay Please do not misunderstand, I am not disagreeing. I would also appreciate this feature native in terraform, or I would not be following this issue. I am simply pragmatic. I will do whatever I need to do, using whatever tool is available to me today, to meet the use cases that would otherwise be simplified if this feature were available natively in terraform.

LoicGombeaud commented 4 years ago

This feature would be a great improvement for our use-case:

in our root module, we have a nested "environment" module, instantiated with a for_each
in this nested environment module, we (would like to) use the mysql provider to create MySQL databases and users

Since every environment uses a different different MySQL server, thus a different provider, it's currently not possible to do this.

I understand the reason for keeping provider definition outside of nested modules, but it would be great to be able to:

create providers with a for_each in the root module
pass these providers in nested modules instantiated with a for_each.

alkalinecoffee commented 4 years ago

Mostly everyone in this thread wants to use one set of credentials to create resources across multiple regions. From my naive point of view, I have to ask: what's the purpose for having a region tied explicitly to credentials in the first place?

Credentials, while generated from a single region, are inherently global, whereas most resources are not. Yet by terraform design, the credentials and region values seem to be tightly coupled. Perhaps they should not be? The provider block already seems overloaded in the functionality it provides (where the core provider, credentials, and region are all configured), and is clearly becoming a hindrance.

If we can create resources across multiple regions with a single set of credentials using current AWS CLI tools, why should terraform be any different? Why couldn't we specify a region_override type of value on specific resources so that we can use a single provider but have the flexibility to manage resources across regions in a single account?

nikolay commented 4 years ago

@alkalinecoffee I agree. Many providers make the same assumption, unfortunately. It's the same situation with the GitHub provider and many more.

LoicGombeaud commented 4 years ago

@alkalinecoffee I agree, but this is an issue to raise with the maintainers of the AWS provider; the issue being discussed on this thread is not specific to one or the other provider.

nikolay commented 4 years ago

I think using region_override is a pretty bad workaround although it would be nice to have multiple provider instances without having to duplicate all other attributes just when one or two vary.

alan2wong commented 3 years ago

Recently I've been researching handling multi-region at the module level (as opposed to an entire stack level) and came up with a partial workaround to at least allow conditionals with providers, which could end up simulating a for_each using a lot of duplication:

provider aws {
  alias = "us_east_1"
}

data aws_region us_east_1 {
  provider = aws.us_east_1
}

module test {
  count = data.aws_region.name == "us-east-1" ? 1 : 0
}

Essentially, I have my module setup with a list of provider aliases using the name of all valid AWS regions. I noticed (at least with 0.13) that if the module consumes an aliased provider that was not passed into the providers map, it defaults to the default provider, which allows for the following trick: pass the provider into an aws_region and then check its resolved name versus its expected name.

Using this, I was able to dynamically generate one submodule per supplied provider by just having one conditional definition per region. Not super elegant, but it works.

[The main thing to note is that module deletion becomes complicated because the existence of resources is also determined by the existence of the provider, and Terraform fails to destroy the resource before the provider. I use an optional variable to explicitly define the regions for this case.]

pccowboy commented 3 years ago

using terraform 0.13. I have 3 modules, we'll use these names:

instance
specific
general

Modules "specific" and "general" have proxy provider blocks, each using an alias the same name as the module, i.e.:

provider "aws" {
  alias = "specific"
}

provider "aws" {
  alias = "general"
}

Module "instance" has an explicit provider defined

provider "aws" {
  alias = "instance"

  region = "us-east-1"
}

Module "specific" calls module "general".

module "specific" {
  source = "../general"

  providers = {
    aws = aws.specific
  }

  // do stuff to customise an ec2 instance
  <...>
}

Module "general" actually creates the instance:

resource "aws_instance" "general" {
  provider = aws.general

 <...>
}

Module instance uses "for_each" to call module "specific".

module "instance" {
  source = "../modules/specific"

  providers = {
    aws = aws.instance
  }

  for_each = data.aws_subnet.private.*
  <...>
}

That setup does not work, I get the message:

Module "instance" cannot be used with for_each because it contains a
nested provider configuration for "aws.specific", at 
../modules/specific/_providers.tf:1,10-15.

But if I pass the explicit provider to all the aliases from the "instance" module, it appears to work.

module "nat_instances" {
  source = "../modules/specific"
  providers = {
    aws.specific = aws.primary_pop
    aws.general  = aws.primary_pop
  }

  for_each = data.aws_subnet.private.*

  <...>
}

I am pretty sure I understand why passing the explicit provider works. I'm even more sure I'll regret going this route later.

My question is, why are we not able to pass aliased providers through a chain of modules that use provider proxy definitions in conjunction with for_each?

I've read through https://github.com/hashicorp/terraform/issues/24476#issuecomment-700368878 several times, but I still do not understand the delay in what should be a basic feature. Much as I appreciate tradoffs and wanting to get to a 1.0 release, the release is not much of a milestone if we are still using terragrunt or jsonnet to work around missing and long-needed features of HCL.

nikolay commented 3 years ago

It gets even uglier when you decide to use Terraform to manage a dynamic list of Kubernetes clusters where you need a specific copy for each provider. This limitation alone uglified our IaC to the level of ugly copypasta mess. I wonder why such a high-trafficked issue gets no attention from HashiCorp. This really ruins the day for people like us who use Terraform daily and who want to keep their IaC DRY!

madwolfa commented 3 years ago

It gets even uglier when you decide to use Terraform to manage a dynamic list of Kubernetes clusters where you need a specific copy for each provider. This limitation alone uglified our IaC to the level of ugly copypasta mess. I wonder why such a high-trafficked issue gets no attention from HashiCorp. This really ruins the day for people like us who use Terraform daily and who want to keep their IaC DRY!

I agree, it's pretty terrible. We have to use Jinja template generator for our multi-region deploys to go around this.

nikolay commented 3 years ago

@madwolfa One of the main reasons I got into Terraform was that templates were almost not required. Still, people struggle with these limitations and have to generate .tf files from templates or be stuck with IaSC (Infrastructure as Spaghetti Code).

tiagomatic commented 3 years ago

I think this has been resolved. I've noticed my IDE still doesn't like providers in module foreach loops, but it plans just fine now (I'm on terraform v0.15.1 for Mac).

hashicorp / terraform