Unexpected error in v0.13.1-beta1: Error: Module does not support for_each

[percygrunwald]

Terraform Version

Terraform v0.13.0-beta1

Terraform Configuration Files

# examples/aws/prometheus_ha_pair/variables.tf
...
variable "ha_pairs" {
  description = "Map of Prometheus HA pairs to create in the form id => { hostname => instance_type }"
  type        = map(map(string))
  default = {
    pair0 = {
      usw403 = "m5.xlarge"
      usw404 = "m5.xlarge"
    },
    pair1 = {
      usw405 = "m5.xlarge"
      usw406 = "m5.xlarge"
    }
  }
}
...

# modules/aws/prometheus_ha_pair/main.tf
terraform {
  required_version = ">= 0.12, < 0.14"

  backend "s3" {}
}

provider "aws" {
  region  = var.aws_region
  version = "~> 2.0"
}

module "app_instance" {
  source = "../app_instance"

  for_each = var.ha_pairs

  aws_region = var.aws_region

  instances         = each.value
  ...
}

# modules/aws/app_instance/main.tf
terraform {
  required_version = ">= 0.12, < 0.14"

  backend "s3" {}
}

provider "aws" {
  region  = var.aws_region
  version = "~> 2.0"
}
...
resource "aws_instance" "this" {
  for_each               = var.instances
  ...
}
...

Expected Behavior

Modules should be nestable with each level able to define their own provider config, regardless of whether the child module is called with for_each or not.

Actual Behavior

Modules are currently nestable with each level able to define their own provider config, unless the child module is called with for_each. This is unexpected and would create "special" modules that can't define their own provider blocks and must be nested to be used (i.e. they cannot be used as top-level modules).

Calling a child module that defines its own provider block from a parent module that also defines it's own provider block results in this error only if the child is called with for_each from the parent:

$ terraform init                                     
Initializing modules...
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Module does not support for_each

  on ../../../modules/aws/prometheus_ha_pair/main.tf line 15, in module "app_instance":
  15:   for_each = var.prometheus_ha_pairs

Module "app_instance" cannot be used with for_each because it contains a
nested provider configuration for "aws", at
../../../modules/aws/app_instance/main.tf:7,10-15.

This module can be made compatible with for_each by changing it to receive all
of its provider configurations from the calling module, by using the
"providers" argument in the calling module block.

Steps to Reproduce

Please see code above. I can create code for a minimal PoC, but it is easy enough to reproduce this error for anyone experienced with terraform:

1. Create a child module that creates some resource and defines its own provider block
2. Create a parent module that defines its own provider block and calls the child module without for_each
3. Run terraform init and apply on the parent module and everything works
4. Change the parent module to call the child module with for_each
5. Run terraform init and apply on the parent module and receive an error

Additional Context

Hi there, we have been waiting for the 0.13 so that we can try for_each for a module, but are running into some unexpected behavior based on trying to abstract upon an existing module. Some background on the existing module and our use case:

• We have an app_instance module that launches 1 or more aws_instances based on an input map variable of the form hostname => instance_type
• We would like this module to have the ability to be called directly as a top-level module, as well as a module that can be composed into other modules (with and without for_each)
• We are trying to build a new module called prometheus_ha_pair, which launches 1 or more pairs of instances based on the input variable ha_pairs, which is a map of pair_id => instances (see below for example)
• The prometheus_ha_pair module would create multiple pairs by calling the app_instance module in a for_each loop

We tried to do this with the code below, but hit an error:

$ terraform init                                     
Initializing modules...
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Module does not support for_each

  on ../../../modules/aws/prometheus_ha_pair/main.tf line 15, in module "app_instance":
  15:   for_each = var.prometheus_ha_pairs

Module "app_instance" cannot be used with for_each because it contains a
nested provider configuration for "aws", at
../../../modules/aws/app_instance/main.tf:7,10-15.

This module can be made compatible with for_each by changing it to receive all
of its provider configurations from the calling module, by using the
"providers" argument in the calling module block.

We have been able to resolve the problem by removing the provider block from the app_instance module, which then implicitly uses the provider config from the calling module. The problem with this is that now the app_instance module can no longer be used as a top-level module, since it has no provider configuration of its own. This also means we would have a module that is inconsistent with our other modules, which all define their own provider blocks and which cause no problem with nesting inside other modules that define their own provider block.

We have tried keeping the provider blocks in place in both parent and child modules, then passing in the providers key to the child module from the parent to try to "overwrite" the child's provider, but this just gave the same error.

References

N/A

[apparentlymart]

Hi @percygrunwald! Thanks for opening this issue.

What you've tried to do here is intentionally not supported in v0.13.0, as described in the v0.13 beta guide.

In later versions of Terraform we are intending to make it invalid to write provider blocks anywhere except root modules, as has been the recommendation since Terraform 0.11. For now, this remains supported for modules that are not using count and for_each to retain compatibility with existing usage for now. However, it is technically infeasible to support modules in nested providers when count and for_each are present, so that will never be supported. Instead, we're planning to address it by changing the language features that associate provider configurations with resources so that provider configurations can be assigned more dynamically, which will follow in a later Terraform release.

Since there isn't any immediate action possible to fix this for now, I'm going to close this. We'll open other issues later for the provider-configuration-related language changes once there are more complete details to share.

Thanks again for opening this issue, and for the thorough reproduction steps you included.

[percygrunwald]

@apparentlymart, thanks for the detailed response, I missed that advisory and we will update our modules in accordance with that. Sorry for the noise!

[percygrunwald]

This deprecation of provider blocks in anything other than the highest-level parent module will potentially cause a bit of pain for us, especially around keeping stuff DRY. Is there a thread to discuss the future implementation?

[marlock9]

This makes impossible to use for_each for modules for multiregional cloud setups. For example, my module is one region infrastructure resources. I have different sets of AWS regions in which I need to deploy that infrastructure in every environment (workspace), so every environment setup is unique (no, I can't do anything with that). It makes non sense to have for_each for modules if I need to define AWS providers manually for each possible region for every environment in root module. And even if I define all providers, it's impossible to pass them to for_eached region module because provider could be addressed only by literal alias value and it's impossible to use "each.key" here. So, the feature I've been waiting and really wanting is useless for now for simplifying multiregional setups.

[jbergknoff-rival]

We're in a similar boat. We Terraform our AWS organization, with a subaccount module for each subaccount in the organization, and that module sets up a provider scoped to the subaccount. We were waiting for module for_each to better organize the code for this (less repetition), but it seems that this isn't possible because the providers need to be defined at the root level, and there's no for_each for providers. :-1:

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.