Dynamic provider configuration assignment

[RafPe]

Suggest an existing issue in legacy Terraform to fix in OpenTF.

To increase automation we struggle many times when configuring providers. It would be great to finally be able to configure them dynamically or at least to be able to dynamically reference them instead of having all of that statically typed

[roni-frantchi]

Hey @RafPe thanks for submitting!

To help us maintain a clear separation between opentf and hashicorp's offerings, we're asking that people describe issues that are in other repositories rather than linking those directly. I've thus scrubbed out any links to said issues/PRs.

If there's any more context or description to the problem you think would be good to share and add in please do.

Thanks!

[RafPe]

@roni-frantchi Sure - I understand. Let me explain more in detail what I mean here.

Situation now :

we define our providers more less in a static way ( there is a level where we can use variables ) and then reference them completely static (no vars, no dynamics etc )

provider "aws" {
  region = "eu-central-1"
  alias  = "x1"
}

provider "aws" {
  region = "eu-west-1"
  alias  = "x2"
}

resource "aws_resource" "x1" {
  provider = aws.x1
}

resource "aws_resource" "x2" {
  provider = aws.x2
}

Scenario 1

dynamically referencing providers: Having the above providers, we could either have something in form of singleton object called providers to which one we could reference via alias the pointer to the provider we defined.

provider "aws" {
  region = "eu-central-1"
  alias  = "x1"
}

provider "aws" {
  region = "eu-west-1"
  alias  = "x2"
}

resource "aws_resource" "x1" {
  provider = providers.aws["x1"]
}

resource "aws_resource" "x2" {
  provider = providers.aws["x2"]
}

Scenario 2

dynamically referencing providers with for_each/count: Having the same above providers, we could either have something in form of singleton object called providers to which one we could reference via alias the pointer to the provider we defined but also support for_each/count on the providers level to create them dynamically

locals {
  aws_accounts = [
    { "aws_account_id": "123456789012", "foo_value": "foo",    "bar_value": "bar"    },
    { "aws_account_id": "987654321098", "foo_value": "foofoo", "bar_value": "barbar" },
    ]
}

## Here's the proposed magic... `provider.for_each`
provider "aws" {
  for_each = local.aws_accounts
  alias = each.value.aws_account_id

  assume_role {
    role_arn    = "arn:aws:iam::${each.value.aws_account_id}:role/TerraformAccessRole"
  }
}

## Here is the magic of referencing them in our resources 

resource "aws_resource" "x1" {
  for_each = local.aws_accounts

  provider = providers.aws[each.value.aws_account_id]
}

I believe this approach ( especially in bigger organisations ) would allow for more automation and maintaining the logic within the tool instead of leveraging external templating mechanism

Please let me know if the above makes sense :)

[jwenz723]

In my situation I have multiple AWS accounts (dev, test, prod). All accounts use the us-west-2 region, but the prod account also uses a few other regions.

So in my case I really need one provider enabled when deploying to dev and test and multiple providers enabled when deploying to prod. I am able to work around this currently by just including all regional providers in all accounts and I just don’t use the extra providers during a dev or test deploy.

This work around no longer works though when you have a module that needs to be deployed to multiple AWS partitions. For example standard and govcloud. A govcloud provider will not work when deploying to a standard partition and a standard provider will not work when deploying to govcloud. In this situation I am able to work around this by using some custom code outside of opentofu to generate the provider config before running opentofu.

It would be nice to just have the ability to do loops and conditionals on providers.

[timothyjlaurent]

This could be OpenTF's killer app ;)

[RafPe]

@roni-frantchi as I myself would be open to contributing to this ... would eb anyone else experienced already in the code base that could guide me towards making this possible ?

[roni-frantchi]

Hey @RafPe !

I myself would be open to contributing to this would eb anyone else experienced already in the code base that could guide me towards making this possible ?

Very much appreciated.
It'd be harder for us to provide such guidance, as right now our core team focuses on:

1. Rolling out an alpha in the next couple of days
2. Focusing on a design for a registry to power the stable release

Of course if you're willing to dive in there yourself or assisted by anyone else from the community would love to see such a design from you

[murarisumit]

Ref to issue in terraform repo, quite a bit of discussion over there on it: https://github.com/hashicorp/terraform/issues/19932

[dbhaigh]

Issue 19932 has been open since Jan 7th 2019 - it doesn't look like it's been given much love since

Plenty of people wanting this ability (myself included)

This ability would be fantastic, and yes, bring people over to the Open Source side

[Chandra2614]

This needs to be fixed ASAP, 3 years are a lot

[cube2222]

With the way providers currently work, esp. with for_each blocks, this would require a very technical and detailed RFC prior to being accepted.

But overall, we're open to adding this as an OpenTofu feature, if we come up with a good way to do so.

[RafPe]

@cube2222 maybe I do not see the whole technical picture - but would it for example be possible to split it in two working items ?

1) to make it accessible as just static array so we can index it into via a key 2) Think of making it super dynamic with variable interpolations

Feels like doing the 1 would be less of an impact from a major change as you hihlighted dependencies on that

[cam72cam]

I think that makes sense, as #2 would likely relate to #1042

[ImIOImI]

Scenario 2 feels like it'll cover most of the cases I encounter in the wild... but I say without any hesitation that no matter how unergonomic the solution is, I'll figure it out and use it and be happy to have it.

That being said, would it be possible to configure and pass a provider like:

locals {
  my_object = [
    { provider: providers.aws["dev"], "foo_value": "foo",    "bar_value": "bar"    },
    { provider: providers.aws["dev"], "foo_value": "apple",    "bar_value": "pear"    },
    { provider: providers.aws["prd"], "foo_value": "foo",    "bar_value": "bar"    },
    ]
  aws_accounts = {
    dev = {
      aws_account_id": "123456789012"
     },
    prd = { 
      "aws_account_id": "987654321098"
    }
}

provider "aws" {
  for_each = local.aws_accounts
  alias = each.key

  assume_role {
    role_arn    = "arn:aws:iam::${each.value.aws_account_id}:role/TerraformAccessRole"
  }
}

resource "aws_resource" "x1" {
  for_each = local.my_object

  provider = each.value.provider
}

[cam72cam]

Something worth noting, I think the provider. prefix is probably required as part of this work (at least for expressions). Otherwise your provider names are top level identifiers and can conflict with other things like "local"

[nitrocode]

In this case this helps for people in the mean time.

I've used workspaces with a specific tfvars containing a variable for the region as a workaround for creating the same resources in multiple regions or multiple accounts.

main.tf

variable region {}
variable account {}

provider aws {
  region = var.region

  assume_role {
    role_arn = "arn:aws:iam::${module.account_map.accounts[var.account]}:role/cicd"
  }
}

uw2-dev.tfvars

region = "us-west-2"
account = "dev"

Commands

terraform workspace select uw2-dev
terraform init -var-file uw2-dev.tfvars

[ImIOImI]

I've used workspaces with a specific tfvars containing a variable for the region as a workaround for creating the same resources in multiple regions or multiple accounts. ...

As of now, I think you're right about workspaces being the best answer when needing dynamic providers. I've done something similar like the following in Azure:

locals {
  cntxts = {
    defaults = {
      subscription_id         = "00000000-0000-0000-0000-000000000000"
      dynamic_subscription_id = "12345678-0000-0000-0000-000000000000"
      tenant_id               = "00000000-0000-0000-0000-000000000000"
    }
    dev = {
      dynamic_subscription_id = "abcdefgh-0000-0000-0000-000000000000"
    }
    infra = {
      dynamic_subscription_id = "abc123hi-0000-0000-0000-000000000000"
    }
    stg = {
      dynamic_subscription_id = "foobar12-0000-0000-0000-000000000000"
    }
    prd = {
      dynamic_subscription_id = "RunningO-utOf-Gene-ricG-UIDideas1234"
    }
  }

  contexts                = module.contexts.merged
  default_subscription_id = local.contexts.subscription_id
  dynamic_subscription_id = local.contexts.dynamic_subscription_id
  tenant_id               = local.contexts.tenant_id

  # where workspaces are like (dev, infra, stg, prd)
  context = terraform.workspace
}

module "contexts" {
  source = "Invicton-Labs/deepmerge/null"
  maps   = [
    local.cntxts.defaults,
    local.cntxts[local.context],
  ]
}

provider "azurerm" {
  alias = "default"

  subscription_id = local.default_subscription_id
  tenant_id       = local.tenant_id
}

provider "azurerm" {
  alias = "dynamic"

  subscription_id = local.dynamic_subscription_id
  tenant_id       = local.tenant_id
}

However, this often forces you into architecture patterns that could be more simply solved with a loop of providers

[vierkean]

Hello, since HashiCorp still supported this function in terraform version 0.11 and no longer does since 0.12, I was looking for an alternative to terraform and came across tofu. Unfortunately, the current version of tofu does not support it either. I hope tofu will support it soon and I don't have to rebuild the organization in terraform to use a current version.

provider "aws" { alias = "${var.AccountName}" region = "${var.region}" assume_role { role_arn = "arn:aws:iam::${aws_organizations_account.account.id}:role/someRole" } }

Kind Regards

[cam72cam]

This has been discussed and accepted as defined in https://github.com/opentofu/opentofu/blob/main/rfc/20240513-static-evaluation-providers.md. We are tentatively aiming for including this in OpenTofu 1.9.0 as an extension to the existing static evaluation work that is present in 1.8.0.

[tmccombs]

Something I would like to see, maybe as a future extension of this, would be to have a way to pass a dynamic number of providers to a module. So that a module would be able to use a for_each over a set of providers passed in to the module.

I'm not sure what the syntax for that would look like though.