Single Place for Version Constraints

nuryupin-kr commented 3 months ago

Terraform Version

1.9.0

Use Cases

We have around 20 Terraform modules that we host in a Github repository. Besides those 20 parent modules, there are many child modules that incapsulate reusable logic and act like functions in the same repo, they are called across multiple parent modules. We use local syntax to reference child modules inside of the parent modules. Note - only parent level modules are exposed to the outside, the child level modules are ONLY meant to be used internally within the repo. All of these modules are maintained by one team and we like having them in one place and version them as a single unit for maintainability. These modules get executed EXCLUSIVELY in CI/CD pipeline. Also, almost all of these modules have same provider requirements.

Problem is that every parent module has its own terraform and provider version constraints blocks, so we have to maintain 20 versions.tf files per each module, that have identical constraints because we would like to keep same provider and terraform versions requirements across the board. Instead, we would like to maintain min versions in one place, shared across all parent modules as the 'default'. Only when there is a specific module that has a unique constraint, then we would want to override the 'default' and define those unique constraint on the module level.

Is there a way to achieve this behavior?

Attempted Solutions

We use Terragrunt to pass configurations to terraform. It can also generate terraform files with constraints, but that is a bit of a hack.

Proposal

CLI level configuration where one could specify global level constraints.
Ability to reference a default version within the required_providers and required_version blocks from a configuration file or an environment variable. Make it overridable by supplying an inline value.

OR something like this: ./common-provider-constraints-template.tf:

terraform "common" { #<-- named terraform block to reference in root modules.
  required_version = ">= 1.5.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.111.0"
    }
  }
}

./specific-provider-constraints-template.tf:

terraform "specific_to_some_modules" { #<-- named terraform block to reference in root modules.
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      version = ">= 3.111.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

./provider-configuration-template.tf:

provider "aws " {
  features {}
  # ... some complex configuration ...
}
provider "azurerm" {
  features {}
  alias = "nonprod"
  skip_provider_registration = true
  storage_use_azuread = true
}
provider "azurerm" {
  features {}
  alias = "prod"
  subscription_id = "some-other-sub-id"
  skip_provider_registration = true
  storage_use_azuread = true
}

./modules/root-module-1/versions.tf:

terraform {
  include {
    path    = "../../" # <-- Path for where to look for terraform template code that will get "merged in"
    names = [ "common" ] #<-- reference terraform blocks named "common"
  }

  required_providers {
    azurerm = {
      version = "~> 3.100" # <-- Overrides version from included template
    }
    myprovider = {
      source  = "mynamespace/myprovider"
      version = ">=0.11.2" # <-- Module specific provider
    }
  }
}

provider "azurerm" {
   template = {
    path  = "../../"
    alias  = "prod"
  }
}

./modules/root-module-2/versions.tf:

terraform {
  include {
    path    = "../../"
    # <-- ommiting 'names' parameter will bring in both "common" and "specific_to_some_modules" named terraform blocks.
  }

  include {
    path  = "github.com/myorg/myrepo" # <-- Include some remote configuration.
    names = [ "some-tf-block" ]
  }
}

provider "azurerm" {
   template = {
    path  = "../../"
    alias  = "dev"
  }
}
provider "aws" {
   template = {
    path  = "../../"
  }
}

References

No response

apparentlymart commented 3 months ago

Thanks for sharing this feedback, @nuryupin-kr!

This is in my personal blog and so is just my own idea not necessarily shared by HashiCorp or the Terraform team, but just in case its useful to someone thinking about this in future I described one possible approach to this in my article about dynamic module source addresses from earlier this year.

My suggestion has similar characteristics to what you proposed but a few key differences:

The centralized version constraints live in a separate file from the root Terraform module's .tf files. That file can be placed in the same directory as the root module, or in some ancestor directory above all of your root modules (to automatically share it), or specified on the command line to terraform init for situations where a more complicated selection rule is required.
I framed it as a dependency override file because I also imagined it being useful for situations where someone needs to work temporarily with a different provider or module version to test something, without modifying the main module they're working with. However, if your modules don't specify any version constraints of their own then there would be nothing to override, and so the separate file would effectively be a central source of truth for which provider and module versions to use.

apparentlymart commented 3 months ago

With today's Terraform you can simulate the effect of centralized provider version selections using the following workaround:

Remove the version arguments from all of your current modules so that they leave the provider selections totally unconstrained.
Write a new module that has nothing in it except a required_providers block specifying the providers you use and the exact versions of them you want.
Change all of your root modules to call that new versions-only module using a module block. Since the entire tree of modules under a particular root must agree on a single provider version to use, the constraints in the versions-only module will decide which version to use when you run terraform init.

Since the versions-only module doesn't contain any provider blocks or resource declarations, it won't affect how any other modules interact with those providers. It will only constrain which versions can possibly be selected.

This approach does admittedly have one minor quirk: if some of your configurations don't need all of the providers that you're constraining then the versions-only module will still cause all of them to be installed by terraform init anyway, because it is effectively claiming that the module can't work without those providers even though that isn't really true. However, for any provider that doesn't have any resources associated with it anywhere in the configuration, Terraform won't try to configure it and so it shouldn't affect Terraform's runtime behavior significantly.

nuryupin-kr commented 3 months ago

Thanks for quick reply @apparentlymart ! The approach with a module just for required_providers is really neat! I guess we could also overcome a shortcoming that you described where Terraform would still download providers that aren't necessarily needed by breaking out provider that is used only by some modules into another required_providers module that just for those few and then include them both into the root module. Hope that makes sense :)

nuryupin-kr commented 3 months ago

I've also updated my proposed example above to reflect those shortcomings in a potential feature

crw commented 3 months ago

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

nuryupin-kr commented 3 months ago

I did not realize this when originally created the issue, but in our use case we would also want the actual provider block to be "sharable" across root level modules. For example:

provider "azurerm" {
  features {}
  skip_provider_registration = true
  storage_use_azuread = true
}

the above configuration for azurerm provider should apply to all modules.

I've adjusted the original solution proposal yet again to include that requirement. It does look a bit cluttered, so any improvements feedback is much appreciated!

jnesta-lh commented 3 months ago

With today's Terraform you can simulate the effect of centralized provider version selections using the following workaround:

Remove the version arguments from all of your current modules so that they leave the provider selections totally unconstrained.

Write a new module that has nothing in it except a required_providers block specifying the providers you use and the exact versions of them you want.

Change all of your root modules to call that new versions-only module using a module block. Since the entire tree of modules under a particular root must agree on a single provider version to use, the constraints in the versions-only module will decide which version to use when you run terraform init.

Since the versions-only module doesn't contain any provider blocks or resource declarations, it won't affect how any other modules interact with those providers. It will only constrain which versions can possibly be selected.

This approach does admittedly have one minor quirk: if some of your configurations don't need all of the providers that you're constraining then the versions-only module will still cause all of them to be installed by terraform init anyway, because it is effectively claiming that the module can't work without those providers even though that isn't really true. However, for any provider that doesn't have any resources associated with it anywhere in the configuration, Terraform won't try to configure it and so it shouldn't affect Terraform's runtime behavior significantly.

@apparentlymart Thanks for the detailed explanation. Can you show an example of such a configuration? Specifically, I am stumbling on what to put inside of the required_providers block in the Terraform module that is invoking the upstream module. (In my case, azurerm.)

My naïve attempt was to do this:

monorepo/
└── modules/
    ├── foo
    ├── bar
    └── versions-only

And then in ./modules/foo/version.tf:

terraform {
  required_providers {
    azurerm = {
      source  = "../versions-only"
    }
  }
}

But you can't use a relative path in this context, so I'm kind of stumped.

hashicorp / terraform