Support Provider Differences in Modules

[NYRangers30]

Terraform Version

Terraform v1.5.1

Use Cases

We have a library of common modules that are used by hundreds of deployments. We are also in a regulated environment, so change control is key, and therefore we pin all of our deployments to a specific version of Terraform and Providers. We run into a problem often where we need to expose a new property, or a new resource type, but in order to do so we need to force every deployment to upgrade providers, whether they need the new feature or not, otherwise their pipelines will fail.

The request is for terraform to be able to support differences in provider versions to better support sharing of common modules.

Attempted Solutions

We tried doing this with a dynamic block, but it seems like the property is still sent to the provider even if the content is null.

variable "vnet_encryption_mode" {
  description = "Setting this flag enables VNET Encryption. Allowed values are AllowUnencrypted or DropUnencrypted."
  type        = string
  default     = null
}

resource "azurerm_virtual_network" "this" {
  address_space       = 10.0.0.0/24
  location            = "eastus2"
  name                = "myvnet"
  resource_group_name = "my-rg"
  dynamic "encryption" {
    for_each = toset([var.vnet_encryption_mode])
    content = each.key == null ? null : {enforcement = each.key}
  }

  tags = var.tags
}

That resulted in this:

│ Error: Unsupported block type
│ 
│   on .terraform/modules/common/network/main.tf line 198, in resource "azurerm_virtual_network" "this":
│  198:   dynamic "encryption" {
│ 
│ Blocks of type "encryption" are not expected here.

Proposal

This would need to work for full resource types, and also properties within resources.

For resources, I would propose introducing an "init_condition", where if the init condition isn't met, that resource isn't even exposed to the provider.

resource "azurerm_custom_ip_prefix" "this" {
  name                = "example-CustomIPPrefix"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  cidr  = "1.2.3.4/22"
  zones = ["1", "2", "3"]

  commissioning_enabled = true

  roa_validity_end_date         = "2099-12-12"
  wan_validation_signed_message = "signed message for WAN validation"

  tags = {
    env = "test"
  }
  lifecycle {
    init_condition {
      condition = azurerm.default.version >= 3.68.0
    }
  }

  tags = var.tags
}

resource "azurerm_virtual_network" "this" {
  address_space       = 10.0.0.0/24
  location            = "eastus2"
  name                = "myvnet"
  resource_group_name = "my-rg"
  encryption {
      enforcement = var.vnet_encryption_mode
  }
  lifecycle {
    init_condition {
      condition = azurerm.default.version < 3.68.0
    }
  }

  tags = var.tags
}

For property differences, introduce a new lifecycle option called ignore_property which would be evaluated before the plan. This would support some data structure where you could have multiple properties, and different conditions for each of them.

Example:

variable "vnet_encryption_mode" {
  description = "Setting this flag enables VNET Encryption. Allowed values are AllowUnencrypted or DropUnencrypted."
  type        = string
  default     = null
}

resource "azurerm_virtual_network" "this" {
  address_space       = 10.0.0.0/24
  location            = "eastus2"
  name                = "myvnet"
  resource_group_name = "my-rg"
  encryption {
    enforcement = var.vnet_encryption_mode
  }
  lifecycle {
    ignore_properties = {
      [
        property_path = "encryption"
        condition     = provider.azurerm.default.version >= 3.68.0
      }
    ]
  }

  tags = var.tags
}

In this example the encryption property would only be exposed to the provider if the provider version is greater or equal to 3.68.0.

References

No response

[jbardin]

Hi @NYRangers30,

Thanks for filing the request. I don't think this is something which is technically possible given the architecture of Terraform. Terraform requires the provider schema to decode the configuration, which means that the configuration must match the schema for each resource and having an errant encryption block in the body is going to fail to decode. We would also have to contend with other schema changes besides the addition of a block, like the addition and removal of block attributes, changes to attribute types, and the data upgrade paths for those schema changes.

The usual method for dealing with changes like this in module configuration is similar to how the software is versioned, you would have multiple stable branches which can diverge as necessary to account for the corresponding changes in the provider. In most cases it's usually better to track the provider directly, and require upgrading the provider and module in conjunction, since the improvements to the provider also come with associated bug fixes and required features.

[dustindortch]

@jbardin - I think this could be accomplished if we could have multiple instances of a provider defined in the required_providers block with different local names:

root/main.tf

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 2.0"
    }
    azurerm-3 = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

provider "azurerm-3" {
  features {}
}

Then @NYRangers30 can accomplish this, partially, by following semantic versioning when authoring modules. When adding a new feature, make the default behavior to not implement it. Create a variable and give the default a value of null, of if it is a list, object, or map, you could do empty values [] or {}, respectively. Then, within the module, simply assign the value if it is a scalar/primitive data type and if the default for the resource is to have as an "optional" argument, you've solved it. If you're using lists, objects, or maps, you can do count = var.empty_list != [] ? 1 : 0 to conditionally deploy the resource, or for_each = var.empty_map != {} ? [1] : [] if you're doing a dynamic block in the code.

As far as the versions... if the module implements a required_providers block that specifies the newer version of the module than you're ready to support in your root module (or other modules), create another instance of the module with a different local name and forcibly assign it to the module.

root/main.tf

module "module_name" {
  source = "./modules/terraform-azurerm-example"
  providers = {
    azurerm = azurerm-3
  }
}

resource "azurerm_resource_group" "example" {
  name     = "azurerm-2"
  location = "East US"
}

modules/terraform-azurerm-example/main.tf

terraform {
  required_version = "~> 1.8"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

resource "azurerm_resource_group" "example" {
  name     = "azurerm-3"
  location = "East US"
}

Currently, this doesn't work because two versions of the same provider cannot be defined within the required_providers block, even with different local names. But, allowing that would permit the code pattern described. Now, I think there are some good reasons to support this because there are large environments that may not be setup to ideally facilitate "just do best practices". It would be great and that should be a priority within any organization to move in that direction so that challenges can be worked around more readily, but having "not the best practices" available is a way that organizations to get through a challenge, temporarily, and emerge out the other side using "good enough practices", again. This is the way many "migrations" work.