[Question] How to share data/resource between module or make another module aware of it

Problem

First of all, thank you for this great software, adds many great functions on top of terrafrom.

I've already gotten used to it a bit and I'm getting along pretty well with it. Now I'm just faced with a problem. I would like networks deployed via nsxt and vms via vsphere and that via separate modules. So I would like to follow your path and separate as much as possible from each other. My problem now is that I want to use the networks created in NSXT in Vsphere, but I always get the following error message:

# cd vcf && terragrunt run-all plan
INFO[0000] The stack at <folder_path>/vcf will be processed in the following order for command plan:
Group 1
- Module <folder_path>/vcf/network

Group 2
- Module <folder_path>/vcf/vm

data.nsxt_policy_transport_zone.vlan_tz: Reading...
data.nsxt_policy_transport_zone.overlay_tz: Reading...
data.nsxt_policy_segment_security_profile.nsx_security_profile: Reading...
data.nsxt_policy_mac_discovery_profile.nsx_mac_discovery_profile: Reading...
data.nsxt_policy_transport_zone.overlay_tz: Read complete after 1s [id=<removed>]
data.nsxt_policy_segment_security_profile.nsx_security_profile: Read complete after 1s [id=<removed>]
data.nsxt_policy_mac_discovery_profile.nsx_mac_discovery_profile: Read complete after 1s [id=<removed>]
data.nsxt_policy_transport_zone.vlan_tz: Read complete after 1s [id=<removed>]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.nsxt_policy_segment_realization.tenant_networks["Test-mgmt"] will be read during apply
  # (config refers to values not yet known)
 <= data "nsxt_policy_segment_realization" "tenant_networks" {
      + id           = (known after apply)
      + network_name = (known after apply)
      + path         = (known after apply)
      + state        = (known after apply)
    }

  # nsxt_policy_segment.tenant_networks["500"] will be created
  + resource "nsxt_policy_segment" "tenant_networks" {
      + description         = "provisioned with Terraform"
      + display_name        = "Test-mgmt"
      + id                  = (known after apply)
      + nsx_id              = (known after apply)
      + overlay_id          = (known after apply)
      + path                = (known after apply)
      + replication_mode    = "MTEP"
      + revision            = (known after apply)
      + transport_zone_path = "/infra/sites/default/enforcement-points/default/transport-zones/<removed>"

      + discovery_profile {
          + binding_map_path           = (known after apply)
          + mac_discovery_profile_path = "/infra/mac-discovery-profiles/<removed>"
          + revision                   = (known after apply)
        }

      + security_profile {
          + binding_map_path      = (known after apply)
          + revision              = (known after apply)
          + security_profile_path = "/infra/segment-security-profiles/<removed>"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

data.vsphere_datacenter.datacenter: Reading...
data.vsphere_datacenter.datacenter: Read complete after 0s [id=datacenter-3]
data.vsphere_network.networks["500"]: Reading...
data.vsphere_compute_cluster.cluster: Reading...
data.vsphere_datastore.datastore: Reading...
data.vsphere_resource_pool.pool: Reading...
data.vsphere_virtual_machine.vm_templates["<removed>"]: Reading...
data.vsphere_datastore.datastore: Read complete after 0s [id=<removed>]
data.vsphere_compute_cluster.cluster: Read complete after 0s [id=<removed>]
data.vsphere_resource_pool.pool: Read complete after 0s [id=<removed>]
data.vsphere_virtual_machine.vm_templates["<removed>"]: Read complete after 0s [id=<removed>]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform planned the following actions, but then encountered a problem:

  # vsphere_folder.vcenter_vm_folder will be created
  + resource "vsphere_folder" "vcenter_vm_folder" {
      + datacenter_id = "<removed>"
      + id            = (known after apply)
      + path          = "Test-Scenario"
      + type          = "vm"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: error fetching network: network 'Test-mgmt' not found
│ 
│   with data.vsphere_network.networks["500"],
│   on main.tf line 33, in data "vsphere_network" "networks":
│   33: data "vsphere_network" "networks" {
│ 
╵
time=2023-12-11T14:43:30+01:00 level=error msg=terraform invocation failed in <folder_path>/vcf/vm/.terragrunt-cache/r8sU6Laf2seI_7UNiTg9yI56kt4/RTbWmgnVWRGru1Mk1RZVCPf6Gkg/vsphere prefix=[<folder_path>/vcf/vm] 
ERRO[0004] Module <folder_path>/vcf/vm has finished with an error: 1 error occurred:
    * [<folder_path>/vcf/vm/.terragrunt-cache/r8sU6Laf2seI_7UNiTg9yI56kt4/RTbWmgnVWRGru1Mk1RZVCPf6Gkg/vsphere] exit status 1
  prefix=[<folder_path>/vcf/vm] 
INFO[0004] ╷
│ Error: error fetching network: network 'Test-mgmt' not found
│ 
│   with data.vsphere_network.networks["500"],
│   on main.tf line 33, in data "vsphere_network" "networks":
│   33: data "vsphere_network" "networks" {
│ 
╵
time=2023-12-11T14:43:30+01:00 level=error msg=terraform invocation failed in <folder_path>/vcf/vm/.terragrunt-cache/r8sU6Laf2seI_7UNiTg9yI56kt4/RTbWmgnVWRGru1Mk1RZVCPf6Gkg/vsphere prefix=[<folder_path>/vcf/vm]  
ERRO[0004] 1 error occurred:
    * [<folder_path>/vcf/vm/.terragrunt-cache/r8sU6Laf2seI_7UNiTg9yI56kt4/RTbWmgnVWRGru1Mk1RZVCPf6Gkg/vsphere] exit status 1

I suspect the problem arises because the network does not yet exist and are create in the plan phase of nsxt but vsphere is not aware of it.

Short note: this works in a single Terraform file.

Ideas for a solution would be to combine it in a module, but that wouldn't be very clean. Another idea would be to use the same remote_state file, but as far as I know that doesn't work.

Another idea is to pass on the network name as in the example of nsxt create networks for vsphere to vsphere. It just doesn't do anything for me, because in the planning phase the output is empty. I would rather read the network name directly from the var file and use it in vpshere

output "network" {
  value       = { for network in var.networks : network.name => data.nsxt_policy_segment_realization.tenant_networks[network.name].network_name }
  depends_on  = [data.nsxt_policy_segment_realization.tenant_networks]
}

I hope you have a few ideas on how we can solve the problem. If any information is missing, please just say.

I then attached the complete test_config.

Note: I removed all specific information and inserted a placeholder

Workflow

create all folder and files
adapt vars
terragrunt run-all init
terragrunt run-all plan

Config

Folder structure

Main -> terragrunt.hcl -> env.json -> ci_cd.json -> vcf --> terragrunt.hcl --> env_vcf.json --> network ---> terragrunt.hcl --> vm ---> terragrunt.hcl ->module --> network ---> main.tf ---> var.tf ---> provider.tf --> vm ---> main.tf ---> var.tf ---> provider.tf

Main/terragrunt.hcl

Main/env.json

{
    "tenant" : {
        "name" : "test"
    },
    "tenant_networks" : [    
        {
            "type" : "unmanaged"  ,
            "name" : "Test-mgmt",
            "id"   : 500    
        }
    ],
    "tenant_vms" : [    
        {
            "id"          : 8001001,
            "name"        : "Test-node1",
            "description" : "VM provisioned with Terraform",
            "type"        : "TestVM",
            "config"    : {
                "template_name" : "<vm_template>"
            },
            "interfaces" : [  
                {
                    "id": "0",
                    "network_id": 500,
                    "use_static_mac" : true,
                    "mac_address"   : "<mac_address>",
                    "ipv4_address"   : "<ip_address>",
                    "name": "mgmt"
                }   
            ]
        }
    ]
}

Main/ci_cd.json

{
    "tf_state": {
        "address": "<TF_ADDRESS>",
        "username": "<TF_USERNAME>",
        "password": "<TF_PASSWORD>",
        "state_base_name": "<TF_STATE_BASE_NAME>_<ENVIROMENT>"
    }
}

Main/vcf/terragrunt.hcl

locals {
  config = jsondecode(file(find_in_parent_folders("env.json")))
  config_vcf = jsondecode(file("env_vcf.json"))
  ci_cd  = jsondecode(file(find_in_parent_folders("ci_cd.json")))
}

generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
terraform {
  backend "http" {
    address         = "${local.ci_cd.tf_state.address}/${local.ci_cd.tf_state.state_base_name}-${basename(get_terragrunt_dir())}"
    lock_address    = "${local.ci_cd.tf_state.address}/${local.ci_cd.tf_state.state_base_name}-${basename(get_terragrunt_dir())}/lock"
    unlock_address  = "${local.ci_cd.tf_state.address}/${local.ci_cd.tf_state.state_base_name}-${basename(get_terragrunt_dir())}/lock"
    username        = "${local.ci_cd.tf_state.username}"
    password        = "${local.ci_cd.tf_state.password}"
    lock_method     = "POST"
    unlock_method   = "DELETE"
    retry_wait_min  = 10
    retry_max       = 5
  }
}
EOF
}

Main/vcf/env_vcf.json

{
    "nsx": {
        "ip_address" : "<nsx_address>",
        "username" : "<nsx_username>",
        "password" : "<nsx_password>",
        "transport_zone" : {
            "overlay" : "<nsx_overlay>",
            "vlan" : "<nsx_vlan>"           
        },
        "mac_discovery_profile" : "<nsx_mac_discovery_profile>",
        "security_profile" : "<nsx_security_profile>"
    },
    "vsphere" : {
        "ip_address" : "<vshpere_address>",
        "username" : "<vshpere_username>",
        "password" : "<vshpere_password>",
        "datacenter" : "<vshpere_datacenter>",
        "datastore" : "<vshpere_datastore>",
        "cluster" : "<vshpere_cluster>",
        "resource_pool" : <vshpere_resource_pool>",
        "vm_folder" : "<vshpere_vm_folder>"
    }
}

Main/vcf/network/terragrunt.hcl

include "root" {
  path           = find_in_parent_folders()
  merge_strategy = "deep"
  expose         = true
}

locals {
  config            = jsondecode(file(find_in_parent_folders("env.json")))
  config_vcf        = jsondecode(file(find_in_parent_folders("env_vcf.json")))
  ci_cd             = jsondecode(file(find_in_parent_folders("ci_cd.json")))
}

terraform {
  source = "${dirname(find_in_parent_folders())}/../modules//network"
}

inputs = {
  nsx        = local.config_vcf.nsx
  tenant     = local.config.tenant
  networks   = local.config.tenant_networks
}

generate "provider_conf" {
  path = "provider_conf.tf"

  if_exists = "overwrite"

  contents = <<EOF
provider "nsxt" {
    host                     = var.nsx.ip_address
    username                 = var.nsx.username
    password                 = var.nsx.password
    allow_unverified_ssl     = true
    max_retries              = 10
    retry_min_delay          = 500
    retry_max_delay          = 5000
    retry_on_status_codes    = [429]
}

EOF
}

Main/vcf/vm/terragrunt.hcl

include "root" {
  path           = find_in_parent_folders()
  merge_strategy = "deep"
  expose         = true
}

locals {
  config            = jsondecode(file(find_in_parent_folders("env.json")))
  config_vcf        = jsondecode(file(find_in_parent_folders("env_vcf.json")))
  ci_cd             = jsondecode(file(find_in_parent_folders("ci_cd.json")))
}

terraform {
  source = "${dirname(find_in_parent_folders())}/../modules//vsphere"
}

dependency "network" {
  config_path  = "../network"
  skip_outputs = true
}

inputs = {
  vsphere             = local.config_vcf.vsphere
  tenant              = local.config.tenant
  networks            = local.config.tenant_networks
  vms                 = local.config.tenant_vms
  vms_config          = local.config.tenant_vms_config
}

generate "provider_conf" {
  path = "provider_conf.tf"

  if_exists = "overwrite"

  contents = <<EOF
provider "vsphere" {
    user                 = var.vsphere.username
    password             = var.vsphere.password
    vsphere_server       = var.vsphere.ip_address
    allow_unverified_ssl = true
}

EOF
}

Main/module/network/main.tf

data "nsxt_policy_transport_zone" "overlay_tz" {
    display_name = var.nsx.transport_zone.overlay
}

data "nsxt_policy_transport_zone" "vlan_tz" {
    display_name = var.nsx.transport_zone.vlan
}

data "nsxt_policy_mac_discovery_profile" "nsx_mac_discovery_profile" {
    display_name = var.nsx.mac_discovery_profile
} 

data "nsxt_policy_segment_security_profile" "nsx_security_profile" {
  display_name = var.nsx.security_profile
}

resource "nsxt_policy_segment" "tenant_networks" { 
    for_each = { for network in var.networks : network.id => network}

  display_name        = "${each.value.name}"
  description         = "provisioned with Terraform"
  transport_zone_path = "${data.nsxt_policy_transport_zone.overlay_tz.path}"

  discovery_profile {
    mac_discovery_profile_path = "${data.nsxt_policy_mac_discovery_profile.nsx_mac_discovery_profile.path}"
  }

  security_profile {
    security_profile_path   = "${data.nsxt_policy_segment_security_profile.nsx_security_profile.path}"
  }  
}

data "nsxt_policy_segment_realization" "tenant_networks" {
    for_each = { for network in var.networks : network.name => network}
    path = resource.nsxt_policy_segment.tenant_networks["${each.value.id}"].path
  depends_on = [resource.nsxt_policy_segment.tenant_networks]
}

Main/module/network/var.tf

variable "nsx" {
    type = object({
        ip_address = string
        username = string
        password = string
        transport_zone = object({
            overlay = string
            vlan = string
            edge = optional(string)
        })
        mac_discovery_profile = string
        security_profile = string       
    })
    sensitive = true
}
variable "tenant" {
    type = object({
        name = string
    })
}

variable "networks" {
    type = list(object({
        type = string
        name = string
        id = number     
    }))
}

Main/module/network/provider.tf

terraform {
    required_providers {
        nsxt = {
            source = "vmware/nsxt"
            version = "3.4.0"
            configuration_aliases = [
                nsxt,
            ]
        }
    }
}

Main/module/vm/main.tf

data "vsphere_datacenter" "datacenter" {
  name = var.vsphere.datacenter
}

data "vsphere_compute_cluster" "cluster" {
  name          = var.vsphere.cluster
  datacenter_id = "${data.vsphere_datacenter.datacenter.id}"
}

data "vsphere_datastore" "datastore" {
  name          = var.vsphere.datastore
  datacenter_id = "${data.vsphere_datacenter.datacenter.id}"
}

data "vsphere_resource_pool" "pool" {
  name          = var.vsphere.resource_pool
  datacenter_id = "${data.vsphere_datacenter.datacenter.id}"
}

resource "vsphere_folder" "vcenter_vm_folder" {
  path          = "${var.tenant.name}-Scenario"
  type          = "vm"
  datacenter_id = "${data.vsphere_datacenter.datacenter.id}"
}

data "vsphere_virtual_machine" "vm_templates" {
  for_each        = toset(distinct(var.vms.*.config.template_name))
  name          = "${each.value}"
  datacenter_id = "${data.vsphere_datacenter.datacenter.id}"
} 

data "vsphere_network" "networks" {
  for_each      = { for network in var.networks : network.id => network}

  name          = "${each.value.name}"
  datacenter_id = "${data.vsphere_datacenter.datacenter.id}"
}

resource "vsphere_virtual_machine" "create_vms" {
  for_each                    = { for vm in var.vms : vm.name => vm}

  name                        = "${var.tenant.name}-${each.key}"
  resource_pool_id            = "${data.vsphere_resource_pool.pool.id}"
  datastore_id                = "${data.vsphere_datastore.datastore.id}"
  folder                      = "${resource.vsphere_folder.vcenter_vm_folder.path}"
  firmware                    = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].firmware}"
  hardware_version            = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].hardware_version}"
  num_cpus                    = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].num_cpus}"
  memory                      = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].memory}"
  wait_for_guest_net_timeout  = -1
  guest_id                    = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].guest_id}"
  scsi_type                   = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].scsi_type}"

  dynamic "network_interface" {
    for_each          = { for network in each.value.interfaces : network.id => network}

    content {
      network_id      = data.vsphere_network.networks[tostring(network_interface.value.network_id)].id
      adapter_type    = data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].network_interfaces[0].adapter_type
      use_static_mac  = network_interface.value.mac_address != null ? "true" : "false"
      mac_address     = network_interface.value.mac_address != null ? network_interface.value.mac_address : ""
    }    
  }

  dynamic "disk" {
    for_each            = data.vsphere_virtual_machine.vm_templates[each.value.config.template_name].disks

    content {
      label             = disk.value.label
      size              = disk.value.size
      thin_provisioned  = disk.value.thin_provisioned
    }
  }

  clone {
    linked_clone  = false
    template_uuid = "${data.vsphere_virtual_machine.vm_templates["${each.value.config.template_name}"].id}"
  }
}

Main/module/vm/var.tf

variable "vsphere" {
    type = object({
        ip_address = string
        username   = string
        password   = string 
        datacenter = string 
        datastore  = string
        cluster    = string
        resource_pool = string
        vm_folder  = string
    })
    sensitive = true
}

variable "tenant" {
    type = object({
        name = string
    })
}

variable "vms" {
    type = list(object({
        id          = number
        name        = string
        description = string
        type        = string
        config      = object({
                template_name = string
        })
        interfaces  = list(object({
            id              = number
            name            = string
            network_id      = number
            ipv4_address    = optional(string)
            mac_address     = optional(string)
        })) 
    }))
}

variable "networks" {
    type = list(object({
        type = string
        name = string
        id   = number       
    }))
}

Main/module/vm/provider.tf

terraform {
  required_version = ">= 0.13"
  required_providers {
    vsphere = {
      source = "hashicorp/vsphere"
      version = "2.5.1"
      configuration_aliases = [
        vsphere,
      ]
    }
  }
}

Version

system: Manjaro
terragrunt (terragrunt -v): terragrunt version latest
terraform: v1.4.2

gruntwork-io / terragrunt