Use a for_each loop for a source block

[dustindortch]

When working with a provider, it would great to be able to use a for_each loop in the source block for a golden image pipeline where you have different source images from the same provider that you want to adhere to your golden image standards.

For instance, with Azure, I'd like a golden image for Windows Server 2019, but I need a regular server and a SQL Server instance:

• 2019-datacenter
• sql2019-ws2019

so, for the source:

source "azurerm-arm" "golden-image" {
  for_each = var.image

  # Common elements
  tenant_id       = var.TENANT_ID
  subscription_id = var.SUBSCRIPTION_ID
  client_id       = var.CLIENT_ID
  client_secret   = var.client_secret

  managed_image_resource_group_name  = var.RESOURCE_GROUP_NAME
  managed_image_name                 = "${var.PREFIX}${each.key}_${formatdate("YYYY-MM-DD_hh-mm-ss",timestamp())}"
  managed_image_storage_account_type = "Premium_LRS"

  location = var.LOCATION
  vm_size  = var.VM_SIZE

  # Looped elements
  os_type = each.value["type"]

  image_publisher = each.value["publisher"]
  image_offer     = each.value["offer"]
  image_sku       = each.value["sku"]

  communicator   = each.value["communicator"]
  winrm_use_ssl  = true
  winrm_insecure = true
  winrm_timeout  = "5m"
  winrm_username = "packer"
}

This would be made with a variable definition, as such:

variable "IMAGE" {
  type = map(object({
    communicator = string
    publisher    = string
    offer        = string
    sku          = string
    type         = string
  }))
}

Then, the variables could be build out:

IMAGE = {
  "WindowsServer2019" = {
    "communicator" = "winrm"
    "publisher"    = "MicrosoftWindowsServer"
    "offer"        = "WindowsServer"
    "sku"          = "2019-Datacenter"
    "type"         = "Windows"
  },
  "SQLServer2019" = {
    "communicator" = "winrm"
    "publisher"    = "microsoftsqlserver"
    "offer"        = "sql2019-ws2019"
    "sku"          = "standard"
    "type"         = "Windows"
  }
}

Finally, in the build:

build {
  source = sources.azure-arm.golden-image

  # Build stuff
}

Then the various golden images could be built in parallel without having to define a separate source blockfor each one. Alternatively, you could have multiple source blocks if you have multiple platforms, but you could have multiple source images for each platform.

[sylviamoss]

Hey there, thanks for reaching out!! If I understood correctly your intentions, you can achieve that using dynamic source block within your build. I rewrote your template with what I'm suggesting.

variables {
  images = {
    windows_server_2019 = {
      communicator = "winrm"
      publisher    = "MicrosoftWindowsServer"
      offer        = "WindowsServer"
      sku          = "2019-Datacenter"
      type         = "Windows"
    }
    sql_server_2019 = {
      communicator = "winrm"
      publisher    = "microsoftsqlserver"
      offer        = "sql2019-ws2019"
      sku          = "standard"
      type         = "Windows"
    }
  }
}

source "azure-arm" "golden-image" {
  # Common elements
  tenant_id       = var.TENANT_ID
  subscription_id = var.SUBSCRIPTION_ID
  client_id       = var.CLIENT_ID
  client_secret   = var.client_secret

  managed_image_resource_group_name  = var.RESOURCE_GROUP_NAME
  managed_image_name                 = "${var.PREFIX}${each.key}_${formatdate("YYYY-MM-DD_hh-mm-ss", timestamp())}"
  managed_image_storage_account_type = "Premium_LRS"

  location = var.LOCATION
  vm_size  = var.VM_SIZE

  winrm_use_ssl  = true
  winrm_insecure = true
  winrm_timeout  = "5m"
  winrm_username = "packer"
}

build {
  dynamic "source" {
    for_each = var.images
    labels   = ["azure-arm.golden-image"]
    content {
      name = source.key

      image_publisher = source.value.publisher
      image_offer     = source.value.offer
      image_sku       = source.value.sku
      os_type         = source.value.type
      communicator    = source.value.communicator
    }
  }

  # Build stuff
}

Let me know if that works for you. 😄

[dustindortch]

What about a case where I have multiple platforms and I want to loop through a source for one platform and then a source for another platform? I think the for_each logic would be an improvement.

[azr]

In this case — each builder (or platform) — has a very specific set of fields, behaviours and settings one can set, and I think it is better to define them in their own source block. So if you wanted to run the same provisioning steps on a different source, you'd have to do another dynamic "source" call, for example:

  dynamic "source" {
    for_each = var.azure_images
    labels   = ["azure-arm.golden-image"]

    content {
      name = source.key

      image_publisher = source.value.publisher
    // ...
  }

  dynamic "source" {
    for_each = var.gcp_images
    labels   = ["googlecompute.golden-image"]

    content {
      name = source.key

      account_file = source.value.account_file
    // ...
  }

[kevincormier-toast]

+1 to this request.

For a bit more context, imagine you have a matrix of sources and builds. For example, you have 3 different AWS accounts and 3 different builds and you want each build to run in each AWS account (total of 9 images created). Assume your provisioner is going to take the account & build as input and configure each of the 9 images differently so you can't just build a single image and share it between accounts.

Only allowing a dynamic source inside of the builder means anything that has to adjust between accounts (ex. security groups) has to be duplicated across all of the builds. This is currently leading us to lots of copy/paste code. To get around this, we're instead generating our hcl2 with other scripts which is also suboptimal.

Consider this working code as an example:

source "null" "first-example" {
  communicator = "none"
}

build {
  name = "roles"

  dynamic source {
    for_each = ["consul", "nomad", "vault"]
    labels = ["null.first-example"]

    content {
      name = source.value
    }
  }

  provisioner "shell-local" {
    inline = ["echo hello ${source.name} and ${source.type}"]
  }
}

build {
  name = "roles"

  dynamic source {
    for_each = ["consul", "nomad", "vault"]
    labels = ["null.first-example"]

    content {
      name = source.value
    }
  }

  provisioner "shell-local" {
    inline = ["echo goodbye ${source.name} and ${source.type}"]
  }
}

If dynamic top level sources were possible, I think this would be much easier to understand and to maintain.

dynamic "source" {
  for_each = ["consul", "nomad", "vault"]
  content {
    type = "null"
    name = source.value
    communicator = "none"
  }
}

build {
  name = "roles"

  sources = [
    "null.consul",
    "null.nomad",
    "null.vault",
  ]

  provisioner "shell-local" {
    inline = ["echo hello ${source.name} and ${source.type}"]
  }
}

build {
  name = "roles"

  sources = [
    "null.consul",
    "null.nomad",
    "null.vault",
  ]

  provisioner "shell-local" {
    inline = ["echo goodbye ${source.name} and ${source.type}"]
  }
}

[lmayorga1980]

This can also apply for provisioners.

[jboero]

So if you wanted to run the same provisioning steps on a different source, you'd have to do another dynamic "source" call, for example:

Thank you so much for this example. I spent hours trying to get the right for_each and dynamic source blocks to cross-build a container image for multiple architectures. This isn't nearly as intuitive as it should be, and it seems Packer is missing the toset function also? This example will cross-build docker containers for a source container image across x86_64, aarch64, and riscv64 all in a single block. Super helpful.

build {
  dynamic "source" {
    for_each = {arch="x86_64",arch="aarch64",arch="riscv64"}
    labels = ["source.docker.base_ubuntu"]

    content {
      name = source.value
      platform = "linux/${source.value}"
    }
  }
}

[huyz]

I would think this is a very common need. Some examples in the documentation would be very helpful.

[huyz]

@jboero Are you sure that "set" syntax works?

for_each = {arch="x86_64",arch="aarch64",arch="riscv64"}

I had to do

for_each = ["x86_64", "aarch64", "riscv64"]