Need ability to reuse provisioners and post-processors just like builders can be reused with source blocks.

[timblaktu]

An extension of what was asked in this issue, I have a single source block in hcl2 that I am reusing in multiple builders. These builders are mostly identical in what they're doing, and I'd like them to be built in parallel.

In my case, however, I have a non-trivial set of provisioners and post-processors that I want to run against all builders. Currently, it appears that to make this work I have to copy/paste these provisioners and post-processors in each build block. This makes managing these builder configurations becomes a bit of a maintenance nightmare moving forward.

I would like for packer to have some convention by which I can configure "run these provisioners, and these post-processors on ALL builders". I see this as entirely analogous to what was implemented for builders in source blocks in HCL.

Consider the following packer project directory, in which I am trying to build, provision, and post-process 2 identical virtualbox machines. The builders come from the same source, and I configure them to run the same provisioners and post-processors. The only difference between them is in the preseed file I'm using.

So, I'd like to be able to build this single source:

./source.pkr.hcl

source "virtualbox-iso" "debian10" {
  boot_command            = ["<esc><wait>", "blah", "blah"]
  guest_os_type           = "debian10_64Guest"
  iso_url                 = "https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-10.7.0-amd64-DVD-1.iso"
  iso_checksum            = "md5:270d5af2e1327a2a6ca6d24c468862b5"
  http_directory          = "http"
  shutdown_command        = "echo '****'|sudo -S shutdown -P now"
  ssh_password            = "****"
  ssh_username            = "****"
}

with the same set of provisioners and post-processors. Today, I have to copy/paste the provisioners/post-processors like this:

./packer.pkr.hcl

build {
  source "source.virtualbox-iso.debian10" {
    vm_name                 = "debian10-headless"
    preseed_filename        = "preseed.cfg"
  }
  provisioner "shell" { echo "A" }
  provisioner "shell" { echo "A" }
  .
  .
  post-processor "checksum" { ...  }
  post-processor "manifest" {  ...  }
}
}
build {
  source "source.virtualbox-iso.debian10" {
    vm_name                 = "debian10-gnome"
    preseed_filename        = "preseed-gnome.cfg"
  }
  provisioner "shell" { echo "A" }
  provisioner "shell" { echo "B" }
  .
  .
  post-processor "checksum" { ...  }
  post-processor "manifest" {  ...  }
}

and what I'd like to have is some HCL constructs, let's call them "provisioner-source" and "post-processor-source", that work like build sources. It might look like this:

./source.pkr.hcl

source "virtualbox-iso" "debian10" {
  boot_command            = ["<esc><wait>", "blah", "blah"]
  guest_os_type           = "debian10_64Guest"
  iso_url                 = "https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-10.7.0-amd64-DVD-1.iso"
  iso_checksum            = "md5:270d5af2e1327a2a6ca6d24c468862b5"
  http_directory          = "http"
  shutdown_command        = "echo '****'|sudo -S shutdown -P now"
  ssh_password            = "****"
  ssh_username            = "****"
}

provisioner-source "shell" "provisionA" { echo "A" }
provisioner-source "shell" "provisionB" { echo "B" }
  .
  .
post-processor-source "checksum" "mychecksum" { ...  }
post-processor-source "manifest" "mymanifest" {  ...  }

..and then refer to the provisioner-sources and post-processor-sources in the build template just like we do for build sources:

./packer.pkr.hcl

build {
  source "source.virtualbox-iso.debian10" {
    vm_name                 = "debian10-headless"
    preseed_filename        = "preseed.cfg" 
  }
  provisioner-source "shell" "provisionA" { <can customize provisioner sources here if desired> }
  provisioner-source "shell" "provisionB" { <can customize provisioner sources here if desired> }
  .
  .
  post-processor-source "checksum" { <can customize post-processor sources here if desired> }
  post-processor-source "manifest" { <can customize post-processor sources here if desired> }
}
}
build {
  source "source.virtualbox-iso.debian10" {
    vm_name                 = "debian10-gnome"
    preseed_filename        = "preseed-gnome.cfg"
  }
  provisioner-source "shell" "provisionA" { <can customize provisioner sources here if desired> }
  provisioner-source "shell" "provisionB" { <can customize provisioner sources here if desired> }
  .
  .
  post-processor-source "checksum" { <can customize post-processor sources here if desired> }
  post-processor-source "manifest" { <can customize post-processor sources here if desired> }
}

This is an improvement, bc when I make changes to the provisioning or post-processing in this use case, I only have to make the change in one place. It's pretty basic DRY principle stuff I'm basing this on.

I apologize in advance if I overlooked some existing way to accomplish provisioner and/or post-processor reuse. I reread https://github.com/hashicorp/packer/issues/9167 and the only thing I'm doing differently here is that my provisioners are the same, instead of different. I cannot glean from that issue, its PR, or the packer docs, how to do this. I can't even figure out the syntax to get packer to build in parallel without specifying multiple sources in a list. But when you do that, there's no way to parameterize each as far as I can see. So, I am left to assume it needs to be a new feature request.

[azr]

Hey @timblaktu thanks for opening,

It looks like to me you are trying to run the same set of provisioners (+pps) on two different sources. Have you tried the following syntax ? :

build {
  source "source.virtualbox-iso.debian10" {
    vm_name                 = "debian10-headless"
    preseed_filename        = "preseed.cfg"
  }
  source "source.virtualbox-iso.debian10" {
    vm_name                 = "debian10-gnome"
    preseed_filename        = "preseed-gnome.cfg"
  }
  provisioner "shell" { echo "A" }
  provisioner "shell" { echo "A" }
  .
  .
  post-processor "checksum" { ...  }
  post-processor "manifest" {  ...  }
}

If so, was it not enough ?

PS: I have though of adding a syntax just like you described but always ended up thinking that allowing multiple sources on one build block would be enough and DRY. Also you can use only excepts to run provisioning (+pp) steps only for certain sources.

[timblaktu]

Thanks. Yes, I have, and I believe they ran in series instead of parallel. Is that expected?

On Mon, Feb 8, 2021, 05:37 Adrien Delorme notifications@github.com wrote:

Hey @timblaktu https://github.com/timblaktu thanks for opening,

It looks like to me you are trying to run the same set of provisioners on two different instances. Have you tried the following syntax ? :

build { source "source.virtualbox-iso.debian10" { vm_name = "debian10-headless" preseed_filename = "preseed.cfg" } source "source.virtualbox-iso.debian10" { vm_name = "debian10-gnome" preseed_filename = "preseed-gnome.cfg" } provisioner "shell" { echo "A" } provisioner "shell" { echo "A" } . . post-processor "checksum" { ... } post-processor "manifest" { ... } }

If so, was it not enough ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/packer/issues/10583#issuecomment-775156236, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6IRAYJPN6YGEGMQVANJ4LS57SJPANCNFSM4XGY2XAQ .

[azr]

They should all run in parallel by default. 🤔

[SwampDragons]

I just confirmed that they run in parallel using this toy template:

source "null" "build1" {
  communicator = "none"
}
source "null" "build2" {
  communicator = "none"
}

build {
  sources = ["source.null.build1", "source.null.build2"]

  provisioner "shell-local"{
    inline = ["echo In the middle of Provisioning run. Sleeping..."]
  }
  provisioner "shell-local"{
    inline = ["sleep 10"]
  }
  provisioner "shell-local"{
    inline = ["echo slept 10s"]
  }
}

I can force them to build serially with packer build -parallel-builds=1 test_concurrent.pkr.hcl but the default is parallel.

[timblaktu]

It turns out that my packer builds were not being parallelized due to the fact that I was using -debug. :/ Sorry for the confusion.

Note that to be able to test @azr's proposed syntax, I had to work around another issue in this scenario, which I'll mention briefly: here, preseed_filename variable is NOT a builder parameter, it is my parameter, defined in my packer variables file. When I run the aforementioned packer build, which tries to invoke the same build source, specifying different values for preseed_filename in the build source block, I get related errors in both builders:

15:38:48  Error: Unsupported argument
15:38:48  
15:38:48    on packer.pkr.hcl line 7:
15:38:48    (source code not available)
15:38:48  
15:38:48  An argument named "preseed_filename" is not expected here.
15:38:48  
15:38:48  Error: Unsupported argument
15:38:48  
15:38:48    on packer.pkr.hcl line 18:
15:38:48    (source code not available)
15:38:48  
15:38:48  An argument named "preseed_filename" is not expected here.

I have asked the question over on Hashi Discuss about this, whether changing an arbitrary variable in a build source block is "supposed to work", but since I have some devs' attention here, I'd love to get your feedback on this. Thanks for your time.

p.p.s. @SwampDragons your example uses a sources list, which as you know is well documented on packer site as the way to "do parallel builds", but IMO it's not at all clear that the syntax @azr proposed above is "supposed to work." I feel a lot of new user time could be saved by clarifying this with a sentence or another example, or even just a comment in the example that alludes to the fact that the parallelization does not come from the use of a sources list [] format, it comes from specifying multiple sources one way or the other.

[azr]

Hey there, on the preseed_filename field issue; a builder has an expected set of fields you can set: for virtualbox-iso, the full list is here: https://www.packer.io/docs/builders/virtualbox/iso Because packer doesn't know what os you are running you have to be more explicit in how you want that pre seed file to run, for example by using a boot command.

Here's a guide that explains how to use a preseed file for ubuntu through boot commands: https://www.packer.io/guides/automatic-operating-system-installs/preseed_ubuntu

Also in the example folder you can find more more examples of this:

https://github.com/hashicorp/packer/blob/88c516b2d53fe1758c1e50b666348c534a3fd239/examples/hcl/linux/variables.16.04.pkr.hcl#L8-L32

https://github.com/hashicorp/packer/blob/88c516b2d53fe1758c1e50b666348c534a3fd239/examples/hcl/linux/build.ubuntu.pkr.hcl#L19-L24

[timblaktu]

@azr thanks, the question wasn't about "how to use a preseed file" - It was "why is packer failing to substitute my variable into a builder field definition in my build source (e.g. boot_command)?

The ubuntu example does evaluate a user variable (preseed_path) into the boot_command, but it only uses the default variable value, and is never set again by any source field (which is what I was doing, and failing).

I think the examples do show what I needed to see; they imply the answer is that "you cannot redefine or substitute arbitrary user-defined variables (var.) in build sources. You can only specify locals (locals.) and builder-specified fields in build sources." Can you confirm/deny that statement?

If so, shouldn't this be in the docs somewhere? I'm sorry if I overlooked this somewhere.

If not, I'm happy to submit another doc improvement PR if you don't have an immediate location in mind.

The repercussion of this limitation is that you cannot completely factor out common elements, e.g. in your example you have completely defined the boot_command string for ubuntu 16 and 18 separately. If you needed to change an aspect of the boot command for all ubuntu builds, you have to change it in 2 places. I'm simply trying to factor out single sources of truth like that in code by defining the common stuff once (really, once).

I suppose that since you can use locals in this way, you could overcome this limitation by, um, using locals. Breaking up boot_command into multiple pieces that get defined differently for different manifestations of the build source, and then joined together in the build source definition.

[azr]

Hey there, inline:

"you cannot redefine or substitute arbitrary user-defined variables (var.) in build sources. You can only specify locals (locals.) and builder-specified fields in build sources." Can you confirm/deny that statement?

That is correct, so you'd have to have a different variable for each different build.

If so, shouldn't this be in the docs somewhere? I'm sorry if I overlooked this somewhere.

I don't think this is specifically documented; maybe its because I know the code too well there, but it felt like this would be the expected behaviour: that nothing lets you set existing variables during a build. In a general manner input variables should be treated as constants and locals should be treated as compounds of functions, input variables, other locals, etc. But after this evaluation they are constant too. You can 'foreach' on variables though; which will create a subcontext giving you access to the scoped variable value. But these are constant too.

If not, I'm happy to submit another doc improvement PR if you don't have an immediate location in mind.

That would be neat ! 🙂

I suppose that since you can use locals in this way, you could overcome this limitation by, um, using locals. Breaking up boot_command into multiple pieces that get defined differently for different manifestations of the build source, and then joined together in the build source definition.

Yup, that's how I'd do it to DRY it up !

Note that HCL2 in Packer is very young and so the way you use it and what you feel like could be a nice addition is great feedback here.

[timblaktu]

Thanks @azr, I'm getting closer. I'm now able to substitute some locals and variables in the definitions of builder fields in build source.

EDIT: finally found the syntax for concatenating lists: https://www.packer.io/docs/from-1.5/functions/collection/concat

---

ATM, I'm getting an error trying to concatenate my 3 locals that comprise the desired boot_command string:

boot_command = local.boot_command_start + local.boot_command_preseed_url_headless + local.boot_command_end

and

locals { boot_command_start = [ "", "install " ] boot_command_preseed_url_headless = ["preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg "] boot_command_preseed_url_gnome = ["preseed/url=http://{{` .HTTPIP }}:{{ .HTTPPort }}/preseed-gnome.cfg "] boot_command_end = [ "debian-installer=en_US.UTF-8 ", "auto ", "locale=en_US.UTF-8 ", "kbd-chooser/method=us ", "keyboard-configuration/xkb-keymap=us ", "netcfg/get_hostname={{ .Name }} ", "netcfg/get_domain=biamp.com ", "fb=false ", "debconf/frontend=noninteractive ", "console-setup/ask_detect=false ", "console-keymaps-at/keymap=us ", "vga=839 ", "grub-installer/bootdev=/dev/sda ", "" ] }

I cannot find the right syntax to append these lists of string together...

[timblaktu]

@azr do you know anything wrong with trying to reference build.name and source.name from a build source block, as I'm doing below:

  source "source.virtualbox-iso.debian10" {
    name                    = "headless"
    vm_name                 = "${build.name}-${source.name}-${local.scm_suffix}"
    output_directory        = "${var.output_directory_root}/${build.name}-${source.name}-${local.scm_suffix}"
    boot_command            = concat(local.boot_command_start, local.boot_command_preseed_url_gnome, local.boot_command_end)
    disk_size               = 100000
    vboxmanage >>--->--->---= [
      ["modifyvm", "{{.Name}}", "--memory", "4096"],
      ["modifyvm", "{{.Name}}", "--cpus", "6"],
    ]
  }

This gives me:

11:49:18  Error: Unsupported attribute
11:49:18  
11:49:18    on packer.pkr.hcl line 13:
11:49:18    (source code not available)
11:49:18  
11:49:18  This object does not have an attribute named "name".

I'm guessing I don't have access to these Contextual variables in the build.source block. I'm confused by what are the old-style TEmplated {{.XXX}} equivalents to the above? I simply want to grab the build name and source name to compose the vm_name and output_directory.

[timblaktu]

OK, I found the docs about the build_name template function, and I'm using that to get me the build name, but I'm unable to get the source name in the same spot, so I have to hard-code it like "headless" below. How do I get the source name ("headless" in this case)?

  source "source.virtualbox-iso.debian10" {
    name                    = "headless"
    vm_name                 = "{{build_name}}-headless-${local.scm_suffix}"
    output_directory        = "${var.output_directory_root}/{{build_name}}-headless-${local.scm_suffix}"
    boot_command            = concat(local.boot_command_start, local.boot_command_preseed_url_gnome, local.boot_command_end)
    disk_size               = 100000
    vboxmanage              = [
      ["modifyvm", "{{.Name}}", "--memory", "4096"],
      ["modifyvm", "{{.Name}}", "--cpus", "6"],
    ]
  }

[timblaktu]

So.. I converged on the following packer dir to implement the aforementioned objective. Let me know if you notice anything off, or think of any improvements. Still couldn't figure out how to get the source name in the build source block without hard-coding it..

packer.pkr.hcl

build {
  name = "debian10"
  description = <<EOF
This build creates Debian 10 base images for the following builders:
* virtualbox-iso
It builds separate headless and gnome-desktop versions in parallel
and places them in uniquely-named output folders.
This build is intended to run on an ansible-managed Jenkins agent.
EOF
  source "source.virtualbox-iso.debian10" {
    name                    = "headless"
    vm_name                 = "{{build_name}}-headless-${local.scm_suffix}"
    output_directory        = "${var.output_directory_root}/{{build_name}}-headless-${local.scm_suffix}"
    boot_command            = concat(local.boot_command_start, local.boot_command_preseed_url_headless, local.boot_command_end)
  }
  source "source.virtualbox-iso.debian10" {
    name                    = "gnome"
    vm_name                 = "{{build_name}}-gnome-${local.scm_suffix}"
    output_directory        = "${var.output_directory_root}/{{build_name}}-gnome-${local.scm_suffix}"
    boot_command            = concat(local.boot_command_start, local.boot_command_preseed_url_gnome, local.boot_command_end)
  }
  provisioner "shell" {
    execute_command = "echo 'ansible' | sudo -S env {{ .Vars }} {{ .Path }}"
    timeout = "2m"
    max_retries = 0
    # VBoxLinuxAdditions.run is returning 2 every time, even when "successful"
    valid_exit_codes = [0, 2]
    inline_shebang = "/bin/bash -el"
    inline = [
      "set -x",
      "echo installing guest additions from ${var.guest_additions_iso_path}...",
      "ls -la ${var.guest_additions_iso_path}",
      "cd /tmp",
      "mkdir /tmp/isomount",
      "mount -t iso9660 -o loop ${var.guest_additions_iso_path} /tmp/isomount",
      "echo working around this issue: https://stackoverflow.com/a/25943638/532621...",
      "REMOVE_INSTALLATION_DIR=0 /tmp/isomount/VBoxLinuxAdditions.run --nox11",
      "umount isomount",
      "rm -rf isomount ${var.guest_additions_iso_path}"
    ]
  }
  provisioner "ansible" {
    # packer user becomes ansible_user. Defaults to the user running packer (jenkins), so we MUST set this here.
    # From: https://www.packer.io/docs/provisioners/ansible#user
    user = "ansible"
    playbook_file = "../../ansible/all-machines.yml"
  }
  post-processor "checksum" {
    checksum_types = ["md5"]
    # Could use build/source contextual variables here, but I choose to use exact same syntax
    # as is used in the build source to ensure correctness.
    # output = "${var.output_directory_root}/${build.name}-${source.name}-${local.scm_suffix}/checksum.{{.ChecksumType}}"
    output = "${var.output_directory_root}/{{build_name}}-headless-${local.scm_suffix}/checksum.{{.ChecksumType}}"
    keep_input_artifact = true
  }
  post-processor "manifest" {
    # Could use build/source contextual variables here, but I choose to use exact same syntax
    # as is used in the build source to ensure correctness. 
    # output = "${var.output_directory_root}/${build.name}-${source.name}-${local.scm_suffix}/manifest.json"
    output = "${var.output_directory_root}/{{build_name}}-headless-${local.scm_suffix}/manifest.json"
    strip_path = false
    keep_input_artifact = true
  }
}

sources.pkr.hcl

source "virtualbox-iso" "debian10" {
  # NB: Builder fields cannot be defined in multiple places or there will be runtime error.

  cpus                    = 16     # CPU used by host to build the VM
  memory                  = 16384  # RAM (MB) used by host to build the VM
  boot_wait               = "5s"
  guest_os_type           = "debian10_64Guest"
  iso_url                 = "https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-10.7.0-amd64-DVD-1.iso"
  iso_checksum            = "md5:270d5af2e1327a2a6ca6d24c468862b5"
  headless                = true
  http_directory          = "http"
  shutdown_command        = "echo 'ansible'|sudo -S shutdown -P now"
  ssh_password            = "ansible"
  ssh_username            = "ansible"
  ssh_wait_timeout        = "30m"
  hard_drive_interface    = "sata"
  guest_additions_path    = var.guest_additions_iso_path  # this constant variable is needed in provisioner step
  format                  = "ovf"
  vrdp_bind_address       = "0.0.0.0"
  vrdp_port_min           = 5000
  vrdp_port_max           = 5050
  disk_size               = 100000
  vboxmanage              = [
    ["modifyvm", "{{.Name}}", "--memory", "4096"],
    ["modifyvm", "{{.Name}}", "--cpus", "6"],
  ]
}

variables.pkr.hcl

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
  scm_suffix  = "${var.branch}-${var.commit}"

  # These aren't very useful since @azr confirms you can't set vars or locals inside a
  # source block, requiring a separate variable for each manifestation (headless, gnome),
  # so since using variables in this case is overly verbose we just hard-code the string literals.
  # supported_base_image_types = ['headless,' 'gnome']
  # base_image_type = "headless"

  # <esc> captures grub boot: prompt
  # See "Debian Installer Parameters" section for details on other installer-specific boot paramters
  #   https://www.debian.org/releases/stable/amd64/ch05s03.en.html
  boot_command_start = [
    "<esc><wait>",
    "install <wait>"
  ]
  boot_command_preseed_url_headless = ["preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg <wait>"]
  boot_command_preseed_url_gnome = ["preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed-gnome.cfg <wait>"]
  boot_command_end = [
    "debian-installer=en_US.UTF-8 <wait>",
    "auto <wait>",
    "locale=en_US.UTF-8 <wait>",
    "kbd-chooser/method=us <wait>",
    "keyboard-configuration/xkb-keymap=us <wait>",
    "netcfg/get_hostname={{ .Name }} <wait>",
    "netcfg/get_domain=biamp.com <wait>",
    "fb=false <wait>",
    "debconf/frontend=noninteractive <wait>",
    "console-setup/ask_detect=false <wait>",
    "console-keymaps-at/keymap=us <wait>",
    "vga=839 <wait>",
    "grub-installer/bootdev=/dev/sda <wait>",
    "<enter><wait>"
  ]
}

variable "ansible_active_directory_password" {
  type = string
  sensitive = true
  default = ""
}
variable "branch" {
  type = string
  sensitive = false
  default = ""
}
variable "commit" {
  type = string
  sensitive = false
  default = ""
}
variable "guest_additions_iso_path" {
  type    = string
  default = "/tmp/VBoxGuestAdditions.iso"
  description = "This path is needed in provisioner step, and is to be constant."
}
variable "output_directory_root" {
  type    = string
  default = "artifacts"
}

[azr]

Hey @timblaktu I just reading back at this; this looks super nice ! One thing that I see after quickly taking a look over this is that in the headless build you are using the local.boot_command_preseed_url_gnome command instead of the headless one.

I'll be analysing this, this is super insightful ! Thanks !

[azr]

Yeah one thing is that we have to do is up our HCL2 variables game as currently you have to use go templating variables. I really wish that we could use only HCL2 everywhere. 🙂 It's on the todo list. Other than that, these files looks super fine to me ! Love the comments ! Super readable and maintainable !

Yeah and for clarity please note that hcl2 variables and go templating variables will not be interpolated at the same moment, so for now we advise on avoiding calling hcl2 functions on go templating strings and vice versa.

[timblaktu]

Thanks @azr, and good catch. I edited my post above to correct the typo.

Did you have any feedback about whether it is possible to dynamically fetch the source name from either a contextual variable or a go template, so that in my build source blocks I don't have to hard-code, e.g. "headless" here in multiple places:

  source "source.virtualbox-iso.debian10" {
    name                    = "headless"
    vm_name                 = "{{build_name}}-headless-${local.scm_suffix}"
    output_directory        = "${var.output_directory_root}/{{build_name}}-headless-${local.scm_suffix}"
    boot_command            = concat(local.boot_command_start, local.boot_command_preseed_url_headless, local.boot_command_end)
  }

Before closing this one out, perhaps you could point me at where in the code or docs that I could learn about where in the underlying packer process all of the hcl2 variables and the go templating evaluation happens WRT each other? I (and others) would probably ask a lot fewer questions if I understood this better. I'm imagining a simple block diagram of packer internal application flow. Does this exist anywhere? It'd help not only users in understanding how to use packer, but would probably encourage more project contribution.

Similarly, I'd love to get some advice on where/how I could find reference to ALL of the items available through the go templating in packer. Would love to see follow ups to comments like the insightful ones by @SwampDragons in this issue to empower users and other would-be contributors to find their own answers, by not just saying there exists a "special option" and "template prepare" and "build" phases, but provide context by illustrating the big picture or at the least point to the place in the code they're referring to.

I don't mean to burden you with extra requests, I just wanted to provide my opinion about one little way in which the project might get more contribution and adoption. Certainly I can take it upon myself to start grokking the source code more; I was just hoping for a little introduction/overview guidance.

I love packer, and I really truly appreciate all the work your are doing here, @azr and @SwampDragons!

[SwampDragons]

@timblaktu you're absolutely right that we could do a better job of explaining variable interpolation in a way that is meaningful for end users. It's extra "fun" now because of the fact that HCL and JSON templates do things very differently on the front end, which is going to be a source of confusion for a while as we get people to make the jump over to HCL. I'll take a stab at a diagram explaining the difference between golang template variables and HCL variables before we consider this issue closed.

[timblaktu]

@SwampDragons, any progress on this? I'm now running into a new issue very much related to confusion about where you can use variables and where you cannot. I've defined a variable in an HCL Packer project that I'm passing in to Packer command line from a Jenkins project, and trying to use this variable to define the name of a builder (i.e. instantiation of a build source). Running Packer validate on this returns the error: "variables may not be used here"

[SwampDragons]

Sorry, I've been swamped and haven't had a chance to work on this. I don't think it's possible to define builder names based on variables. They're special and reserved as constants.

[timblaktu]

Thanks for the quick response. I think what I may do is just change my model for using packer to move my variables out of the packer project and into the Jenkins pipeline (which invokes packer), and render multiple .pkr.hcl files from jinja2 templates, instead of relying on Packer's partially-implemented HCL variable interpolation. Taking this path would have saved me weeks of time in my experience using Packer for anything but the most trivial projects.

Please don't take this as criticism, I know you're a limited team working this project and probably have higher priority things to work on. (Much higher, I bet, now that HCP Packer was announced ;-) ) Thanks for your good work.

[SwampDragons]

Thanks for your understanding. It does sound like that solution would be better for you in this case.

[anarazel]

@timblaktu I hit a similar problem, and just found a way to work around it, using dynamic blocks. Not extremely pretty, but might still be sufficient to help you. Gets around that whole business with {{build_name}} / ${build.name}/${source.name} not really working in source blocks.

locals {
  debian_gcp_images = [
    {
      name = "bullseye"
      zone = "us-west1-a"
      machine = "e2-highcpu-4"
    },
    {
      name = "sid"
      zone = "us-west1-a"
      machine = "e2-highcpu-4"
    },
    {
      name = "sid-newkernel"
      zone = "us-west2-a"
      machine = "c2-standard-8"
    },
    {
      name = "sid-newkernel-uring"
      zone = "us-west2-a"
      machine = "c2-standard-8"
    },
  ]
...
source "googlecompute" "bullseye-vanilla" {
  project_id              = "pg-vm-images-aio"
  ...
}
...
build {
  name="linux"

  # Generate debian gcp images. Unfortunately variable expansion inside source
  # and build blocks doesn't work well yet on packer 1.7.7. Hence this.

  dynamic "source" {
    for_each = local.debian_gcp_images
    labels = ["source.googlecompute.bullseye-vanilla"]
    iterator = tag

    content {
      name = tag.value.name
      image_name = "${var.prefix}pg-aio-${tag.value.name}-${var.image_date}"
      image_family = "${var.prefix}pg-aio-${tag.value.name}"

      zone = tag.value.zone
      machine_type = tag.value.machine
      instance_name = "build-${tag.value.name}"
    }
  }
...

[prashantkalkar]

@azr @timblaktu Does it make sense to make the provisioners and post processors as top level block / resources? And provisioners should reference back to the builder rather than builder referring to provisioner (this is similar to aws_security_group and aws_security_group_rule in case of terraform).

This should allow people to extend existing packer configuration by providing additional provisioners.

(Let me know if this should be a separate issue)

[github-actions[bot]]

This issue has been synced to JIRA for planning.

JIRA ID: HPR-756