search

hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.1k stars 3.33k forks source link

Unexpected behaviour when referencing named build in another build source #11264

Open Vinrobot opened 3 years ago

Vinrobot commented 3 years ago

Overview of the Issue

When specifying a source in a build block using the syntax <another build name>.<source type>.<source name>, the <build name> is not considered, therefore the two builds are run in parallel with the same source block. (Example in Simplified Packer Buildfile) While this could be the "normal" behaviour, I expected this to execute the first build and when it's finished, to start the second build (because in the log, builds are referenced with this syntax).

In my opinion, either this syntax should not be allowed or the dependency should be respected. (Or I might just not see the use case of this syntax)

Finally, in the documentation, the two used prefix are sources (ex.: sources.null.first-example) and source (ex.: source.null.first-example) but I did not find any reference about what is exactly allowed or how it's used.

Reproduction Steps

Packer version

Packer v1.7.4

Simplified Packer Buildfile

source "null" "first-example" {
  communicator = "none"
}
build {
  name = "step1"
  # Use the source block
  sources = ["source.null.first-example"]
  provisioner "shell-local" {
    inline = ["echo Hello from ${build.name}.${source.type}.${source.name}"]
  }
}
build {
  name = "step2"
  # Use the step1 build block (that's how I understood this)
  sources = ["step1.null.first-example"]
  provisioner "shell-local" {
    inline = ["echo Hello from ${build.name}.${source.type}.${source.name}"]
  }
}

Operating system and Environment details

macOS Big Sur 11.5.2

Log Fragments and crash.log files

Packer logs ```console $ PACKER_LOG=1 packer build main.pkr.hcl 2021/09/11 17:17:55 [INFO] Packer version: 1.7.4 [go1.16.6 darwin amd64] 2021/09/11 17:17:55 [TRACE] discovering plugins in /usr/local/bin 2021/09/11 17:17:55 [TRACE] discovering plugins in /Users/packer/.packer.d/plugins 2021/09/11 17:17:55 [TRACE] discovering plugins in . 2021/09/11 17:17:55 [INFO] PACKER_CONFIG env var not set; checking the default config file path 2021/09/11 17:17:55 [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/packer/.packerconfig 2021/09/11 17:17:55 [WARN] Config file doesn't exist: /Users/packer/.packerconfig 2021/09/11 17:17:55 [INFO] Setting cache directory: /Users/packer/Documents/Git/GitHub/Packer/packer_cache 2021/09/11 17:17:55 [TRACE] Starting internal plugin packer-builder-null 2021/09/11 17:17:55 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-builder-null"} 2021/09/11 17:17:55 Waiting for RPC address for: /usr/local/bin/packer 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] Packer version: 1.7.4 [go1.16.6 darwin amd64] 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] PACKER_CONFIG env var not set; checking the default config file path 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-builder-null plugin: [WARN] Config file doesn't exist: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] Setting cache directory: /Users/packer/Documents/Git/GitHub/Packer/packer_cache 2021/09/11 17:17:55 packer-builder-null plugin: args: []string{"packer-builder-null"} 2021/09/11 17:17:55 packer-builder-null plugin: Plugin address: unix /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin902877200 2021/09/11 17:17:55 Received unix RPC address for /usr/local/bin/packer: addr is /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin902877200 2021/09/11 17:17:55 packer-builder-null plugin: Waiting for connection... 2021/09/11 17:17:55 packer-builder-null plugin: Serving a plugin connection... 2021/09/11 17:17:55 [TRACE] Starting internal plugin packer-provisioner-shell-local 2021/09/11 17:17:55 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-provisioner-shell-local"} 2021/09/11 17:17:55 Waiting for RPC address for: /usr/local/bin/packer 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] Packer version: 1.7.4 [go1.16.6 darwin amd64] 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] PACKER_CONFIG env var not set; checking the default config file path 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [WARN] Config file doesn't exist: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] Setting cache directory: /Users/packer/Documents/Git/GitHub/Packer/packer_cache 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: args: []string{"packer-provisioner-shell-local"} 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: Plugin address: unix /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin265123305 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: Waiting for connection... 2021/09/11 17:17:55 Received unix RPC address for /usr/local/bin/packer: addr is /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin265123305 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: Serving a plugin connection... 2021/09/11 17:17:55 [TRACE] Starting internal plugin packer-builder-null 2021/09/11 17:17:55 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-builder-null"} 2021/09/11 17:17:55 Waiting for RPC address for: /usr/local/bin/packer 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] Packer version: 1.7.4 [go1.16.6 darwin amd64] 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] PACKER_CONFIG env var not set; checking the default config file path 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-builder-null plugin: [WARN] Config file doesn't exist: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] Setting cache directory: /Users/packer/Documents/Git/GitHub/Packer/packer_cache 2021/09/11 17:17:55 packer-builder-null plugin: args: []string{"packer-builder-null"} 2021/09/11 17:17:55 packer-builder-null plugin: Plugin address: unix /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin754335914 2021/09/11 17:17:55 packer-builder-null plugin: Waiting for connection... 2021/09/11 17:17:55 Received unix RPC address for /usr/local/bin/packer: addr is /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin754335914 2021/09/11 17:17:55 packer-builder-null plugin: Serving a plugin connection... 2021/09/11 17:17:55 [TRACE] Starting internal plugin packer-provisioner-shell-local 2021/09/11 17:17:55 Starting plugin: /usr/local/bin/packer []string{"/usr/local/bin/packer", "plugin", "packer-provisioner-shell-local"} 2021/09/11 17:17:55 Waiting for RPC address for: /usr/local/bin/packer 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] Packer version: 1.7.4 [go1.16.6 darwin amd64] 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] PACKER_CONFIG env var not set; checking the default config file path 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [WARN] Config file doesn't exist: /Users/packer/.packerconfig 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] Setting cache directory: /Users/packer/Documents/Git/GitHub/Packer/packer_cache 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: args: []string{"packer-provisioner-shell-local"} 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: Plugin address: unix /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin131698075 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: Waiting for connection... 2021/09/11 17:17:55 Received unix RPC address for /usr/local/bin/packer: addr is /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-plugin131698075 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: Serving a plugin connection... 2021/09/11 17:17:55 Build debug mode: false 2021/09/11 17:17:55 Force build: false 2021/09/11 17:17:55 On error: step1.null.first-example: output will be in this color. 2021/09/11 17:17:55 Waiting on builds to complete... 2021/09/11 17:17:55 Starting build run: step2.null.first-example 2021/09/11 17:17:55 Starting build run: step1.null.first-example 2021/09/11 17:17:55 Running builder: 2021/09/11 17:17:55 [INFO] (telemetry) Starting builder 2021/09/11 17:17:55 Running builder: 2021/09/11 17:17:55 [INFO] (telemetry) Starting builder step2.null.first-example: output will be in this color. 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] communicator disabled, will not connect 2021/09/11 17:17:55 packer-builder-null plugin: Unable to load communicator config from state to populate provisionHookData 2021/09/11 17:17:55 packer-builder-null plugin: Running the provision hook 2021/09/11 17:17:55 packer-builder-null plugin: [INFO] communicator disabled, will not connect 2021/09/11 17:17:55 packer-builder-null plugin: Unable to load communicator config from state to populate provisionHookData 2021/09/11 17:17:55 packer-builder-null plugin: Running the provision hook 2021/09/11 17:17:55 [INFO] (telemetry) Starting provisioner shell-local 2021/09/11 17:17:55 [INFO] (telemetry) Starting provisioner shell-local 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] (shell-local): Prepending inline script with #!/bin/sh -e 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] (shell-local): Prepending inline script with #!/bin/sh -e ==> step1.null.first-example: Running local shell script: /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-shell788462644 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] (shell-local): starting local command: /bin/sh -c PACKER_BUILDER_TYPE='null' PACKER_BUILD_NAME='first-example' /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-shell788462644 ==> step2.null.first-example: Running local shell script: /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-shell290406974 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] (shell-local): starting local command: /bin/sh -c PACKER_BUILDER_TYPE='null' PACKER_BUILD_NAME='first-example' /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-shell290406974 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] (shell-local communicator): Executing local shell command [/bin/sh -c PACKER_BUILDER_TYPE='null' PACKER_BUILD_NAME='first-example' /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-shell788462644] 2021/09/11 17:17:55 packer-provisioner-shell-local plugin: [INFO] (shell-local communicator): Executing local shell command [/bin/sh -c PACKER_BUILDER_TYPE='null' PACKER_BUILD_NAME='first-example' /var/folders/dn/27dnr6wd2vn105tjp9w5r_h00000gn/T/packer-shell290406974] step2.null.first-example: Hello from step2.null.first-example 2021/09/11 17:17:55 [INFO] (telemetry) ending shell-local 2021/09/11 17:17:55 [INFO] (telemetry) ending Build 'step2.null.first-example' finished after 64 milliseconds 716 microseconds. step1.null.first-example: Hello from step1.null.first-example 2021/09/11 17:17:55 [INFO] (telemetry) ending shell-local 2021/09/11 17:17:55 [INFO] (telemetry) ending ==> Wait completed after 102 milliseconds 984 microseconds ==> Builds finished. The artifacts of successful builds are: 2021/09/11 17:17:55 machine readable: step2.null.first-example,artifact-count []string{"1"} Build 'step1.null.first-example' finished after 102 milliseconds 914 microseconds. 2021/09/11 17:17:55 machine readable: step2.null.first-example,artifact []string{"0", "builder-id", "fnoeding.null"} ==> Wait completed after 102 milliseconds 984 microseconds 2021/09/11 17:17:55 machine readable: step2.null.first-example,artifact []string{"0", "id", "Null"} ==> Builds finished. The artifacts of successful builds are: 2021/09/11 17:17:55 machine readable: step2.null.first-example,artifact []string{"0", "string", "Did not export anything. This is the null builder"} 2021/09/11 17:17:55 machine readable: step2.null.first-example,artifact []string{"0", "files-count", "0"} 2021/09/11 17:17:55 machine readable: step2.null.first-example,artifact []string{"0", "end"} 2021/09/11 17:17:55 machine readable: step1.null.first-example,artifact-count []string{"1"} --> step2.null.first-example: Did not export anything. This is the null builder 2021/09/11 17:17:55 machine readable: step1.null.first-example,artifact []string{"0", "builder-id", "fnoeding.null"} 2021/09/11 17:17:55 machine readable: step1.null.first-example,artifact []string{"0", "id", "Null"} 2021/09/11 17:17:55 machine readable: step1.null.first-example,artifact []string{"0", "string", "Did not export anything. This is the null builder"} 2021/09/11 17:17:55 machine readable: step1.null.first-example,artifact []string{"0", "files-count", "0"} 2021/09/11 17:17:55 machine readable: step1.null.first-example,artifact []string{"0", "end"} --> step1.null.first-example: Did not export anything. This is the null builder 2021/09/11 17:17:55 [INFO] (telemetry) Finalizing. 2021/09/11 17:17:56 waiting for all plugin processes to complete... 2021/09/11 17:17:56 /usr/local/bin/packer: plugin process exited 2021/09/11 17:17:56 /usr/local/bin/packer: plugin process exited 2021/09/11 17:17:56 /usr/local/bin/packer: plugin process exited 2021/09/11 17:17:56 /usr/local/bin/packer: plugin process exited ```
azr commented 3 years ago

Hey @Vinrobot, thanks for opening. This is the currently expected behavior. For now, the HCL2 part of Packer can't do that, and it will remain like this; at least until we deprecate the legacy JSON core (in other word, not until Packer v2).

Once we can get rid of the old JSON core, we would very much like to implement what we call 'The DAG' (dependency acyclic graph), which, should allow you to do exactly that. But for now, because the execution model of Packer is pretty much staged, as in (first variables, then data sources, then sources and then finally builds) Packer can't do that.

The only/current way to chain builds is to do as described in here: https://www.packer.io/guides/packer-on-cicd/pipelineing-builds I hope this helps; thanks !

github-actions[bot] commented 2 years ago

This issue has been synced to JIRA for planning.

JIRA ID: HPR-752