search

hashicorp / packer-plugin-googlecompute

Packer plugin for Google Compute Builder
https://www.packer.io/docs/builders/googlecompute
Mozilla Public License 2.0
23 stars 53 forks source link

Unable to build a new AMI due to fact that IAP gives 4047 error #37

Open Endriuz1805 opened 2 years ago

Endriuz1805 commented 2 years ago

Hi,

We are building new AMI's monthly through a pipeline each month, which on last month we successfully did it. This month it fails with the following error:

2021/09/02 11:48:24 1630572504,,ui,say,==> googlecompute: Step Launch IAP Tunnel...
2021/09/02 11:48:24 packer-builder-googlecompute plugin: Finding an available TCP port for IAP proxy
1630572504,,ui,say,==> googlecompute: Step Launch IAP Tunnel...
2021/09/02 11:48:24 packer-builder-googlecompute plugin: Found available port: 8144 on IP: 0.0.0.0
2021/09/02 11:48:24 packer-builder-googlecompute plugin: Setting up proxy to listen on localhost at 8144
2021/09/02 11:48:24 packer-builder-googlecompute plugin: Creating tunnel launch script with args []string{"compute", "start-iap-tunnel", "packer-61308fc5-8a55-bd3a-ac52-xxxx", "22", "--local-host-port=localhost:8144", "--zone", "europe-west2-c", "--project", "xxxxxx"}
2021/09/02 11:48:24 packer-builder-googlecompute plugin: [INFO] (google): Prepending inline gcloud setup script with #!/bin/sh
2021/09/02 11:48:25 packer-builder-googlecompute plugin: stderr: Testing if tunnel connection works.
2021/09/02 11:48:26 packer-builder-googlecompute plugin: stderr: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
2021/09/02 11:48:26 packer-builder-googlecompute plugin: NOT RETRYABLE: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
2021/09/02 11:48:26 1630572506,,ui,error,==> googlecompute: Non-retryable tunnel error: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].\n==> googlecompute:
2021/09/02 11:48:26 packer-builder-googlecompute plugin: Couldn't find IAP tunnel process to kill. Continuing.
2021/09/02 11:48:26 packer-builder-googlecompute plugin: Error asking for input: machine-readable UI can't ask
1630572506,,ui,error,==> googlecompute: Non-retryable tunnel error: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].\n==> googlecompute:
2021/09/02 11:48:26 packer-builder-googlecompute plugin: Error asking for input: machine-readable UI can't ask
2021/09/02 11:48:26 packer-builder-googlecompute plugin: Error asking for input: machine-readable UI can't ask
2021/09/02 11:48:26 1630572506,,ui,say,==> googlecompute: Deleting instance...
1630572506,,ui,say,==> googlecompute: Deleting instance...
2021/09/02 11:48:27 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/02 11:48:29 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/02 11:48:31 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING

I tried to run it from locally but i have the same error.

"builders": [
    {
      "type": "googlecompute",
      "project_id": "xxxxxxxxxx",
      "source_image_project_id": "cis-public",
      "source_image_family": "cis-rhel7-l1",
      "ssh_username": "cxxxx",
      "account_file": "packer-sa.json",
      "zone": "europe-west2-c",
      "image_name": "jlr-gcp-cloudimages-cis-redhat78-{{timestamp}}",
      "image_family": "jlr-gcp-cloudimages-cis-redhat78",
      "machine_type": "c2-standard-4",
      "disk_size": "20",
      "network": "https://www.googleapis.com/compute/v1/projects/xxxxxxglobal/networks/xxxxxxxxx",
      "subnetwork": "https://www.googleapis.com/compute/v1/projects/xxxxxxxxxx/regions/europe-west2/subnetworks/xxxxxxxxx",
      "tags": "allow-ssh-build",
      "wrap_startup_script": "false",
      "use_iap": "true",
      "omit_external_ip": "true",
      "use_internal_ip": "true",
      "enable_secure_boot": "true",
erwin-willems commented 2 years ago

I expect that this commit will fix this issue: https://github.com/hashicorp/packer-plugin-googlecompute/commit/bb5b4a5d57ef4092b09aa335dbe9e38a7901a740#diff-1906a4e18aa77344dec719024aac1d2d4f45d3535aa81963507eeae5786063d8

erwin-willems commented 2 years ago

That commit indeed solves this issue.

To fix this issue, update googlecompute plugin to 1.0.2 or higher:

anjil commented 2 years ago

I did the mentioned update but the issue still persist, please help. I have issue with GCP instances.

erwin-willems commented 2 years ago

I did the mentioned update but the issue still persist, please help. I have issue with GCP instances.

Without the logs and the packer file it's hard to say what the issue is. Can you share the logs and your packer file?

Endriuz1805 commented 2 years ago

That commit indeed solves this issue.

To fix this issue, update googlecompute plugin to 1.0.2 or higher:

  • Add to your config.pkr.hcl:
packer {
  required_plugins {
    googlecompute = {
      version = ">= 1.0.2"
      source  = "github.com/hashicorp/googlecompute"
    }
  }
}
  • Run packer init config.pkr.hcl to install this plugin version
  • Run packer build

Indeed by converting the files from legacy json to .pkr.hcl and adding the required_plugins to the template fixes the issue. Any chance that this would work with the legacy templates as well?

anjil commented 2 years ago

I did the mentioned update but the issue still persist, please help. I have issue with GCP instances.

Without the logs and the packer file it's hard to say what the issue is. Can you share the logs and your packer file?

Thanks for replying. I am attaching the log and required packer files. ================= Logs ======================

2021/09/06 12:50:43 [INFO] Packer version: 1.7.4 [go1.16.6 linux amd64]
2021/09/06 12:50:43 Detected config directory from env var: ~/
2021/09/06 12:50:43 [TRACE] discovering plugins in /home/ext_anjil_p
2021/09/06 12:50:43 Detected config directory from env var: ~/
2021/09/06 12:50:43 [TRACE] discovering plugins in ~/.packer.d/plugins
2021/09/06 12:50:43 [TRACE] discovering plugins in .
2021/09/06 12:50:43 [INFO] PACKER_CONFIG env var set; attempting to open config file: ~/.packerconfig
2021/09/06 12:50:43 [WARN] Config file doesn't exist: ~/.packerconfig
2021/09/06 12:50:43 Detected config directory from env var: ~/
2021/09/06 12:50:43 [INFO] Setting cache directory: /home/ext_anjil_p/packer_cache
2021/09/06 12:50:43 Detected config directory from env var: ~/2021/09/06 12:50:43 [TRACE] Starting internal plugin packer-builder-googlecompute
2021/09/06 12:50:43 Starting plugin: /home/ext_anjil_p/packer []string{"/home/ext_anjil_p/packer", "plugin", "packer-builder-googlecompute"}
2021/09/06 12:50:43 Waiting for RPC address for: /home/ext_anjil_p/packer
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO] Packer version: 1.7.4 [go1.16.6 linux amd64]2021/09/06 12:50:44 packer-builder-googlecompute plugin: Detected config directory from env var: ~/
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: ~/.packerconfig
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [WARN] Config file doesn't exist: ~/.packerconfig
2021/09/06 12:50:44 packer-builder-googlecompute plugin: Detected config directory from env var: ~/
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO] Setting cache directory: /home/ext_anjil_p/packer_cache
2021/09/06 12:50:44 packer-builder-googlecompute plugin: args: []string{"packer-builder-googlecompute"}
2021/09/06 12:50:44 packer-builder-googlecompute plugin: Detected config directory from env var: ~/2021/09/06 12:50:44 packer-builder-googlecompute plugin: Plugin address: unix /tmp/packer-plugin360020066
2021/09/06 12:50:44 packer-builder-googlecompute plugin: Waiting for connection...
2021/09/06 12:50:44 Received unix RPC address for /home/ext_anjil_p/packer: addr is /tmp/packer-plugin360020066
2021/09/06 12:50:44 packer-builder-googlecompute plugin: Serving a plugin connection...2021/09/06 12:50:44 [TRACE] Starting internal plugin packer-provisioner-shell
2021/09/06 12:50:44 Starting plugin: /home/ext_anjil_p/packer []string{"/home/ext_anjil_p/packer", "plugin", "packer-provisioner-shell"}2021/09/06 12:50:44 Waiting for RPC address for: /home/ext_anjil_p/packer
2021/09/06 12:50:44 packer-provisioner-shell plugin: [INFO] Packer version: 1.7.4 [go1.16.6 linux amd64]
2021/09/06 12:50:44 packer-provisioner-shell plugin: Detected config directory from env var: ~/
2021/09/06 12:50:44 packer-provisioner-shell plugin: [INFO] PACKER_CONFIG env var set; attempting to open config file: ~/.packerconfig
2021/09/06 12:50:44 packer-provisioner-shell plugin: [WARN] Config file doesn't exist: ~/.packerconfig
2021/09/06 12:50:44 packer-provisioner-shell plugin: Detected config directory from env var: ~/
2021/09/06 12:50:44 packer-provisioner-shell plugin: [INFO] Setting cache directory: /home/ext_anjil_p/packer_cache
2021/09/06 12:50:44 packer-provisioner-shell plugin: args: []string{"packer-provisioner-shell"}
2021/09/06 12:50:44 packer-provisioner-shell plugin: Detected config directory from env var: ~/
2021/09/06 12:50:44 packer-provisioner-shell plugin: Plugin address: unix /tmp/packer-plugin211264554
2021/09/06 12:50:44 packer-provisioner-shell plugin: Waiting for connection...
2021/09/06 12:50:44 Received unix RPC address for /home/ext_anjil_p/packer: addr is /tmp/packer-plugin211264554
2021/09/06 12:50:44 packer-provisioner-shell plugin: Serving a plugin connection...
2021/09/06 12:50:44 ui: googlecompute.ubuntu18: output will be in this color.
2021/09/06 12:50:44 ui:
2021/09/06 12:50:44 Build debug mode: false
2021/09/06 12:50:44 Force build: false
2021/09/06 12:50:44 On error:
2021/09/06 12:50:44 Waiting on builds to complete...
2021/09/06 12:50:44 Starting build run: googlecompute.ubuntu18
2021/09/06 12:50:44 Running builder:
2021/09/06 12:50:44 [INFO] (telemetry) Starting builder
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO] Requesting Google token via account_file...
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO]   -- Email: <PROJECT_SERVICE_ACCOUNT>
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO]   -- Scopes: [https://www.googleapis.com/auth/compute https://www.googleapis.com/auth/devstorage.full_control]
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO]   -- Private Key Length: 1704
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO] Instantiating GCE client...
2021/09/06 12:50:44 packer-builder-googlecompute plugin: [INFO] Instantiating OS Login client...
2021/09/06 12:50:44 ui: ==> googlecompute.ubuntu18: Checking image does not exist...
2021/09/06 12:50:44 ui: ==> googlecompute.ubuntu18: Using existing SSH private key
2021/09/06 12:50:44 ui: ==> googlecompute.ubuntu18: No public SSH key found; skipping SSH public key import for OSLogin...
2021/09/06 12:50:48 ui: ==> googlecompute.ubuntu18: Using image: ubuntu-1804-bionic-v20210825
2021/09/06 12:50:48 ui: ==> googlecompute.ubuntu18: Creating instance...
2021/09/06 12:50:48 packer-builder-googlecompute plugin: [DEBUG] Adding metadata during instance creation...
2021/09/06 12:50:48 ui:     googlecompute.ubuntu18: Loading zone: europe-west2-a
2021/09/06 12:50:48 ui:     googlecompute.ubuntu18: Loading machine type: n1-standard-1
2021/09/06 12:50:48 ui:     googlecompute.ubuntu18: Requesting instance creation...
2021/09/06 12:50:49 ui:     googlecompute.ubuntu18: Waiting for creation operation to complete...
2021/09/06 12:50:49 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:50:52 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:50:54 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:50:56 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:50:58 ui:     googlecompute.ubuntu18: Instance has been created!
2021/09/06 12:50:58 ui: ==> googlecompute.ubuntu18: Waiting for the instance to become running...
2021/09/06 12:50:59 ui:     googlecompute.ubuntu18: IP: 10.50.212.83
2021/09/06 12:50:59 ui: ==> googlecompute.ubuntu18: Step Launch IAP Tunnel...
2021/09/06 12:50:59 packer-builder-googlecompute plugin: Finding an available TCP port for IAP proxy
2021/09/06 12:50:59 packer-builder-googlecompute plugin: Found available port: 8434 on IP: 0.0.0.0
2021/09/06 12:50:59 packer-builder-googlecompute plugin: Setting up proxy to listen on localhost at 8434
2021/09/06 12:50:59 packer-builder-googlecompute plugin: Creating tunnel launch script with args []string{"compute", "start-iap-tunnel", "packer-61360ea4-b045-676d-532d-05933d5a218d", "22", "--local-host-port=localhost:8434", "--zone", "europe-west2-a", "--project", "<PROJECT_ID>"}
2021/09/06 12:50:59 packer-builder-googlecompute plugin: [INFO] (google): Prepending inline gcloud setup script with #!/bin/sh
2021/09/06 12:51:02 packer-builder-googlecompute plugin: stderr: Testing if tunnel connection works.
2021/09/06 12:51:03 packer-builder-googlecompute plugin: stderr: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
2021/09/06 12:51:03 packer-builder-googlecompute plugin: NOT RETRYABLE: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
2021/09/06 12:51:03 ui error: ==> googlecompute.ubuntu18: Non-retryable tunnel error: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
==> googlecompute.ubuntu18:
2021/09/06 12:51:03 packer-builder-googlecompute plugin: Couldn't find IAP tunnel process to kill. Continuing.
2021/09/06 12:51:03 ui: ==> googlecompute.ubuntu18: Deleting instance...
2021/09/06 12:51:04 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:06 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:09 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:11 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:13 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:15 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:18 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:20 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:22 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:24 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:27 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:29 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:31 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:34 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:36 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:38 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:40 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:43 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:45 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:47 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:50 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:52 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:54 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:56 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:51:59 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:01 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:03 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:05 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:08 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:10 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:12 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:15 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:17 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:19 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:21 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:24 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:26 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:28 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:31 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:33 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:35 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:37 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:40 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:42 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:44 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:46 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:49 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:51 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:53 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:56 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:52:58 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:00 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:02 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:05 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:07 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:09 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:12 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:14 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:16 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:18 ui:     googlecompute.ubuntu18: Instance has been deleted!
2021/09/06 12:53:18 ui: ==> googlecompute.ubuntu18: Deleting disk...
2021/09/06 12:53:19 packer-builder-googlecompute plugin: Retryable error: retrying for state DONE, got RUNNING
2021/09/06 12:53:22 ui:     googlecompute.ubuntu18: Disk has been deleted!
2021/09/06 12:53:22 [INFO] (telemetry) ending
2021/09/06 12:53:22 ui error: Build 'googlecompute.ubuntu18' errored after 2 minutes 37 seconds: Non-retryable tunnel error: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].

2021/09/06 12:53:22 ui:
==> Wait completed after 2 minutes 37 seconds
2021/09/06 12:53:22 machine readable: error-count []string{"1"}
2021/09/06 12:53:22 ui error:
==> Some builds didn't complete successfully and had errors:
2021/09/06 12:53:22 machine readable: googlecompute.ubuntu18,error []string{"Non-retryable tunnel error: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].\n"}
2021/09/06 12:53:22 ui error: --> googlecompute.ubuntu18: Non-retryable tunnel error: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
2021/09/06 12:53:22 ui:
==> Builds finished but no artifacts were created.
2021/09/06 12:53:22 [INFO] (telemetry) Finalizing.
2021/09/06 12:53:22 waiting for all plugin processes to complete...
2021/09/06 12:53:22 /home/ext_anjil_p/packer: plugin process exited
2021/09/06 12:53:22 /home/ext_anjil_p/packer: plugin process exited

================= config.pkr.hcl ==================

packer {
  required_plugins {
    googlecompute = {
      version = ">= 1.0.2"
      source = "github.com/hashicorp/googlecompute"
    }
  }
}

================ ubuntu.pkr.hcl ==================

source "googlecompute" "ubuntu18" {

  account_file      = "sa/sa-packer.json"
  image_description = "GCP Custom image based on Ubuntu 18.04 64-bit"
  image_family      = "ansible-ubuntu"
  image_labels = {
    generated_by = "packer"
    role         = "ansible_image"
  }
  image_name           = "ansible-image-ubuntu18"
  omit_external_ip     = true
  project_id           = "<PROJECT_ID>"
  source_image_family  = "ubuntu-1804-lts"
  ssh_private_key_file = ".ssh/packer"
  ssh_username         = "<SSH_USER_NAME>"
  subnetwork           = "<SHARED_NETWORK_SELF_LINK>"
  tags                 = ["allowsshrdpfromcloudiap"]
  use_iap              = true
  use_internal_ip      = true
  use_os_login         = true
  zone                 = "europe-west2-a"

}

build {
  sources = [
    "source.googlecompute.ubuntu18"
  ]

  provisioner "shell" {
    inline = [
      "sudo apt-add-repository --yes --update ppa:ansible/ansible",
      "sudo apt install ansible -y"
    ]
  }
}

================= script that runs packer commands ==============

export PACKER_CONFIG_DIR="~/"
export PACKER_CONFIG="${PACKER_CONFIG_DIR}.packerconfig"
export PACKER_LOG=1
export PACKER_LOG_PATH="packerlog.txt"

./packer init .
# ./packer init config.pkr.hcl
./packer validate ubuntu.pkr.hcl
./packer build ubuntu.pkr.hcl

The service account has all the required permissions as this setup was working few days back. Kindly check and help pls

erwin-willems commented 2 years ago

================= config.pkr.hcl ==================

packer {
  required_plugins {
    googlecompute = {
      version = ">= 1.0.2"
      source = "github.com/hashicorp/googlecompute"
    }
  }
}

================ ubuntu.pkr.hcl ==================

source "googlecompute" "ubuntu18" {

  account_file      = "sa/sa-packer.json"
  image_description = "GCP Custom image based on Ubuntu 18.04 64-bit"
  image_family      = "ansible-ubuntu"
  image_labels = {
    generated_by = "packer"
    role         = "ansible_image"
  }
  image_name           = "ansible-image-ubuntu18"
  omit_external_ip     = true
  project_id           = "<PROJECT_ID>"
  source_image_family  = "ubuntu-1804-lts"
  ssh_private_key_file = ".ssh/packer"
  ssh_username         = "<SSH_USER_NAME>"
  subnetwork           = "<SHARED_NETWORK_SELF_LINK>"
  tags                 = ["allowsshrdpfromcloudiap"]
  use_iap              = true
  use_internal_ip      = true
  use_os_login         = true
  zone                 = "europe-west2-a"

}

build {
  sources = [
    "source.googlecompute.ubuntu18"
  ]

  provisioner "shell" {
    inline = [
      "sudo apt-add-repository --yes --update ppa:ansible/ansible",
      "sudo apt install ansible -y"
    ]
  }
}

================= script that runs packer commands ==============

export PACKER_CONFIG_DIR="~/"
export PACKER_CONFIG="${PACKER_CONFIG_DIR}.packerconfig"
export PACKER_LOG=1
export PACKER_LOG_PATH="packerlog.txt"

./packer init .
# ./packer init config.pkr.hcl
./packer validate ubuntu.pkr.hcl
./packer build ubuntu.pkr.hcl

The service account has all the required permissions as this setup was working few days back. Kindly check and help pls

I'm not sure if "config.pkr.hcl" is parsed automatically. I see you're configuring a PACKER_CONFIG_DIR to your home directory, but also configure the config file "PACKER_CONFIG". So I'm unsure that config.pkr.hcl is used during the validate and build steps.

To make things easier:

If that works then config.pkr.hcl is not parsed during the validate and build step.

wyardley commented 5 months ago

Did you ever figure this out?

I'm seeing the same thing sporadically (got the same error, then the config started working, and then suddenly stopped again today). I can connect to existing instances via IAP fine, and the firewall rule allowing ssh traffic from IAP is in place, but I can verify that the packer-created instance gives the same IAP error if I try manually running gcloud compute ssh [instance] or gcloud compute start-iap-tunnel [instance] ..... Running gcloud with --log-http doesn't seem to give me any good clues. Just get into a loop with

2024/02/01 09:35:39 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:39 Creating tunnel launch script with args []string{"compute", "start-iap-tunnel", "packer-65bbd65d-ed4d-0ae5-609c-1edf09b225f9", "22", "--local-host-port=localhost:8822", "--zone", "us-central1-b", "--project", "xxx"}
2024/02/01 09:35:39 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:39 [INFO] (google): Prepending inline gcloud setup script with #!/bin/sh
2024/02/01 09:35:41 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:41 stderr: WARNING:
2024/02/01 09:35:42 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:42 stderr:
2024/02/01 09:35:43 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:43 stderr: To increase the performance of the tunnel, consider installing NumPy. For instructions,
2024/02/01 09:35:44 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:44 stderr: please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth
2024/02/01 09:35:45 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:45 stderr:
2024/02/01 09:35:46 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:46 stderr: Testing if tunnel connection works.
2024/02/01 09:35:47 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:35:47 stderr: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
[...]
2024/02/01 09:37:03 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:37:03 stderr: Testing if tunnel connection works.
2024/02/01 09:37:04 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:37:04 stderr: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
2024/02/01 09:37:04 packer-plugin-googlecompute_v1.1.4_x5.0_darwin_arm64 plugin: 2024/02/01 09:37:04 Retryable error: Tunnel start: ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].

Sanitized config:

packer {
  required_plugins {
    googlecompute = {
      source  = "github.com/hashicorp/googlecompute"
      version = "~> 1"
    }
  }
}

source "googlecompute" "foo" {
  disk_size               = "60"
  image_description       = "Description"
  image_family            = "foo"
  image_name              = "bar"
  machine_type            = "n2-standard-2"
  network                 = "main"
  subnetwork              = "${var.region}-01"
  omit_external_ip        = true
  use_internal_ip         = true
  use_iap                 = true
  on_host_maintenance     = "TERMINATE"
  project_id              = var.project
  scopes                  = ["https://www.googleapis.com/auth/cloud-platform"]
  source_image_family     = "ubuntu-minimal-2004-lts"
  source_image_project_id = ["ubuntu-os-cloud"]
  ssh_username            = "packer"
  use_os_login            = true

  service_account_email = "gce-imagebuilder@${var.project}.iam.gserviceaccount.com"
  startup_script_file   = "${path.root}/startup.sh"
  zone                  = "${var.region}-b"
}

build {
  sources = ["source.googlecompute.foo"]
}

I do have a Twingate tunnel setup to the private network, but don't think that should be the issue, esp. as I can use IAP to connect to other hosts. I did try setting use_iap = false, and that also doesn't work.

Wondering if it's a caching thing? If I run

% gcloud compute start-iap-tunnel packer-65bbd65d-ed4d-0ae5-609c-1edf09b225f9 22 --local-host-port=localhost:8834 --zone us-central1-b --project xxx --log-http

I can actually see it returning the information about the instance, but then still errors with

Testing if tunnel connection works.
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4047: 'Failed to lookup instance'].
wyardley commented 5 months ago

Further notes:

corinz commented 5 months ago

@wyardley Do you think this may be a small outage? We are experiencing the same issue at the same time...

wyardley commented 5 months ago

@corinz that would make a lot of sense if that were the issue. I was seeing the issue even with non-packer created VMs (but not existing ones), and was eventually able to get into one I created earlier today. So maybe it's just a really long lag? Yesterday, it was working consistently.