search

YaleUniversity / packer-plugin-goss

Goss Provisioner for Packer
MIT License
136 stars 45 forks source link

Issue when using HCL2 for packer templates #40

Closed Lirt closed 2 years ago

Lirt commented 3 years ago

Hello,

as reported in PR https://github.com/YaleUniversity/packer-provisioner-goss/pull/37 there is an issue with using packer-provisioner-goss with HCL2 packer config.

This issue happens even when I put anything into goss block, but also when I use empty goss block like this:

provisioner "goss" {
}

Error reported error looks following:

Error: Failed preparing provisioner-block "goss" ""

  on default.pkr.hcl line 98:
  (source code not available)

2 errors occurred:
    * unknown configuration key: "download_path"; raws is []interface
{}{map[string]interface {}{"packer_build_name":"ubuntu-18-04",
"packer_builder_type":"openstack"}, map[string]interface {}{"arch":interface
{}(nil), "download_path":interface {}(nil), "format":interface {}(nil),
"format_options":interface {}(nil), "goss_file":interface {}(nil),
"inspect":interface {}(nil), "password":interface {}(nil),
"remote_folder":interface {}(nil), "remote_path":interface {}(nil),
"retry_timeout":interface {}(nil), "skip_install":interface {}(nil),
"skip_ssl":interface {}(nil), "sleep":interface {}(nil), "tests":interface
{}(nil), "url":interface {}(nil), "use_sudo":interface {}(nil),
"username":interface {}(nil), "vars_env":interface {}(nil),
"vars_file":interface {}(nil), "vars_inline":interface {}(nil),
"version":interface {}(nil)}, map[string]interface {}(nil)} 

 and ctx data is map[interface {}]interface {}(nil)
    * unknown configuration key: "skip_install"; raws is []interface
{}{map[string]interface {}{"packer_build_name":"ubuntu-18-04",
"packer_builder_type":"openstack"}, map[string]interface {}{"arch":interface
{}(nil), "download_path":interface {}(nil), "format":interface {}(nil),
"format_options":interface {}(nil), "goss_file":interface {}(nil),
"inspect":interface {}(nil), "password":interface {}(nil),
"remote_folder":interface {}(nil), "remote_path":interface {}(nil),
"retry_timeout":interface {}(nil), "skip_install":interface {}(nil),
"skip_ssl":interface {}(nil), "sleep":interface {}(nil), "tests":interface
{}(nil), "url":interface {}(nil), "use_sudo":interface {}(nil),
"username":interface {}(nil), "vars_env":interface {}(nil),
"vars_file":interface {}(nil), "vars_inline":interface {}(nil),
"version":interface {}(nil)}, map[string]interface {}(nil)} 

 and ctx data is map[interface {}]interface {}(nil)
Lirt commented 3 years ago

Bump @fishnix

fishnix commented 3 years ago

@Lirt I think this is fixed with the latest releases, can you confirm? Thanks!

Lirt commented 3 years ago

It doesn't work for me. I tried second time with packer 1.7.0 where I updated to new syntax and used required_plugins like this:

packer {
  required_version = ">= 1.7.0, < 1.8"
  required_plugins {
    packer-provisioner-goss = {
      version = "= 3.0.0"
      source  = "github.com/YaleUniversity/packer-provisioner-goss"
    }
  }
}

But it fails with this error. I tried all kinds of combinations of name and source, but it doesn't work.

$ packer init --upgrade default.pkr.hcl
Error: Invalid plugin type

  on default.pkr.hcl line 43, in packer:
  43:     goss = {
  44:       version = "= 3.0.0"
  45:       source  = "github.com/YaleUniversity/packer-provisioner-goss"
  46:     }

Plugin source "github.com/yaleuniversity/packer-provisioner-goss" has a type
with the prefix "packer-", which isn't allowed because it would be redundant to
name a Packer plugin with that prefix. If you are the author of this plugin,
rename it to not include the prefix.

Looks like the repository has to be renamed to allow this to work -- source: https://www.packer.io/docs/plugins#source-addresses

Lirt commented 3 years ago

I'm actually not even able to run the 3.0.0 release with manual download as before. It says I'm missing target_os. But when I add it to provisioner section it reports the same error.

Error: Failed preparing provisioner-block "goss" ""

  on default.pkr.hcl line 102:
  (source code not available)

1 error occurred:
    * unknown configuration key: '"target_os"'
fishnix commented 3 years ago

@Lirt does the example work for you? I just tried it and it worked. I had to update the centos mirror btw:

   "variables": {
-    "iso_checksum": "101bc813d2af9ccf534d112cbe8670e6d900425b297d1a4d2529c5ad5f226372",
+    "iso_checksum": "b79079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6",
     "iso_checksum_type ": "sha256",
-    "iso_url": "http://mirror.net.cen.ct.gov/centos/7.8.2003/isos/x86_64/CentOS-7-x86_64-NetInstall-2003.iso"
+    "iso_url": "http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso"
   }
==> centos-7-x86_64: Retrieving Guest additions
==> centos-7-x86_64: Trying /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
==> centos-7-x86_64: Trying /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
==> centos-7-x86_64: /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso => /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
==> centos-7-x86_64: Retrieving ISO
==> centos-7-x86_64: Trying http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso
==> centos-7-x86_64: Trying http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso?checksum=sha256%3Ab79079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6
    centos-7-x86_64: CentOS-7-x86_64-NetInstall-2009.iso 575.00 MiB / 575.00 MiB [===================================================================] 100.00% 22s
==> centos-7-x86_64: http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso?checksum=sha256%3Ab79079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6 => /Users/fish/tmp/yale/packer-provisioner-goss/example/packer_cache/fa66676a0cae1de79d03c97423c2624780bea13a.iso
==> centos-7-x86_64: Starting HTTP server on port 8786
==> centos-7-x86_64: Creating virtual machine...
==> centos-7-x86_64: Creating hard drive...
==> centos-7-x86_64: Mounting ISOs...
    centos-7-x86_64: Mounting boot ISO...
==> centos-7-x86_64: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 2912)
==> centos-7-x86_64: Executing custom VBoxManage commands...
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615214848 --paravirtprovider kvm
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615214848 --nictype1 virtio
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615214848 --memory 1024
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615214848 --cpus 1
==> centos-7-x86_64: Starting the virtual machine...
    centos-7-x86_64: The VM will be run headless, without a GUI. If you want to
    centos-7-x86_64: view the screen of the VM, connect via VRDP without a password to
    centos-7-x86_64: rdp://127.0.0.1:5970
==> centos-7-x86_64: Waiting 10s for boot...
==> centos-7-x86_64: Typing the boot command...
==> centos-7-x86_64: Using ssh communicator to connect: 127.0.0.1
==> centos-7-x86_64: Waiting for SSH to become available...
Cancelling build after receiving interrupt
==> centos-7-x86_64: Cleaning up floppy disk...
==> centos-7-x86_64: Deregistering and deleting VM...
==> centos-7-x86_64: Deleting output directory...
Build 'centos-7-x86_64' errored after 2 minutes 24 seconds: Build was cancelled.

==> Wait completed after 2 minutes 24 seconds
Cleanly cancelled builds after being interrupted.
➜  example git:(fishnix-gomod-v3) ✗ packer build packer.json
centos-7-x86_64: output will be in this color.

==> centos-7-x86_64: Retrieving Guest additions
==> centos-7-x86_64: Trying /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
==> centos-7-x86_64: Trying /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
==> centos-7-x86_64: /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso => /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
==> centos-7-x86_64: Retrieving ISO
==> centos-7-x86_64: Trying http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso
==> centos-7-x86_64: Trying http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso?checksum=sha256%3Ab79079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6
==> centos-7-x86_64: http://mirror.net.cen.ct.gov/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-NetInstall-2009.iso?checksum=sha256%3Ab79079ad71cc3c5ceb3561fff348a1b67ee37f71f4cddfec09480d4589c191d6 => /Users/fish/tmp/yale/packer-provisioner-goss/example/packer_cache/fa66676a0cae1de79d03c97423c2624780bea13a.iso
==> centos-7-x86_64: Starting HTTP server on port 8289
==> centos-7-x86_64: Creating virtual machine...
==> centos-7-x86_64: Creating hard drive...
==> centos-7-x86_64: Mounting ISOs...
    centos-7-x86_64: Mounting boot ISO...
==> centos-7-x86_64: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 2992)
==> centos-7-x86_64: Executing custom VBoxManage commands...
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615215012 --paravirtprovider kvm
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615215012 --nictype1 virtio
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615215012 --memory 1024
    centos-7-x86_64: Executing: modifyvm packer-centos-7-x86_64-1615215012 --cpus 1
==> centos-7-x86_64: Starting the virtual machine...
==> centos-7-x86_64: Waiting 10s for boot...
==> centos-7-x86_64: Typing the boot command...
==> centos-7-x86_64: Using ssh communicator to connect: 127.0.0.1
==> centos-7-x86_64: Waiting for SSH to become available...
==> centos-7-x86_64: Connected to SSH!
==> centos-7-x86_64: Uploading VirtualBox version info (6.1.12)
==> centos-7-x86_64: Uploading VirtualBox guest additions ISO...
==> centos-7-x86_64: Provisioning with shell script: scripts/vagrant.sh
    centos-7-x86_64: Configuring vagrant-specific stuff ...
    centos-7-x86_64: Loaded plugins: fastestmirror
    centos-7-x86_64: Determining fastest mirrors
    centos-7-x86_64:  * base: centos.mirrors.hoobly.com
    centos-7-x86_64:  * extras: mirror.centos.iad1.serverforge.org
    centos-7-x86_64:  * updates: mirrors.tripadvisor.com
    centos-7-x86_64: Package wget-1.14-18.el7_6.1.x86_64 already installed and latest version
    centos-7-x86_64: Nothing to do
==> centos-7-x86_64: /tmp/script_7446.sh: line 10: /etc/init.d/iptables: No such file or directory
==> centos-7-x86_64: error reading information on service iptables: No such file or directory
    centos-7-x86_64: Changing password for user vagrant.
    centos-7-x86_64: passwd: all authentication tokens updated successfully.
==> centos-7-x86_64: --2021-03-08 10:05:50--  https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub
==> centos-7-x86_64: Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.108.133, ...
==> centos-7-x86_64: Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
==> centos-7-x86_64: HTTP request sent, awaiting response... 200 OK
==> centos-7-x86_64: Length: 409 [text/plain]
==> centos-7-x86_64: Saving to: ‘/home/vagrant/.ssh/authorized_keys’
==> centos-7-x86_64:
==> centos-7-x86_64:      0K                                                       100% 41.6M=0s
==> centos-7-x86_64:
==> centos-7-x86_64: 2021-03-08 10:05:50 (41.6 MB/s) - ‘/home/vagrant/.ssh/authorized_keys’ saved [409/409]
==> centos-7-x86_64:
==> centos-7-x86_64: Provisioning with Goss
==> centos-7-x86_64: Configured to run on Linux
    centos-7-x86_64: Creating directory: /tmp/goss
    centos-7-x86_64: Installing Goss from, https://github.com/aelsabbahy/goss/releases/download/v0.3.9/goss-linux-amd64
    centos-7-x86_64: Downloading Goss to /tmp/goss-0.3.9-linux-amd64
==> centos-7-x86_64:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
==> centos-7-x86_64:                                  Dload  Upload   Total   Spent    Left  Speed
==> centos-7-x86_64: 100   622  100   622    0     0   1759      0 --:--:-- --:--:-- --:--:--  1757
==> centos-7-x86_64: 100 10.0M  100 10.0M    0     0  6192k      0  0:00:01  0:00:01 --:--:-- 17.6M
    centos-7-x86_64: goss version v0.3.9
==> centos-7-x86_64: Uploading goss tests...
    centos-7-x86_64: Uploading goss/goss.yaml
==> centos-7-x86_64:
==> centos-7-x86_64:
==> centos-7-x86_64:
==> centos-7-x86_64: Running goss tests...
==> centos-7-x86_64: Running GOSS render command: cd /tmp/goss &&  /tmp/goss-0.3.9-linux-amd64 --gossfile goss.yaml   render > /tmp/goss-spec.yaml
==> centos-7-x86_64: Goss render ran successfully
==> centos-7-x86_64: Running GOSS render debug command: cd /tmp/goss &&  /tmp/goss-0.3.9-linux-amd64 --gossfile goss.yaml   render -d > /tmp/debug-goss-spec.yaml
==> centos-7-x86_64: Goss render debug ran successfully
==> centos-7-x86_64: Running GOSS validate command: cd /tmp/goss &&   /tmp/goss-0.3.9-linux-amd64 --gossfile goss.yaml   validate --retry-timeout 5s --sleep 2s
    centos-7-x86_64: ..........................
    centos-7-x86_64:
    centos-7-x86_64: Total Duration: 0.009s
    centos-7-x86_64: Count: 26, Failed: 0, Skipped: 0
==> centos-7-x86_64: Goss validate ran successfully
==> centos-7-x86_64:
==> centos-7-x86_64:
==> centos-7-x86_64:
==> centos-7-x86_64: Downloading spec file and debug info
    centos-7-x86_64: Downloading Goss specs from, /tmp/goss-spec.yaml and /tmp/debug-goss-spec.yaml to current dir
==> centos-7-x86_64: Provisioning with shell script: scripts/cleanup.sh
    centos-7-x86_64: Cleaning up ...
    centos-7-x86_64: Loaded plugins: fastestmirror
==> centos-7-x86_64: No Match for argument: gtk2
==> centos-7-x86_64: No Match for argument: libX11
==> centos-7-x86_64: No Match for argument: hicolor-icon-theme
==> centos-7-x86_64: No Match for argument: avahi
==> centos-7-x86_64: No Match for argument: bitstream-vera-fonts
    centos-7-x86_64: No Packages marked for removal
    centos-7-x86_64: Loaded plugins: fastestmirror
    centos-7-x86_64: Cleaning repos: base extras updates
    centos-7-x86_64: Cleaning up list of fastest mirrors
==> centos-7-x86_64: Provisioning with shell script: scripts/zerodisk.sh
    centos-7-x86_64: Zeroing out free space ...
==> centos-7-x86_64: dd: error writing ‘/EMPTY’: No space left on device
==> centos-7-x86_64: 6671+0 records in
==> centos-7-x86_64: 6670+0 records out
==> centos-7-x86_64: 6994001920 bytes (7.0 GB) copied, 4.63503 s, 1.5 GB/s
==> centos-7-x86_64: Gracefully halting virtual machine...
==> centos-7-x86_64: Preparing to export machine...
    centos-7-x86_64: Deleting forwarded port mapping for the communicator (SSH, WinRM, etc) (host port 2992)
==> centos-7-x86_64: Exporting virtual machine...
    centos-7-x86_64: Executing: export packer-centos-7-x86_64-1615215012 --output img_centos_7_virtualbox/packer-centos-7-x86_64-1615215012.ovf
==> centos-7-x86_64: Cleaning up floppy disk...
==> centos-7-x86_64: Deregistering and deleting VM...
==> centos-7-x86_64: Running post-processor: vagrant
==> centos-7-x86_64 (vagrant): Creating a dummy Vagrant box to ensure the host system can create one correctly
==> centos-7-x86_64 (vagrant): Creating Vagrant box for 'virtualbox' provider
    centos-7-x86_64 (vagrant): Copying from artifact: img_centos_7_virtualbox/packer-centos-7-x86_64-1615215012-disk001.vmdk
    centos-7-x86_64 (vagrant): Copying from artifact: img_centos_7_virtualbox/packer-centos-7-x86_64-1615215012.ovf
    centos-7-x86_64 (vagrant): Renaming the OVF to box.ovf...
    centos-7-x86_64 (vagrant): Compressing: Vagrantfile
    centos-7-x86_64 (vagrant): Compressing: box.ovf
    centos-7-x86_64 (vagrant): Compressing: metadata.json
    centos-7-x86_64 (vagrant): Compressing: packer-centos-7-x86_64-1615215012-disk001.vmdk
Build 'centos-7-x86_64' finished after 16 minutes 38 seconds.

==> Wait completed after 16 minutes 38 seconds

==> Builds finished. The artifacts of successful builds are:
--> centos-7-x86_64: 'virtualbox' provider box: packer_centos-7-x86_64_virtualbox.box
Lirt commented 3 years ago

I am using HCL2 language. The example is in json. The issue happens even with empty goss provisioner. Here is my minimal example:

# Packer block
packer {
  required_version = ">= 1.7.0, < 1.8"
}

source "openstack" "ubuntu-18-04" {
  flavor                  = "my-flavor"
  floating_ip_network     = "my-network"
  image_tags              = ["my-image-tag"]
  network_discovery_cidrs = ["10.10.0.0/24"]
  region                  = "RegionOne"
  security_groups         = ["my-secgroup"]
  source_image_filter {
    filters {
      name = "ubuntu-18.04-minimal-raw"
      tags       = ["production"]
      visibility = "private"
    }
  }
  ssh_timeout  = "2h"
  ssh_username = "ubuntu"
}

# Kubernetes 1.18
build {
  name = "ubuntu-18.04_kubernetes-v1.18.16"

  source "openstack.ubuntu-18-04" {
    name = "openstack-ubuntu-18.04-kube-v1.18.16"
    image_name = "kube-v1.18.16"
  }

  # Goss tests
  provisioner "goss" {
    arch           = "amd64"
    download_path  = "/tmp/goss-0.3.15-linux-amd64"
    # If Inspect mode is on, Goss reports failure but doesn't fail
    inspect        = false
    # Releases - https://github.com/YaleUniversity/packer-provisioner-goss/releases
    url            = "https://github.com/aelsabbahy/goss/releases/download/v0.3.15/goss-linux-amd64"
    version        = "0.3.15"
    # Test output format + options
    format         = "json"
    format_options = "pretty"
    target_os      = "Linux"
    # Paths
    tests          = [
      "goss-tests/goss-command.yaml",
      "goss-tests/goss-kernel-params.yaml",
      "goss-tests/goss-package.yaml",
      "goss-tests/goss-service.yaml",
      "goss-tests/goss.yaml",
    ]
    goss_file      = "goss.yaml"
    vars_file      = "goss-tests/goss-vars.yaml"
    vars_inline    = {
      OS          = "ubuntu"
      # Provider variable is not used yet, but can be in the future
      # to specify special goss tests for our platform
      PROVIDER    = "openstack"
      K8S_VERSION = "1.18"
    }
    use_sudo       = true
  }
}

Then error:

  on asdf.pkr.hcl line 35:
  (source code not available)

1 error occurred:
    * unknown configuration key: '"target_os"'

Or did I misunderstand you @fishnix?

fishnix commented 3 years ago

ah, now I understand @Lirt, sorry for being dense. I'll look into this.

marco-m-pix4d commented 2 years ago

Hello @fishnix, @Lirt, for my understanding: does the merge of #46 also fixes this issue? if yes, this can be closed.

Lirt commented 2 years ago

Hi @marco-m-pix4d I can recheck.

Lirt commented 2 years ago

@marco-m-pix4d latest release v3.1.2 works well for me.

I will close this, but can you please also look at this issue https://github.com/YaleUniversity/packer-provisioner-goss/issues/58? It would help me very much to make this work.

marco-m-pix4d commented 2 years ago

Hello @Lirt, I am sorry I might have mislead you, I am just a user of this project, not a maintainer.