After good packer init: Error: Failed to initialize build "upcloud"

bbros-dev commented 3 years ago

Using Packer 1.7.0 and the following init file:

# init.pkr.hcl
packer {
  required_plugins {
    upcloud = {
      version = ">=v1.0.0"
      source  = "github.com/UpCloudLtd/upcloud"
    }
  }
}

The packer init succeeds:

$ packer init init.pkr.hcl
✔

However the build fails:

$ packer build \
>         -var-file=ubuntu-18.04-chef0-development-amd64-vars-uncommented.json \
>         -only=${BUILDER} \
>         ./ubuntu/ubuntu-18.04-chef0-development-amd64-uncommented.json 2>&1 | \
>         tee ${PACKER_BUILD_LOG_FILE}
Error: Failed to initialize build "upcloud"

error initializing builder 'upcloud': fork/exec
/usr/local/bin/packer-1.7.0/packer-builder-upcloud: no such file or directory

==> Wait completed after 47 microseconds

==> Builds finished but no artifacts were created.

maxfrei commented 3 years ago

@bbros-dev Packer init command forces you to use config files in pkr.hcl format containing required_plugins block

packer {
    required_plugins {
        upcloud = {
            version = ">=v1.0.0"
            source = "github.com/UpCloudLtd/upcloud"
        }
    }
}

But you still able to use json config files. For this, you need to install the plugin in an old way:

clone repo and run make install
or download package for your system (https://github.com/UpCloudLtd/packer-plugin-upcloud/releases/tag/v1.0.0) and put executable as /home/user/.packer.d/plugins/packer-plugin-upcloud

maxfrei commented 3 years ago

Packer provides a command to transform json to pkr.hcl:

$ packer hcl2_upgrade example.json
Successfully created example.json.pkr.hcl

To run hcl2_upgrade you still need to have upcloud plugin installed in an old wat (see the previous comment). Otherwise, you'll get something like

unknown builder type: "upcloud"

bbros-dev commented 3 years ago

Ahh, OK I thought the init was in place of the old plugin system.
So, if we're using JSON we don't get any benefit from the paker init ... and just set everything up like pre 1.7.0 - correct?

Yeah there is a substantial problem with moving from JSON to HCL2, we've posted a question to the Packer forum, to keep our momentum we have had to revert back to JSON in the meantime.

FYI - If the HCL2 limitation we encountered is permanent we'll likely never move away from JSON. Not sure if that affects your plugin development but thought it might be useful feedback

bbros-dev commented 3 years ago

Closing as operator error.

maxfrei commented 3 years ago

and just set everything up like pre 1.7.0 - correct?

yep, correct.

I hope Hashicorp will continue to support json config format so do we. Thank you for the feedback!

bbros-dev commented 3 years ago

Reopening because, after setting up the pieces correctly, packer build does not work for us and I think there are enough novel details for a doc update once we workout what was the root cause.

Install ~/.packer.d/plugins/packer-plugin-upcloud
Make local user the owner chown <>:<> ~/.packer.d/plugins/packer-plugin-upcloud
Grant liberal permissions chmod a+x ~/.packer.d/plugins/packer-plugin-upcloud
We encounter this error:

$ PACKER_LOG=1 packer build -debug         -var-file=ubuntu-18.04-chef0-development-amd64-vars-uncommented.json         -only=${BUILDER}         ./ubuntu/ubuntu-18.04-chef0-development-amd64-uncommented.json 2>&1 |         tee ${PACKER_BUILD_LOG_FILE}
2021/02/22 09:13:00 [INFO] Packer version: 1.7.0 [go1.15.8 linux amd64]
2021/02/22 09:13:00 [TRACE] discovering plugins in /usr/local/bin/packer-1.7.0
2021/02/22 09:13:00 [TRACE] discovering plugins in /home/<>/.packer.d/plugins
2021/02/22 09:13:00 [DEBUG] Discovered plugin: upcloud = /home/<>/.packer.d/plugins/packer-builder-upcloud
2021/02/22 09:13:00 [DEBUG] Discovered plugin: vultr = /home/<>/.packer.d/plugins/packer-builder-vultr
2021/02/22 09:13:00 [INFO] using external builders: [upcloud vultr]
2021/02/22 09:13:00 [DEBUG] Discovered plugin: upcloud = /home/<>/.packer.d/plugins/packer-plugin-upcloud
Error loading configuration: 

fork/exec /home/<>/.packer.d/plugins/packer-plugin-upcloud: permission denied

bbros-dev commented 3 years ago

We are not golang proficient, but we have a 10k lines log from strace packer build ... If you'd like to review that let me know where how to upload it.

maxfrei commented 3 years ago

@bbros-dev could you give some info please:

you were building plugin from source or downloaded a zip?
if from source, then which go version you were using?
if zip, then which one?
on which system you are running packer build?

Please send me the log alex.sviridov@upcloud.com

bbros-dev commented 3 years ago

Thanks for looking into this. I've emailed the strace log.

you were building plugin from source or downloaded a zip? I've emailed strace log.

From release zip file

if zip, then which one?

Linux amd64, sha256sum: 036c9aaa7749fd50028b952e9d1e92e5ca2eb07cce37b01099e12894b74b8b1e Our Chef recipe validates the checksum of the download file.

on which system you are running packer build?

Localhost:

$ uname -a
Linux desktop.local.lan 5.4.0-54-generic #60~18.04.1-Ubuntu SMP Fri Nov 6 17:25:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

maxfrei commented 3 years ago

@bbros-dev thanks for the log, it was useful. From your comment, I see that in ~/.packer.d/plugins directory you have both old packer-builder-upcloud and new packer-plugin-upcloud. You need only packer-plugin-upcloud. But that is not a core problem. From your comment:

2021/02/22 09:13:00 [DEBUG] Discovered plugin: upcloud = /home/<>/.packer.d/plugins/packer-builder-upcloud
2021/02/22 09:13:00 [DEBUG] Discovered plugin: vultr = /home/<>/.packer.d/plugins/packer-builder-vultr
2021/02/22 09:13:00 [INFO] using external builders: [upcloud vultr]
2021/02/22 09:13:00 [DEBUG] Discovered plugin: upcloud = /home/<>/.packer.d/plugins/packer-plugin-upcloud

Packer core 1.7.0 communicates with a plugin over RPC and needs to open unix socket for it. Example of output from my local system:

...
2021/02/22 20:31:04 Starting plugin: /home/alex/.packer.d/plugins/packer-plugin-upcloud []string{"/home/alex/.packer.d/plugins/packer-plugin-upcloud", "start", "builder", "-packer-default-plugin-name-"}
2021/02/22 20:31:04 Waiting for RPC address for: /home/alex/.packer.d/plugins/packer-plugin-upcloud
2021/02/22 20:31:04 Received unix RPC address for /home/alex/.packer.d/plugins/packer-plugin-upcloud: addr is /tmp/packer-plugin202908151
2021/02/22 20:31:04 packer-plugin-upcloud plugin: 2021/02/22 20:31:04 Plugin address: unix /tmp/packer-plugin202908151
2021/02/22 20:31:04 packer-plugin-upcloud plugin: 2021/02/22 20:31:04 Waiting for connection...
2021/02/22 20:31:04 packer-plugin-upcloud plugin: 2021/02/22 20:31:04 Serving a plugin connection...
2021/02/22 20:31:04 packer-plugin-upcloud plugin: 2021/02/22 20:31:04 [TRACE] starting builder -packer-default-plugin-name
...

Could you check does your user has write permission in /tmp?

maxfrei commented 3 years ago

For the test, you could try to export $TMPDIR and point it to the user's home directory. https://github.com/hashicorp/packer-plugin-sdk/blob/main/tmp/tmp.go#L7

bbros-dev commented 3 years ago

Yes, we saw an earlier golang issue report along those lines, and we tried export TMPDIR=~/tmp to no effect.

$ ls ~/tmp
✔

In case the strace log I sent was not from a run with TMPDIR exported, I've sent another strace log with TMPDIR=~/tmp and PACKER_LOG=1 exported.

bbros-dev commented 3 years ago

I should also note that without PACKER_LOG=1 the log file is empty.

bbros-dev commented 3 years ago

Apologies I overlooked this question

Could you check does your user has write permission in /tmp?

Yes the Packer operator has write access to ~/tmp and /tmp.

maxfrei commented 3 years ago

@bbros-dev yeah, I got it ;). Still trying to reproduce it on my local system. Meanwhile, could you perform one test? As I understood from the logs, currently you have three files in your plugin directory. Something like:

$ ls -l
-rwxr-xr-x 1 alex alex 25362432 Feb 22 20:21 packer-builder-upcloud
-rwxr-xr-x 1 alex alex 24735744 Feb 22 20:22 packer-builder-vultr
-rwxr-xr-x 1 alex alex 25362432 Feb 22 20:31 packer-plugin-upcloud

Could you remove packer-plugin-upcloud (leave only builders) and run builder? In my case, packer-builder-upcloud and packer-plugin-upcloud are the same file with different names. Is it the same in your case?

maxfrei commented 3 years ago

Or if I wrong and in your case packer-builder-upcloud is an old version of upcloud builder, then test the other way round. Leave new packer-plugin-upcloud and packer-builder-vultr in the .packer.d/plugins and run builder.

maxfrei commented 3 years ago

@bbros-dev and did you try to run packer with root privileges? Sorry, for debugging using you. But I really can't reproduce your error on my local machine (tried multiple ways).

bbros-dev commented 3 years ago

Hi @maxfrei, some details that will give context I think you are after in terms of versioning:

$ ls -la ~/.packer.d/plugins/
total 40
drwxrwxrwt 7 <user> <user> 4096 Feb 22 09:07 .
drwxrwxrwt 3 <user> <user> 4096 Feb 18 17:19 ..
drwxr-xr-x 3 <user> <user> 4096 Feb 21 13:45 github.com
lrwxrwxrwx 1 root  root    89 Feb 22 09:02 packer-builder-upcloud -> /home/<user>/.packer.d/plugins/upcloud-1.0.0/packer-plugin-upcloud_v1.0.0_x5.0_linux_amd64
lrwxrwxrwx 1 root  root    62 Feb 21 12:56 packer-builder-vultr -> /home/<user>/.packer.d/plugins/vultr-2.2.0/packer-builder-vultr
lrwxrwxrwx 1 root  root    89 Feb 22 09:07 packer-plugin-upcloud -> /home/<user>/.packer.d/plugins/upcloud-1.0.0/packer-plugin-upcloud_v1.0.0_x5.0_linux_amd64
drwxr-xr-x 2 root  root  4096 Feb 22 08:47 upcloud-1.0.0
drwxr-xr-x 2 root  root  4096 Nov 10 11:30 vultr
drwxr-xr-x 2 root  root  4096 Feb 21 02:58 vultr-2.1.0
drwxr-xr-x 2 root  root  4096 Feb 21 12:56 vultr-2.2.0

did you try to run packer with root privileges?

No we haven't tried that. Will do so now.

bbros-dev commented 3 years ago

Hmm, interesting, sudo -E packer build ... gets us past this and a build is running. So at least we have a workaround :)

It looks like there is a chown ... required for this plugin?

maxfrei commented 3 years ago

@bbros-dev strange, I replicated your directory structure in my system (with links and permissions, same packer/plugins versions) and still able to start packer build without sudo. I think there is something specific in your system. Do you have selinux/apparmor enabled?

bbros-dev commented 3 years ago

Do you have selinux/apparmor enabled?

No, I've never activated either of those.

Working on the earlier hunch....

sudo chown -R <user>:<user> /home/<user>/.packer.d/plugins/

seems to fix the issue! I'll dig into our install recipes and see just what they are doing - from memory installs use ark resource which is quite common and somthing we've always used so not sure why this plugin gets told it doesn't have the right permissions - which it appears is an ownership issue.

bbros-dev commented 3 years ago

This is the install recipe for the upcloud plugin so nothing funky is going on here in terms of permissions or ownership:

  upcloud_ver='1.0.0'
  unless ::File.exist?("/home/#{usr}/.packer.d/plugins/upcloud-#{upcloud_ver}/packer-builder-upcloud")
    ark 'Install Vultr Packer builder plugin binary' do
      action :put
      checksum '036c9aaa7749fd50028b952e9d1e92e5ca2eb07cce37b01099e12894b74b8b1e'
      mode '0755'
      name "upcloud-#{upcloud_ver}"
      path "/home/#{usr}/.packer.d/plugins"
      strip_components 0
      url "https://github.com/UpCloudLtd/packer-plugin-upcloud/releases/download/v#{upcloud_ver}/packer-plugin-upcloud_v#{upcloud_ver}_x5.0_linux_amd64.zip"
      version upcloud_ver
    end

    link "/home/#{usr}/.packer.d/plugins/packer-plugin-upcloud" do
      to "/home/#{usr}/.packer.d/plugins/upcloud-#{upcloud_ver}/packer-plugin-upcloud_v#{upcloud_ver}_x5.0_linux_amd64"
    end
  end

bbros-dev commented 3 years ago

Okay, very strange. When I reverted the ownership change:

sudo chown -R root:root /home/<user>/.packer.d/plugins/

Then packer build .. now runs without error.

The key likely lies in the strace files, but since I can no longer reproduce this I'm happy if you want to close the issue. I'll leave that decision to you.

Thank you for the prompt, patient and courteous help.

maxfrei commented 3 years ago

No problem, any time ;)

bbros-dev commented 3 years ago

@maxfrei, for a different client we encountered this error again. We opened a issue upstream, hashicorp/packer/issues/10749, because it initially seemed it wasn't this issue - we thought we had resolved it. Now we are not so sure....

We don't know the plugin/upstream code boundaries.... is it possible the permission error, seen here is related to a needed capability (setcap cap_block_suspend=ep /file/path), which cannot be set on a linked file?

If so would it be resolved by resolving the linked file before using it?

Perhaps you might have some insight to share with upstream?

maxfrei commented 3 years ago

@bbros-dev thanks for the report. I'll check it out.

bbros-dev commented 3 years ago

Workaround is: # sysctl -w fs.protected_symlinks=0.

Proper fix is to not install plugins manually -rather to leave this to the (new?) packer init functionality.

Perhaps update Readme sections on "Prebuilt binary" and "Installing from source" to explicitly state any use of symlinks while # sysctl -w fs.protected_symlinks=1 will break Packer functionality.

maxfrei commented 3 years ago

@bbros-dev I've updated readme, thanks. Unfortunately, packer init force you to upgrade your configs to pkr.hcl format and add required_plugins block.

UpCloudLtd / packer-plugin-upcloud

After good packer init: Error: Failed to initialize build "upcloud" #3