search

hashicorp / packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
http://www.packer.io
Other
15.05k stars 3.32k forks source link

Unable to get a manually installed plugin to load #12749

Open jjo93sa opened 10 months ago

jjo93sa commented 10 months ago

Community Note

Overview of the Issue

We are unable to make a manually-installed plugin work. Our scenario is that we have a custom version of the OpenStack plugin, which we want to test alongside the stock Ansible plugin specified in the packer block:

packer {
  required_plugins {
    ansible = {
      source  = "github.com/hashicorp/ansible"
      version = "~> 1"
    }
  }
}

Our custom build of the OpenStack plugin is installed in the same directory (/usr/local/bin) as the Packer binary itself. According to the trace, the plugin is "discovered" but then an error is produced that the file just found doesn't exist, a configuration error.

What we've tried to fix this:

Reproduction Steps

Clone plugin repo:

git clone  https://github.com/hashicorp/packer-plugin-openstack.git

Build repo:

go build

Copy built binary to directory where Packer binary lives:

cp packer-plugin-openstack /usr/local/bin/
chmod 0777 packer-plugin-openstack
chown root:root packer-plugin-openstack

Execute Packer build:

# Packer v1.9.4 requires init to fetch plugins
export PR_CFG="packer-runner.pkr.hcl"
PACKER_PLUGIN_PATH="/usr/local/bin" PACKER_LOG=1 packer init "${PR_CFG}"

PACKER_PLUGIN_PATH="/usr/local/bin" PACKER_LOG=1 packer plugins installed

PACKER_PLUGIN_PATH="/usr/local/bin" PACKER_LOG=1 packer validate "${PR_CFG}"

PACKER_PLUGIN_PATH="/usr/local/bin" PACKER_LOG=1 packer build "${PR_CFG}"

Packer version

v1.9.4

Simplified Packer Template

TBC

Operating system and Environment details

Executing in a GitLab runner Docker container

Alpine Linux: alpine:3.15.4

Log Fragments and crash.log files

Note in the following there's no output from the packer plugins installed command, so it seems the init must be failing?

<snip>
2023/12/06 15:53:54 [INFO] Packer version: 1.9.4 [go1.20.7 linux amd64]
2023/12/06 15:53:54 [TRACE] discovering plugins in /usr/local/bin
2023/12/06 15:53:54 [INFO] Discovered potential plugin: openstack = /usr/local/bin/packer-plugin-openstack
 Error loading configuration: 
fork/exec /usr/local/bin/packer-plugin-openstack: no such file or directory
<snap>

At that point the runner process fails and cleans up.

lbajolet-hashicorp commented 10 months ago

Hi @jjo93sa,

Since you're using required_plugins you won't be able to use the plugin located alongside the packer executable, instead, that method of loading only considers what's in the configuration's plugin directory. It can take several forms depending on the environment, I encourage you to take a look at our docs for more information.

Here's some links that may be useful to understand the problem:

Also, if you want to manually install a plugin to your plugin directory, with Packer v1.10.0 (released yesterday), we added a --path flag to the packer plugins install command, which installs a binary to the plugin directory with the right name/hierarchy as packer expects for required_plugins.

For your case, that'd be something like packer plugins install --path "/usr/local/bin/packer-plugin-openstack" "github.com/hashicorp/openstack", which will automatically install the plugin in the right place.

As for packer plugins installed, it only reports plugins that are installed through packer init or packer plugins install, i.e. those that match the convention documented in plugin loading workflow.

Hope that helps!

jjo93sa commented 10 months ago

@lbajolet-hashicorp Thanks for your reply. So, if I understand correctly, one method of including plugins, i.e. the resource block:

  required_plugins {
    ansible = {
      source  = "github.com/hashicorp/ansible"
      version = "~> 1"
    }
  }
}

... excludes the use of "local" plugins, for example those next to the Packer binary? Even if the two methods load two different plugins? I couldn't see anything about that in the documents?

Is there anyway to install a manually built plugin in the plugins directory, such that both can be mentioned in the required_plugins directory?

I experimented this morning by removing the required_plugins block, and downloading the stock (v1.1.1) OpenStack and Anisble plugins, placing them in /usr/bin/local and indeed that seems to work OK. Great: progress!

However, if I do the following:

  1. git clone https://github.com/hashicorp/packer-plugin-openstack.git
  2. git checkout tags/v1.1.1
  3. go build
  4. Copy the built binary in place of the stock now downloaded from GitHub
  5. Run Packer build (which now has stock Ansible and locally built OpenStack plugins)

I see the same error as I first reported:

023/12/07 09:56:25 [INFO] Packer version: 1.9.4 [go1.20.7 linux amd64]
2023/12/07 09:56:25 [TRACE] discovering plugins in /usr/local/bin
2023/12/07 09:56:25 [INFO] Discovered potential plugin: ansible = /usr/local/bin/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64
2023/12/07 09:56:25 [INFO] Discovered potential plugin: openstack = /usr/local/bin/packer-plugin-openstack_v1.1.1_x5.0_linux_amd64
2023/12/07 09:56:25 found external [-packer-default-plugin-name- local] provisioner from ansible plugin
 Error loading configuration: 
fork/exec /usr/local/bin/packer-plugin-openstack_v1.1.1_x5.0_linux_amd64: no such file or directory

So the only thing I can conclude is that I'm doing something wrong in the build of the OpenStack plugin, and the configuration error results from that rather than the placement of the plugin? The plugin documents only mention go build and no other commands or dependencies.

jjo93sa commented 10 months ago

As an aside, I think there is a typo in the Manual Installation guide

For example, if your configuration directory is located in ~/.config/packer, you can copy the binary to ~/.config/plugins/packer-plugin-NAME, and Packer will be able to load it afterwards.

Shouldn't that read: ~/.config/packer/plugins/packer-plugin-NAME?

nywilken commented 10 months ago

@lbajolet-hashicorp Thanks for your reply. So, if I understand correctly, one method of including plugins, i.e. the resource block:

  required_plugins {
    ansible = {
      source  = "github.com/hashicorp/ansible"
      version = "~> 1"
    }
  }
}

... excludes the use of "local" plugins, for example those next to the Packer binary? Even if the two methods load two different plugins? I couldn't see anything about that in the documents?

Your understanding is correct. The plugin information described within the required _plugins block provides additional metadata to Packer for loading a specific plugin. Metadata that is provided through the plugin directory structure and filename. Local plugins that sit along side the executable do not provide this information and are ignored when a template contains a required_plugins block.

The use of required_plugins with Packer's historical plugin loading has been a pain point we are working to resolve. We attempted to capture the gotcha of required_plugins and local binaries here. Do you have suggestions on how we can make it clearer?

I see the same error as I first reported:

023/12/07 09:56:25 [INFO] Packer version: 1.9.4 [go1.20.7 linux amd64]
2023/12/07 09:56:25 [TRACE] discovering plugins in /usr/local/bin
2023/12/07 09:56:25 [INFO] Discovered potential plugin: ansible = /usr/local/bin/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64
2023/12/07 09:56:25 [INFO] Discovered potential plugin: openstack = /usr/local/bin/packer-plugin-openstack_v1.1.1_x5.0_linux_amd64
2023/12/07 09:56:25 found external [-packer-default-plugin-name- local] provisioner from ansible plugin
 Error loading configuration: 
fork/exec /usr/local/bin/packer-plugin-openstack_v1.1.1_x5.0_linux_amd64: no such file or directory

So the only thing I can conclude is that I'm doing something wrong in the build of the OpenStack plugin, and the configuration error results from that rather than the placement of the plugin? The plugin documents only mention go build and no other commands or dependencies.

I believe there may be an issue with the compiled binary. The error fork/exec ... no such file or directory is the an error you will can when a binary expects / requires glibc.

Before copying the binary you can validate that it works by running ./packer-plugin-openstack describe

~>  ./packer-plugin-openstack describe
{"version":"1.1.1","sdk_version":"0.4.0","api_version":"x5.0","builders":["-packer-default-plugin-name-"],"post_processors":[],"provisioners":[],"datasources":[]}

I see you are running Alpine which wont have glibc I believe you need to use musl libc.

If the describe command fails as I expect it would can you try building with CGO_ENABLED=0.

CGO_ENABLED=0 go build

When we build our binaries for releases we disable cgo on all binaries so that they are statically linked and don't dependent on the system's glibc. I don't have direct experience with building on Alpine if disabling CGO is not the option. Maybe @lbajolet-hashicorp could provide more info if this turns out to be the issue.

Is there anyway to install a manually built plugin in the plugins directory, such that both can be mentioned in the required_plugins directory?

In Packer 1.10.0 there packer plugins install command has been updated to support the installation of locally built binaries so that they work with required_plugins. Taking your current workflow and using Packer 1.10.0 you can run the following to get things working as expected.

  1. git clone https://github.com/hashicorp/packer-plugin-openstack.git
  2. git checkout tags/v1.1.1
  3. go build
  4. packer plugins install --path ./packer-plugin-openstack github.com/hashicorp/openstack
  5. Run Packer build (which now has stock Ansible and locally built OpenStack plugins)

However, this would install the plugin into the Packer Plugind directory and not /usr/local/bin.

Is there a reason why it needs to be in /usr/local/bin?

jjo93sa commented 10 months ago

@nywilken Thanks for your help:

Looking at the documentation link you provided, I'd say this section was misleading, at least in my mind:

This logic however is ignored when plugins are defined in required_plugins blocks; instead, for every plugin required in this way, Packer will only consider them if they're installed in Packer's plugin directory, under a directory hierarchy that matches the source, with the plugin name respecting a convention.

I think the problem here is the "for every plugin" part. That, again to me, makes it seem that the following part of the sentence only applies to plugins in the required_plugins block, not all plugins. So, I think something like this would more accurately represent the situation (as I understand it):

Note, however, that it is not possible to mix "local" plugins alongside those defined in a required_plugins block; the two mechanisms are mutually exclusive, if a required_plugins block exists in a template, no local plugins will be used.

HTH

Some good news:

If the describe command fails as I expect it would can you try building with CGO_ENABLED=0.

CGO_ENABLED=0 go build

This worked! (Alongside removing the required_plugins block and no longer executing packer init). Thanks a lot for your help.

I had seen the packer plugins install command in v1.10.0. Unfortunately, the licence change to BSL in v1.10.0 means we're not permitted to use that version.

Is there a reason why it needs to be in /usr/local/bin?

No, there's no reason why the plugins need to be in that specific location, I can place them anywhere, accepting the limitations of the Packer version we're using.

torarnv commented 6 months ago

One use case I have is working on a Packer template that contains required_plugins, but where I want to use a locally developed/modified plugin.

I can install the development version via packer plugins install --path, but since it's a pre-release version.InitializePluginVersion("1.2.3", "dev") results in the plugin being installed as packer-plugin-foo_v1.2.3-dev_x5.0_darwin_arm64.

Which does not match a Packer template of:

packer {
  required_plugins {
    foo = {
      version = ">= 1.2.0"
      source = "github.com/someting/foo"
    }
  }
}

Right now I'm working around it by either:

  1. Commenting out the required_plugins in the template while developing
  2. Changing the version requirement in the template to 1.2.3-dev during development
  3. Passing "" to the prerelease argument of InitializePluginVersion to avoid adding any suffix

Neither of these are great workarounds, so it would be nice if there was an easier way to use development versions with required_plugins.

torarnv commented 6 months ago

To be clear I want to:

Having a pre-release suffix on the installed development version seems good/useful, to be able to distinguish a released version from a pre-release, so ideally the solution is something that doesn't require modifying the Packer template, and doesn't require faking non-pre-releases in the plugin 😅

lbajolet-hashicorp commented 6 months ago

Hi @torarnv,

Timing is 10/10 on your updating this issue, we've just released v1.11.0-alpha that should address your concerns. Namely you can install -dev binaries with the packer plugins install --path command, and it will be loaded by Packer, even with a required_plugins statement in your template, assuming the following is true:

  1. The plugin's version is the highest that matches your constraints
  2. No release version alternative is installed, otherwise it will have precedence

I believe this should fill-in the blanks for your use-case, please let us know if there's something unclear or unintuitive with this workflow!

torarnv commented 6 months ago

Damn @lbajolet-hashicorp , you just keep saving my day! 🤩 This is perfect, will test the alpha and give feedback ❤️

No release version alternative is installed, otherwise it will have precedence

Does this mean that if 0.0.4 is installed and I install 0.0.5-dev, I will not get my dev version? Or does it only apply if I have both 0.0.5 and 0.0.5-dev installed?

lbajolet-hashicorp commented 6 months ago

That's only if you have 0.0.5-dev and 0.0.5 installed yep; in this case 0.0.5 will mask 0.0.5-dev, so the latter won't be loadable anymore. If you have 0.0.4 and 0.0.5-dev, assuming your constraints allow it, 0.0.5-dev will have precedence over 0.0.4.

If you choose to also, we included a --ignore-prerelease-plugins flag to circumvent this behaviour and forbid loading -dev binaries for both the validate and build commands.

torarnv commented 6 months ago

That sounds like a perfect fit, thanks!

torarnv commented 6 months ago

Having played with this new feature a bit, one thing that is a bit unintuitive is the override logic of 0.0.5 being preferred over 0.0.5-dev.

The typical case is that a plugin is released, and then installed, as version 0.0.5 (git describe giving v0.0.5).

Then someone clones the repo of the plugin to fix an issue or add a feature, and want to test the changes before submitting them upstream. Maybe they have a dirty working tree, or maybe they have made a commit. Their local git describe would be e.g. v0.0.5-1-gb1299dd, and produce v0.0.5-dev. Which would not be picked up.

With the current approach, the only way seems to be to preemptively modify the version of the plugin you're working on. Even if you're not the one in charge of releasing 😄 And to make sure that version change doesn't end up in the commit you're submitting upstream.

Do you have any tips to improve this process? Perhaps 0.0.5-dev should override 0.0.5 ? Possibly via a optional flag (--prefer-prerelease-plugins)?

Once the change has been tested and merged upstream, the maintainer of the plugin releases them as e.g. 0.0.6 (git describe giving v0.0.6), at which point it would of course make sense to override v0.0.5-dev.

Thanks!

torarnv commented 6 months ago

I was thinking I could work around this by turning git describe --tags --long --dirty v0.0.5-1-gb1299dd into a version number v0.0.5.1-gb1299dd, but of course that doesn't work for v0.0.5-0-gb1299dd-dirty, where you've made some changes locally on top of a released version, but haven't committed them yet.

Would be nice to be able to test the plugin in this scenario by installing it (into possibly a custom PACKER_PLUGIN_PATH).

torarnv commented 6 months ago

Ignoring the corner case of v0.0.5-0-gb1299dd-dirty this still doesn't seem to be a viable workaround right now:

❯ make && ./packer-plugin-ipsw describe
go build -ldflags="-X 'main.GitDescribe=v0.0.12-3-g9724548'" -o packer-plugin-ipsw
{"version":"0.0.12.3-g9724548","sdk_version":"0.4.0","api_version":"x5.0","builders":[],"post_processors":[],"provisioners":[],"datasources":["-packer-default-plugin-name-"]}

❯ make install
go build -ldflags="-X 'main.GitDescribe=v0.0.12-3-g9724548'" -o packer-plugin-ipsw
packer plugins install --path packer-plugin-ipsw github.com/torarnv/ipsw
Error: Invalid version

Packer can only install plugin releases with this command (ex: 1.0.0) or
development pre-releases (ex: 1.0.0-dev), the binary's reported version is
"0.0.12.3-g9724548"

Could we relax https://github.com/hashicorp/packer/blob/main/command/plugins_install.go#L260 so that it doesn't care what exactly the pre-release format is? The general Packer plugin requirements for versioning doesn't seem to require "dev" based on https://github.com/hashicorp/packer/blob/main/version/version.go#L27

torarnv commented 6 months ago

I tried to please Packer by hard-coding the pre-release version to dev, while still producing 0.0.5.1001 as the version to keep it higher than the 0.0.5:

https://gist.github.com/torarnv/fbcfd990e84e510a2e5b5a8f285b8b5d#file-stdin-patch-L44

But the Packer plugin version regex only allows major.minor.patch version:

https://github.com/hashicorp/packer/blob/main/packer/plugin-getter/plugins.go#L103

😅

As we use http://github.com/hashicorp/go-version for the versions, could we perhaps relax this? It seems to allow more than three version components based on https://github.com/hashicorp/go-version/blob/main/version_test.go#L69 ?

lbajolet-hashicorp commented 6 months ago

Hi @torarnv,

Thanks for the feedback, there's a lot to process so I'll try to respond to each point one-by-one, but forgive me if there's something I don't address in this message, feel free to press some more!

Having played with this new feature a bit, one thing that is a bit unintuitive is the override logic of 0.0.5 being preferred over 0.0.5-dev.

The typical case is that a plugin is released, and then installed, as version 0.0.5 (git describe giving v0.0.5). [...] Perhaps 0.0.5-dev should override 0.0.5 ?

Regarding this one, I can understand your issue, but I don't think we'll change the implementation regarding versioning, our assumption is that version is generally updated in this fashion (I'll take your example to illustrate):

  1. v0.0.5 is released - a commit with Version = "0.0.5" is pushed on the repo, and tagged, so it triggers the job that performs the release
  2. v0.0.6-dev is pushed on top - a commit with Version = "0.0.6" and VersionPrerelease = "dev" is pushed on main: from now on we're building v0.0.6-dev each time

Both actions should happen when the release is made. This is the intended workflow for plugin development, and one approach we use for the plugins we develop internally (ex: https://github.com/hashicorp/packer-plugin-amazon/pull/464, https://github.com/hashicorp/packer-plugin-azure/pull/377, etc.). This is also what the attribute's documentation indicates.

I can understand if this feels kinda weird though, especially if you're used to bumping the version at release, and not beforehand; the workflows are deeply incompatible.

The flag you're suggesting --prefer-prerelease-plugins could be an idea, though I'm not certain this is where we want to go right now. Our goal with that change is essentially to make the development process uniform so that it becomes easier and more predictable how Packer behaves with respect to plugin discovery and loading, changing this order with extra flags may in my opinion make it a bit confusing in several regards. For example: one repo may adopt our workflow, others will adopt a similar workflow as yours, so the usage of the flag will need to be documented in that plugin's README for example. Also, having this as a flag means that if testing/building multiple plugins at the same time, and their versioning conventions are different, this will make it impossible to test without tinkering with either's Version and re-building.

I'm open to discussing this of course, but for those reasons, I'm enclined to adopt a convention, which could be what you suggested to be clear.

Do you have any tips to improve this process?

My advice would be, if the version.Version is 0.0.5 in your local tree, you can always change it to 0.0.6, and this 0.0.6-dev plugin will now have precedence over 0.0.5, so it will be loaded in your case.

Alternatively, you can remove the installed v0.0.5 version of the plugin, and install the one you're building instead, this way there won't be any conflict.

I was thinking I could work around this by turning [...] a version number v0.0.5.1-gb1299dd

Unfortunately in this case the -gb1299dd part will be incompatible with our version checks, since we restricted pre-releases to only dev. We opted to go this route in order to have a simple ordering, without having to specify an order that took into account alpha/beta/rc, etc.

I'd love to get your thoughts on that process, is there something that'd prevent you from adopting this workflow?

The general Packer plugin requirements for versioning doesn't seem to require "dev" based on https://github.com/hashicorp/packer/blob/main/version/version.go#L27

Good call, this comment does not represent how this works nowadays it seems. This should be updated in the scaffolding. Though this could change depending on the previous discussion point.

I tried to please Packer by hard-coding the pre-release version to dev, while still producing 0.0.5.1001 as the version to keep it higher than the 0.0.5:

https://gist.github.com/torarnv/fbcfd990e84e510a2e5b5a8f285b8b5d#file-stdin-patch-L44

But the Packer plugin version regex only allows major.minor.patch version

Indeed for now it is how it behaves, we're experimenting on some changes to that workflow however with this PR, be advised though, it is bound to change a bit. I'll let you know when the PR is in a testable state so you can let us know if this improves the experience for you?

Sorry for the wall of text answer here :sweat_smile:, let's continue discussing this and get to a point where we'll all be happy with how Packer behaves :)

PS: regarding those changes and the rationale behind them, we've published a blog post detailing our thought process, please feel free to give it a read, we'd be interested to hear what you think about this.

torarnv commented 6 months ago

Thank you very much for the detailed reply @lbajolet-hashicorp 🙌🏻

I can fully appreciate not wanting to change the version matching logic to prefer 0.0.5-dev over 0.0.5 😃 The motivation for the suggestion was, as you touch upon, that plugins typically bump version on release, to the version released. A workflow as you describe where the bump to the next version happens when releasing the previous would fix that. Perhaps the Packer scaffolding template could include Makefile and/or goreleaser magic/best practices to do releases that way, to promote plugins to adopt that flow (if it doesn't already, I might be missing something).

Having thought about the --prefer-prerelease-plugins some more, I think it's not really that useful or critical after all. If I know I'm working with development version of a plugin I can always make sure I don't have any released version of the same plugin installed. And if I'm in a setting where I can't uninstall existing plugins I can always run with PACKER_PLUGIN_PATH to create an isolated environment.

The blog post was very informative, thank you for that! I think the choice of dev as the only pre-release version that Packer accepts can benefit from being highlighted in documentation and elsewhere, as it might help others who, like me, assumed Packer's version parsing/support was generic semver with x.y.z-foo+baz.

Other than that, ship it! 🎉😅