search

hashicorp / packer-plugin-amazon

Packer plugin for Amazon AMI Builder
https://www.packer.io/docs/builders/amazon
Mozilla Public License 2.0
72 stars 110 forks source link

ebs: add fast-launch support with AMI copies #451

Closed lbajolet-hashicorp closed 8 months ago

lbajolet-hashicorp commented 8 months ago

Prior to this commit, enabling fast-launch was done before copying the AMI to other regions, if the option was selected.

This was problematic, as the pre-provisioned snapshots produced by enabling fast-launch weren't copied over to other regions, so only the source image was supporting fast-launch.

To remedy this problem, we now enable fast-launch after the image is copies to other regions. This fixes the problem, but takes more time to complete the build, since each enabling takes time.

Also, this requirement means that each region may have its own launch template to use for enabling fast-launch, so we introduce a new block in the configuration to specify which region should use which template. This option does not directly supersede the original fast_launch_template* options, but instead assumes that if specified, will only apply to the region the AMI is built in, and other regions will default if unspecified.

Closes: #445

lbajolet-hashicorp commented 8 months ago

~Note: I'll try to add some unit-tests for the configurations before we can merge.~ Done!

@tivanov-qb: if possible, could I ask you to test this fix on your configuration? I'd like to have a confirmation that this does fix your problem, even if some acceptance tests point in the right direction, we can't be too careful :)

tivanov-qb commented 8 months ago

@lbajolet-hashicorp looking forward for this to be merged, so I can kick off a test. Thank you!

lbajolet-hashicorp commented 8 months ago

@tivanov-qb any chance you can build the plugin and test it locally on one of your configurations? I'd prefer if you could test it before we release it since we'd have to crank out another release quickly after if that change doesn't fix your issue

tivanov-qb commented 8 months ago

@tivanov-qb any chance you can build the plugin and test it locally on one of your configurations? I'd prefer if you could test it before we release it since we'd have to crank out another release quickly after if that change doesn't fix your issue

@lbajolet-hashicorp absolutely, could you please share how to point to branch/tag ? this is how i'm using the plugin (hcl format)

packer { required_plugins { amazon = { source = "github.com/hashicorp/amazon" version = "~> 1" } } }

lbajolet-hashicorp commented 8 months ago

At the moment unfortunately you won't be able to use a required_plugins with a manually installed plugin, we're working on supporting this workflow right now, but it's not been released yet.

I would advise you to remove the required_plugins block from your template, and store the built version of the plugin for your OS/arch alongside the working directory you invoke Packer from. Also I would suggest invoking packer with PACKER_LOG=1 to make sure the right plugin binary is invoked for the test.

If you need more guidance I suggest referring to the docs on installing plugins, especially under the Manual installation tab. Let me know if you need some help with that.

tivanov-qb commented 8 months ago

At the moment unfortunately you won't be able to use a required_plugins with a manually installed plugin, we're working on supporting this workflow right now, but it's not been released yet.

I would advise you to remove the required_plugins block from your template, and store the built version of the plugin for your OS/arch alongside the working directory you invoke Packer from. Also I would suggest invoking packer with PACKER_LOG=1 to make sure the right plugin binary is invoked for the test.

If you need more guidance I suggest referring to the docs on installing plugins, especially under the Manual installation tab. Let me know if you need some help with that.

sorry.. but I think I will need help with that.. the document says replace the binary, but how do I get the testing binary to replace it with the actual one ?

lbajolet-hashicorp commented 8 months ago

In this case you'll have to build it; if you clone the repository and checkout the branch in which I made those changes, you'll be able to build the plugin and move it where needed.

You'll need the Go toolchain for that, and possibly make if you want to run the commands through the provided Makefile, though for something as simple as this build, you can directly invoke go build -o packer-plugin-amazon, which will build the plugin for your environment. At that point, you should be able to use it with Packer

tivanov-qb commented 8 months ago

@lbajolet-hashicorp i already wasted couple of hours and this just does not work. if the plugin is called packer-plugin-amazon is not being picked by packer.. if the name is packer-plugin-amazon_v1_linux_amd64 it fails because it cannot find a checksum file..

tivanov-qb commented 8 months ago 
> `09:59:43  2024/01/24 07:59:43 [INFO] Packer version: 1.10.0 [go1.20.11 linux amd64]
> 09:59:43  2024/01/24 07:59:43 Detected config directory from env var: /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config
> 09:59:43  2024/01/24 07:59:43 [TRACE] discovering plugins in /bin
> 09:59:43  2024/01/24 07:59:43 [TRACE] discovering plugins in .
> 09:59:43  2024/01/24 07:59:43 [TRACE] discovering plugins in /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins
> 09:59:43  2024/01/24 07:59:43 [INFO] Discovered potential plugin: amazon = /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1_linux_amd64
> 09:59:43  2024/01/24 07:59:43 [TRACE] GetChecksumOfFile("/mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1_linux_amd64") failed: open /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1_linux_amd64_SHA256SUM: no such file or directory
> 09:59:43  2024/01/24 07:59:43 [WARN] No checksum found for "/mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins" ignoring possibly unsafe binary`
tivanov-qb commented 8 months ago 
[10:23:04  2024/01/24 08:23:04 Detected config directory from env var: /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config
10:23:04  2024/01/24 08:23:04 [TRACE] discovering plugins in /bin
10:23:04  2024/01/24 08:23:04 [TRACE] discovering plugins in .
10:23:04  2024/01/24 08:23:04 [TRACE] discovering plugins in /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins
10:23:04  2024/01/24 08:23:04 [INFO] Discovered potential plugin: amazon = /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon_linux_amd64
10:23:04   Error loading configuration: 
10:23:04  
10:23:04  fork/exec /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon_linux_amd64: no such file or directory](url)
tivanov-qb commented 8 months ago

I give up, i don't know how to workaround/fix this error 

`11:11:13  2024/01/24 09:11:13 [INFO] Packer version: 1.10.0 [go1.20.11 linux amd64]
11:11:13  2024/01/24 09:11:13 [TRACE] discovering plugins in /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins
11:11:13  2024/01/24 09:11:13 [INFO] Discovered potential plugin: amazon = /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon
11:11:13   Error loading configuration: 
11:11:13  
11:11:13  fork/exec /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon: no such file or directory`
tivanov-qb commented 8 months ago

possibly related to https://github.com/hashicorp/packer/pull/12414 ?

lbajolet-hashicorp commented 8 months ago

I give up, i don't know how to workaround/fix this error

`11:11:13  2024/01/24 09:11:13 [INFO] Packer version: 1.10.0 [go1.20.11 linux amd64]
11:11:13  2024/01/24 09:11:13 [TRACE] discovering plugins in /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins
11:11:13  2024/01/24 09:11:13 [INFO] Discovered potential plugin: amazon = /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon
11:11:13   Error loading configuration: 
11:11:13  
11:11:13  fork/exec /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon: no such file or directory`

This does look weird. Is this the binary you built? I'm surprised it cannot start it at all, can you run it from your shell at least? If you try to run /mnt/ssd/workspace/workspace/1_EnableFastLaunchForWindowsAMIs/packer_config/.packer.d/plugins/packer-plugin-amazon describe, does it return the description of the plugin?

Let's try to figure out the source of the problem if possible, there could be some other bug lurking around that we can maybe fix by the next release.

In the meantime, I'll address @JenGoldstrich's comment and merge this PR so we can release the plugin, hopefully the problem's fixed and you can roll with it then.

tivanov-qb commented 8 months ago

@lbajolet-hashicorp seems you fixed it! AMIs are now created with fast launch enabled and same goes for the copied AMIs. Great work! Thank you!

tivanov-qb commented 8 months ago

@lbajolet-hashicorp i just noticed that AMIs shared via org arn are not actually having the fast launch enabled. so in my source ci account all looks good, but fast launch is not enabled in the destination accounts :( perhaps aws issue and not packer ? i will open a ticket and check with them

tivanov-qb commented 8 months ago

might not be a surprise for you, but aws confirmed fast launch setting is not automatically carried over to the destination accounts

lbajolet-hashicorp commented 8 months ago

Hi @tivanov-qb,

This doesn't surprise me that much since fast-launch basically pre-provisions snapshots for deploying the AMI with, which AFAIK are account-bound, so that checks out that for other accounts, fast-launch won't be exported.

I don't think this is something we can support in Packer tbh.