Add -test Flag to Packer CLI for testing that a Build Works

[naftulikay]

I have currently written a Packer template which uses Ansible and Goss to provision and test an AMI. I am running this in Travis CI against pull request branches so as to wrap CI/CD around AMIs. Workflow looks like this:

• Pull requests run packer build and upon failure report the failed Goss test case. Upon success, I use the AWS CLI to deregister and remove the AMI and its snapshots.
• Upon merge to develop, I run packer build -var release_type=unstable which tags my build accordingly.
• Upon merge to master, I run packer build -var release_type=stable, which tags my build accordingly.

What would be really great would be if packer build supported a -test flag which would tear everything down no matter what, and return zero if no commands failed and the error code if a command did fail during the build. This would make things much easier, as I now have wrapped my build in a complicated Bash script that uses tee to pipe logs to a temporary file, finds the AMI after build, and manually tears it down using the AWS CLI.

[rickard-von-essen]

You can use the manifest post-processor to get a clean output of the built AMI's and a less complicated bash script to clean those up.

[naftulikay]

@rickard-von-essen This is certainly helpful thank you!

My exact use case though is that I'd like Packer to simply return the return code after shutting down and terminating the EC2 instance. I basically want to test that my automation works and passes tests. Generating the AMI is often the longest part of the build, so if we could add a flag that causes it to terminate everything before trying to build the AMI, that would be most ideal. I think that this would probably be an enhancement to all builders, as the -test flag would ideally prevent artifacts from being made, Packer would just run through all of the provisioning actions and then tear everything down.

[mwhooker]

duplicates #4543

[levpaul]

also duplicates #1739 too, right? I also want this feature. Given that the actual code changes needed for this were provided long ago, it's kind of frustrating that people are still asking for it and it's not available :(

[rickard-von-essen]

@levilovelock just because people want a feature doesn't mean that it fits the tools purpose.

[levpaul]

I understand. FWIW @naftulikay a tool which can handle this use-case fairly well is Vagrant paired with the vagrant-aws plugin.

[naftulikay]

That will probably be what I use: vagrant-aws for tests and Packer during deploy for final build and shipment of AMI; codebases for provisioning shared between vagrant-aws and packer.

[eredi93]

did anyone found a solution to this? @naftulikay how are you currently de-registering the AMI?

[naftulikay]

Use aws ec2 describe-images to describe the underlying snapshots. Then aws ec2 deregister-image. Then for each snapshot, aws ec2 delete-snapshot. It has to proceed in this order.

[levpaul]

So the way I handle this is as follows.

1. Use a custom variable, i.e. dry_run
2. Create a small provisioner after your main stuff is done to early exit with a known code if flag is set: i.e.

   {
     "type": "shell-local",
     "command": "bash -c 'if [[ {{user `dry_run`}} == true ]]; then touch ../packer/.packer_success && exit 235; else exit 0; fi'"
   },

Then I just wrap my call to packer in another bash script which takes args, passes them to packer etc and of course, checks for exit 235 and/or the .packer_success file if the dry run flag was set. This gives you an early exit and still lets you know if things

[naftulikay]

that's an awesome hack

[eredi93]

@levilovelock i tried that hack and it doesn't seem to work, packer swallows the status code and always return 1 not my special exit code

$ packer build image.json
amazon-ebs output will be in this color.

==> amazon-ebs: Prevalidating AMI Name: test-ami-2018-03-16T16-56-41Z
    amazon-ebs: Found Image ID: ami-3c51ae41
==> amazon-ebs: Creating temporary keypair: packer_5aabf749-ecae-903b-6c2c-9824d2a4c522
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Instance ID: i-0837abd25b4fbb730
==> amazon-ebs: Waiting for instance (i-0837abd25b4fbb730) to become ready...
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Executing local command: exit 254
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Erroneous exit code 254 while executing command: exit 254

Please see output above for more information.

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Erroneous exit code 254 while executing command: exit 254

Please see output above for more information.

==> Builds finished but no artifacts were created.
$  echo $?
1

[levpaul]

@eredi93 - Sorry I simplified it too much and wrote it wrong. The exit 235 is actually to make the provisioner fail, causing packer to stop before burning the AMI. The && touch ../packer/.packer_success is the part that my wrapper script checks for to verify early-exit.

[SwampDragons]

I don't think this is something we're interested in supporting, so I'm going to close this.

[tomaszov]

"I maintain Packer for HashiCorp. Let's work together to build awesome things and make the world a little better." vs. "I don't think this is something we're interested in supporting, so I'm going to close this."

As for me this is a feature that is badly needed, taken how much time necessary for building a VM image, and how much time wasted in case it doesn't work properly, so very surprising answer. But maybe it's just me.

[rickard-von-essen]

@tomaszov There are solutions to do this already without implementing a new feature, there is always a cost for maintaining code. IMHO (but yes I'm biased) Packer wouldn't be any better with this flag, just more complex. And since Packer something of a industry standard tool for building images I don't agree that it's useless, rather it's really useful and used by a lot of people.

If you have a detailed suggestion of how such a feature would work and a good motivation on why it would be better than how you could do this currently I suggest that you open a new issue detailing that or send a email to the mailing list to open up a discussion.

[tomaszov]

You are most probably right. Useless is too harsh, and definitely not true. I was just disappointed as I felt, here is this cool tool that could help a great deal, yet I loose on the other side the advantages with the lack of testing possibility. But of course nothing is black and white, and already found some solutions for this as well. THX for your time for answering and keep up the good work.

[SwampDragons]

@tomaszov I totally understand your frustration here, but this issue had been open for a year and the conversation about it had ended long before I closed it.

Saying "no" is absolutely the worst part of my job and I don't enjoy it, but I think it's more respectful to the community to be honest about what we can and will support than to leave issues open forever because I don't want to be perceived as mean.

I was probably too brusque closing the issue, but my thought process was essentially what Rickard said -- You can replicate the desired functionality by using the manifest and shell-local post-processors to delete images after you run, so the value added by adding a whole new flag is pretty slim, and it adds a nontrivial maintenance burden. I would much rather have a well-supported, streamlined product that requires a little extra legwork from users than a buggy and sprawling mess.

If you need a hand figuring out how to get the test behavior you need to see, I'm happy to help, or you can reach out to our mailing list and get advice there.

I do want to point you to conversation about an issue that happened before I took charge of the project, and that's the PR at #4681 -- I discussed a bit about why that one stalled out and eventually got closed in the comment here https://github.com/hashicorp/packer/issues/7209#issuecomment-463764455, but if someone wants to take it on I'd merge that -- it's a smaller maintenance burden than is described here and, I think, a much bigger win because of the time saved when you don't actually have to create the AMI.

[tomaszov]

@SwampDragons Really sorry for my impolite reaction on this topic, and thank you for your answer and your willingness to help. You are totally right, for not messing up an anyway greatly working product. Thank you again, and keep up the good work.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.