amazon-ami data source executed unexpectedly

matt-richardson commented 3 years ago

Overview of the Issue

I have a packer build that contains two sources amazon-ebs and azure-arm. The amazon-ebs source depends on an amazon-ami data source.

When I run with -only=azure-arm.azure-arm, the amazon-ami data source gets executed as well.

Normally, not much of an issue, but all my amazon builds just failed because amazon took down the base ami. This unfortunately stopped all the azure ones building to.

I tried adding an only block to the data source, but it appears not to be supported.

Reproduction Steps

create a packer file as described above

run packer build "-only=azure-arm.azure-arm" . -> see error:

Error: Datasource.Execute failed: No AMI was found matching filters: {
Filters: [
{
  Name: "platform",
  Values: ["windows"]
},
{
  Name: "root-device-type",
  Values: ["ebs"]
},
{
  Name: "virtualization-type",
  Values: ["hvm"]
},
{
  Name: "architecture",
  Values: ["x86_64"]
},
{
  Name: "name",
  Values: ["Windows_Server-1909-English-Core-Containers*"]
}
],
Owners: ["801119661308"]
}

on myfile.pkr.hcl line 72:
(source code not available)

Packer version

> packer version
Packer v1.7.2

Simplified Packer Buildfile

myfile.pkr.hcl

variable "aws_access_key_id" { type = string default = "${env("AWS_ACCESS_KEY_ID")}" } variable "aws_secret_access_key" { type = string default = "${env("AWS_SECRET_ACCESS_KEY")}" } data "amazon-ami" "autogenerated_1" { access_key = "${var.aws_access_key_id}" filters = { architecture = "x86_64" name = "Windows_Server-1909-English-Core-Containers*" # note, this does not exist platform = "windows" root-device-type = "ebs" virtualization-type = "hvm" } most_recent = true owners = ["801119661308"] # amazon region = "ap-southeast-2" secret_key = "${var.aws_secret_access_key}" } source "amazon-ebs" "amazon-ebs" { access_key = "${var.aws_access_key_id}" ami_name = "my-ami" associate_public_ip_address = true communicator = "winrm" instance_type = "c5.large" region = "ap-southeast-2" secret_key = "${var.aws_secret_access_key}" security_group_id = "sg-XXXXXXXX" source_ami = "${data.amazon-ami.autogenerated_1.id}" subnet_id = "subnet-XXXXXXXX" windows_password_timeout = "20m" winrm_insecure = true winrm_use_ssl = true winrm_username = "Administrator" } source "azure-arm" "azure-arm" { async_resourcegroup_delete = false client_id = "XXXXXXX" client_secret = "XXXXXXX" communicator = "winrm" image_offer = "WindowsServer" image_publisher = "MicrosoftWindowsServer" image_sku = "datacenter-core-1909-with-containers-smalldisk" location = "Australia East" managed_image_name = "my-ami" managed_image_resource_group_name = "XXXXXXX" os_type = "Windows" subscription_id = "XXXXXXXX" tenant_id = "XXXXXXXX" vm_size = "Standard_D2_V4" winrm_insecure = "true" winrm_timeout = "3m" winrm_use_ssl = "true" winrm_username = "packer" } build { sources = [ "source.amazon-ebs.amazon-ebs", "source.azure-arm.azure-arm" ] provisioner "file" { destination = "UserdataExecution.log" direction = "download" only = ["amazon-ebs.amazon-ebs"] source = "C:/ProgramData/Amazon/EC2-Windows/Launch/Log/UserdataExecution.log" } }

Operating system and Environment details

Windows, 21H1 (OS Build 19043.985)

Log Fragments and crash.log files

n/a - nothing of interesting in the log file

nywilken commented 3 years ago

Thanks for reaching out. This is a good catch. The only filter is only supported by the build block so the datasource will still execute. We need to think about this a little more internally.

Looking at the provided build script if there is not too much shared resources between the two builders another workaround until we can get this squared away would be to split your build template into two build templates one for azure and one for amazon.

You can then just run the azure build like packer build azure.pkr.hcl. When you want to run both builds in parallel you packer build . would work as expected and if amazon fails again the azure build will continue to work.

github-actions[bot] commented 2 years ago

This issue has been synced to JIRA for planning.

JIRA ID: HPR-751

hashicorp / packer