Builds using spot pricing fail when used with a Shared Image Gallery base

ianpittwood commented 7 months ago

Overview of the Issue

Recently, I started attempting to upgrade from VHD image artifacts to using Shared Image Galleries to store our images. We do our image builds in a sequence of Base -> Product Base -> Product where Base uses the latest LTS for Ubuntu from the Marketplace. Building the Base image works fine after switching out the VHD-related variables for Shared Image Gallery blocks, but past that we're experiencing failures that seem related to spot pricing. When building off the Base image we first create for Product Base from a Shared Gallery Image with spot pricing configured, we get the following error during Packer's deployment of temporary build resources:

==> byol.azure-arm.azure-byol: ERROR: -> DeploymentFailed : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
==> byol.azure-arm.azure-byol: ERROR:   -> BadRequest
==> byol.azure-arm.azure-byol: ERROR:   -> BadRequest : The request message is invalid.
==> byol.azure-arm.azure-byol: ERROR:     ->  : Could not find member 'billingProfile' on object of type 'Properties'. Path 'properties.billingProfile', line 1, position 135.
==> byol.azure-arm.azure-byol: ERROR:     ->  : Could not find member 'evictionPolicy' on object of type 'Properties'. Path 'properties.evictionPolicy', line 1, position 228.
==> byol.azure-arm.azure-byol: ERROR:     ->  : Could not find member 'priority' on object of type 'Properties'. Path 'properties.priority', line 1, position 1142.
==> byol.azure-arm.azure-byol:
==> byol.azure-arm.azure-byol: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details." Details=[{"code":"BadRequest","message":"{\r\n  \"error\": {\r\n    \"details\": [\r\n      {\r\n        \"message\": \"Could not find member 'billingProfile' on object of type 'Properties'. Path 'properties.billingProfile', line 1, position 135.\",\r\n        \"target\": \"vm.properties.billingProfile\"\r\n      },\r\n      {\r\n        \"message\": \"Could not find member 'evictionPolicy' on object of type 'Properties'. Path 'properties.evictionPolicy', line 1, position 228.\",\r\n        \"target\": \"vm.properties.evictionPolicy\"\r\n      },\r\n      {\r\n        \"message\": \"Could not find member 'priority' on object of type 'Properties'. Path 'properties.priority', line 1, position 1142.\",\r\n        \"target\": \"vm.properties.priority\"\r\n      }\r\n    ],\r\n    \"code\": \"BadRequest\",\r\n    \"message\": \"The request message is invalid.\"\r\n  }\r\n}"}]

If spot pricing configurations are removed, everything works fine. This seems to be specific to when you combine a Shared Gallery Image source base image with a spot pricing configuration.

Reproduction Steps

Create a base image stored in a Shared Image Gallery.
Configure a source that uses the Shared Image Gallery base image as the source image to build off of.
Add a spot pricing configuration block to the source.
Try to initiate a new build using that source. Deploying the build resources will fail.

Plugin and Packer version

Packer v1.10.0 Azure Plugin v1.4.5

Simplified Packer Buildfile

source "azure-arm" "azure-byol" {
  azure_tags                = local.tags
  build_resource_group_name = var.azure_build_resource_group
  client_id                 = var.azure_client_id
  client_secret             = var.azure_client_secret
  os_disk_size_gb           = var.os_disk_size
  os_type                   = "Linux"
  ssh_pty                   = "true"
  subscription_id           = var.azure_subscription_id
  tenant_id                 = var.azure_tenant_id
  use_azure_cli_auth        = var.build_context == "local" ? true : false
  vm_size                   = var.azure_vm_sku
  spot {
    eviction_policy = "Delete"
    max_price       = var.azure_spot_max_price
  }
  shared_image_gallery {
    subscription = var.azure_subscription_id
    resource_group = var.azure_resource_group
    gallery_name = local.azure_base_image_gallery
    image_name   = local.azure_shared_image_definition_name
    image_version = "0.0.${var.build_number}"
  }
  shared_image_gallery_destination {
    subscription = var.azure_subscription_id
    resource_group = var.azure_resource_group
    gallery_name = local.azure_shared_image_gallery_name
    image_name   = local.azure_shared_image_definition_name_byol
    image_version = "0.0.${var.build_number}"
  }
}

Operating system and Environment details

OS: Ubuntu 22.04 Architecture: x86_64

Log Fragments and crash.log files

Azure Build Log

``` byol.azure-arm.azure-byol: output will be in this color. ==> byol.azure-arm.azure-byol: Running builder ... ==> byol.azure-arm.azure-byol: Getting tokens using Azure CLI ==> byol.azure-arm.azure-byol: Getting tokens using Azure CLI byol.azure-arm.azure-byol: Creating Azure Resource Manager (ARM) client ... ==> byol.azure-arm.azure-byol: Getting source image id for the deployment ... ==> byol.azure-arm.azure-byol: -> SourceImageName: '/subscriptions//resourceGroups/packer-marketplaces-images/providers/Microsoft.Compute/galleries/posit_marketplace_base/images/azure-image-gallery/versions/0.0.2' ==> byol.azure-arm.azure-byol: Using existing resource group ... ==> byol.azure-arm.azure-byol: -> ResourceGroupName : 'packer-marketplaces-build' ==> byol.azure-arm.azure-byol: -> Location : 'eastus' ==> byol.azure-arm.azure-byol: Validating deployment template ... ==> byol.azure-arm.azure-byol: -> ResourceGroupName : 'packer-marketplaces-build' ==> byol.azure-arm.azure-byol: -> DeploymentName : 'pkrdpet4wmytqvq' ==> byol.azure-arm.azure-byol: Deploying deployment template ... ==> byol.azure-arm.azure-byol: -> ResourceGroupName : 'packer-marketplaces-build' ==> byol.azure-arm.azure-byol: -> DeploymentName : 'pkrdpet4wmytqvq' ==> byol.azure-arm.azure-byol: ERROR: -> DeploymentFailed : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. ==> byol.azure-arm.azure-byol: ERROR: -> BadRequest ==> byol.azure-arm.azure-byol: ERROR: -> BadRequest : The request message is invalid. ==> byol.azure-arm.azure-byol: ERROR: -> : Could not find member 'billingProfile' on object of type 'Properties'. Path 'properties.billingProfile', line 1, position 135. ==> byol.azure-arm.azure-byol: ERROR: -> : Could not find member 'evictionPolicy' on object of type 'Properties'. Path 'properties.evictionPolicy', line 1, position 228. ==> byol.azure-arm.azure-byol: ERROR: -> : Could not find member 'priority' on object of type 'Properties'. Path 'properties.priority', line 1, position 1142. ==> byol.azure-arm.azure-byol: ==> byol.azure-arm.azure-byol: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details." Details=[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"details\": [\r\n {\r\n \"message\": \"Could not find member 'billingProfile' on object of type 'Properties'. Path 'properties.billingProfile', line 1, position 135.\",\r\n \"target\": \"vm.properties.billingProfile\"\r\n },\r\n {\r\n \"message\": \"Could not find member 'evictionPolicy' on object of type 'Properties'. Path 'properties.evictionPolicy', line 1, position 228.\",\r\n \"target\": \"vm.properties.evictionPolicy\"\r\n },\r\n {\r\n \"message\": \"Could not find member 'priority' on object of type 'Properties'. Path 'properties.priority', line 1, position 1142.\",\r\n \"target\": \"vm.properties.priority\"\r\n }\r\n ],\r\n \"code\": \"BadRequest\",\r\n \"message\": \"The request message is invalid.\"\r\n }\r\n}"}] ==> byol.azure-arm.azure-byol: ==> byol.azure-arm.azure-byol: Deleting Virtual Machine deployment and its attatched resources... ==> byol.azure-arm.azure-byol: Could not retrieve OS Image details: compute.VirtualMachinesClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource 'Microsoft.Compute/virtualMachines/pkrvmet4wmytqvq' under resource group 'packer-marketplaces-build' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix" ==> byol.azure-arm.azure-byol: Adding to deletion queue -> Microsoft.Compute/virtualMachines : 'pkrvmet4wmytqvq' ==> byol.azure-arm.azure-byol: Adding to deletion queue -> Microsoft.Network/networkInterfaces : 'pkrniet4wmytqvq' ==> byol.azure-arm.azure-byol: Adding to deletion queue -> Microsoft.Network/publicIPAddresses : 'pkripet4wmytqvq' ==> byol.azure-arm.azure-byol: Adding to deletion queue -> Microsoft.Network/virtualNetworks : 'pkrvnet4wmytqvq' ==> byol.azure-arm.azure-byol: Waiting for deletion of all resources... ==> byol.azure-arm.azure-byol: Attempting deletion -> Microsoft.Network/virtualNetworks : 'pkrvnet4wmytqvq' ==> byol.azure-arm.azure-byol: Attempting deletion -> Microsoft.Network/networkInterfaces : 'pkrniet4wmytqvq' ==> byol.azure-arm.azure-byol: Attempting deletion -> Microsoft.Compute/virtualMachines : 'pkrvmet4wmytqvq' ==> byol.azure-arm.azure-byol: Attempting deletion -> Microsoft.Network/publicIPAddresses : 'pkripet4wmytqvq' ==> byol.azure-arm.azure-byol: Couldn't delete Microsoft.Network/virtualNetworks resource. Will retry. ==> byol.azure-arm.azure-byol: Name: pkrvnet4wmytqvq ==> byol.azure-arm.azure-byol: Couldn't delete Microsoft.Network/publicIPAddresses resource. Will retry. ==> byol.azure-arm.azure-byol: Name: pkripet4wmytqvq ==> byol.azure-arm.azure-byol: Attempting deletion -> Microsoft.Network/virtualNetworks : 'pkrvnet4wmytqvq' ==> byol.azure-arm.azure-byol: Attempting deletion -> Microsoft.Network/publicIPAddresses : 'pkripet4wmytqvq' ==> byol.azure-arm.azure-byol: Failed to find temporary OS disk on VM. Please delete manually. ==> byol.azure-arm.azure-byol: ==> byol.azure-arm.azure-byol: VM Name: pkrvmet4wmytqvq ==> byol.azure-arm.azure-byol: Error: %!s() ==> byol.azure-arm.azure-byol: Removing the created Deployment object: 'pkrdpet4wmytqvq' ==> byol.azure-arm.azure-byol: ==> byol.azure-arm.azure-byol: The resource group was not created by Packer, not deleting ... Build 'byol.azure-arm.azure-byol' errored after 1 minute 15 seconds: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details." Details=[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"details\": [\r\n {\r\n \"message\": \"Could not find member 'billingProfile' on object of type 'Properties'. Path 'properties.billingProfile', line 1, position 135.\",\r\n \"target\": \"vm.properties.billingProfile\"\r\n },\r\n {\r\n \"message\": \"Could not find member 'evictionPolicy' on object of type 'Properties'. Path 'properties.evictionPolicy', line 1, position 228.\",\r\n \"target\": \"vm.properties.evictionPolicy\"\r\n },\r\n {\r\n \"message\": \"Could not find member 'priority' on object of type 'Properties'. Path 'properties.priority', line 1, position 1142.\",\r\n \"target\": \"vm.properties.priority\"\r\n }\r\n ],\r\n \"code\": \"BadRequest\",\r\n \"message\": \"The request message is invalid.\"\r\n }\r\n}"}] ==> Wait completed after 1 minute 15 seconds ==> Some builds didn't complete successfully and had errors: --> byol.azure-arm.azure-byol: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details." Details=[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"details\": [\r\n {\r\n \"message\": \"Could not find member 'billingProfile' on object of type 'Properties'. Path 'properties.billingProfile', line 1, position 135.\",\r\n \"target\": \"vm.properties.billingProfile\"\r\n },\r\n {\r\n \"message\": \"Could not find member 'evictionPolicy' on object of type 'Properties'. Path 'properties.evictionPolicy', line 1, position 228.\",\r\n \"target\": \"vm.properties.evictionPolicy\"\r\n },\r\n {\r\n \"message\": \"Could not find member 'priority' on object of type 'Properties'. Path 'properties.priority', line 1, position 1142.\",\r\n \"target\": \"vm.properties.priority\"\r\n }\r\n ],\r\n \"code\": \"BadRequest\",\r\n \"message\": \"The request message is invalid.\"\r\n }\r\n}"}] ==> Builds finished but no artifacts were created. error: Recipe `build` failed with exit code 1 ```

JenGoldstrich commented 2 months ago

Hey @ianpittwood can you verify if this issue is still occurring for you, I believe something in the upgrade to v2 fixed this improper template generation, maybe just a deployment API version bump fixed this.

When I try using v1.4.5 of the plugin I am able to reproduce this error, but using the newest version of the plugin (v2.1.6 as of writing) I am able to successfully build a SIG based VM with spot pricing, and capture it in an image version with a template similar to your provided one.

There are a few breaking changes in v2 https://github.com/hashicorp/packer-plugin-azure/releases/tag/v2.0.0 called out in the release notes but I don't think any of those apply to your template, as you aren't using VHD builds, or interactive authentication

JenGoldstrich commented 1 week ago

I am closing this issue as we haven't heard back in a few months and it doesn't appear to happen on newer versions, please re-open it if you are experiencing it still

hashicorp / packer-plugin-azure