[Azure]: save OS disk during image creation

[yaronkaikov]

Due to a lack of resources in our main region EASTUS, we need to copy our images to different regions. In order to do it we need to keep the OS_disk during our main azure image creation.

So instead of creating a temp resource group for every build, I have created one for building purposes only.

Also keeping the original OS disk so we can later copy those images to different regions.

While doing it, I saw it also solves the retry for deleting resources we saw during Azure builds

Refs: https://github.com/scylladb/scylla-pkg/issues/3165

[benipeled]

We have

      "managed_image_resource_group_name": "scylla-images",
      "managed_image_name": "{{user `image_name`| clean_resource_name}}",

which saves the final image on a different resource group - why can't we use it later for the copy to other regions?

What's the next step from pkg side? how we're going to make this copy?

[yaronkaikov]

Verification: https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/azure-image/40/ Then manually copied the image to verify it's working:

[yaronkaikov@london]~/git/scylla-machine-image/packer (azure-save-os-disk)$ az image copy --source-resource-group scylla-images --source-object-name debug-scylla-5.3.0-dev-x86_64-2023-02-13T07-01-18 --target-location eastus2 --target-resource-group "scylla-images-eastus2" --target-name debug-scylla-5.3.0-dev-x86_64-2023-02-13T07-01-18
Getting OS disk ID of the source VM/image
Creating source snapshot
Getting sas url for the source snapshot with timeout: 3600 seconds
Target location count: 1
eastus2 - Creating target storage account (can be slow sometimes)
eastus2 - Creating container in the target storage account
eastus2 - Copying blob to target storage account
eastus2 - Copy progress: 0%
eastus2 - Copy progress: 1%
eastus2 - Copy progress: 2%
eastus2 - Copy progress: 3%
eastus2 - Copy progress: 4%
eastus2 - Copy progress: 5%
eastus2 - Copy progress: 6%
eastus2 - Copy progress: 7%
eastus2 - Copy progress: 8%
eastus2 - Copy progress: 9%
eastus2 - Copy progress: 10%
eastus2 - Copy progress: 11%
eastus2 - Copy progress: 12%
eastus2 - Copy progress: 13%
eastus2 - Copy progress: 14%
eastus2 - Copy progress: 15%
eastus2 - Copy progress: 16%
eastus2 - Copy progress: 17%
eastus2 - Copy progress: 18%
eastus2 - Copy progress: 19%
eastus2 - Copy progress: 31%
eastus2 - Copy progress: 40%
eastus2 - Copy progress: 59%
eastus2 - Copy progress: 72%
eastus2 - Copy progress: 86%
eastus2 - Copy progress: 99%
eastus2 - Copy progress: 100%
eastus2 - Copy time: 0:06:01.627430
eastus2 - Creating snapshot in target region from the copied blob
eastus2 - Creating final image
Image copy finished

[yaronkaikov]

We have

      "managed_image_resource_group_name": "scylla-images",
      "managed_image_name": "{{user `image_name`| clean_resource_name}}",

which saves the final image on a different resource group - why can't we use it later for the copy to other regions?

What's the next step from pkg side? how we're going to make this copy?

There is no built-in option to copy the Azure images (like we have in AWS) using packer. At least I couldn't find one The next step will be in Scylla-pkg to add to the Azure job another stage that actually does the copy. I guess it makes sense to do it only after artifacts passed, so we won't copy a nonworking image

[fruch]

We have

      "managed_image_resource_group_name": "scylla-images",
      "managed_image_name": "{{user `image_name`| clean_resource_name}}",

which saves the final image on a different resource group - why can't we use it later for the copy to other regions?

What's the next step from pkg side? how we're going to make this copy?

There is no built-in option to copy the Azure images (like we have in AWS) using packer. At least I couldn't find one The next step will be in Scylla-pkg to add to the Azure job another stage that actually does the copy. I guess it makes sense to do it only after artifacts passed, so we won't copy a nonworking image

For reference here is the issue:

https://github.com/hashicorp/packer-plugin-azure/issues/20

[yaronkaikov]

We have

      "managed_image_resource_group_name": "scylla-images",
      "managed_image_name": "{{user `image_name`| clean_resource_name}}",

which saves the final image on a different resource group - why can't we use it later for the copy to other regions? What's the next step from pkg side? how we're going to make this copy?

There is no built-in option to copy the Azure images (like we have in AWS) using packer. At least I couldn't find one The next step will be in Scylla-pkg to add to the Azure job another stage that actually does the copy. I guess it makes sense to do it only after artifacts passed, so we won't copy a nonworking image

For reference here is the issue:

hashicorp/packer-plugin-azure#20

Thanks, @fruch I will follow up on this, but it seems to be open for a while. so not sure if and when they will implement it