VHD file was not deleted when Azcopy ongoing process was terminated, however the VHD file in Azure Blob show the same size as AWS S3

[momodedev]

10.16.2 to 10.25.1?

Note: The version is visible when running AzCopy without any argument

Which platform are you using? Azure Windows VM

What command did you run? azcopy.exe copy "https://big-vhd.s3.us-west-2.amazonaws.com" https://pageblob.blob.core.windows.net/bigvhdfile?SAS --recursive=true --list-of-files C:/Users/XXX/t/c/listoffiles-20240717040743912-ae5233e92fa342f49c8fdab763ffaf51

What problem was encountered?

The incomplete VHD file was not deleted and lost in Azure Blob(Destination), ideally and normally the file should be deleted because the migration of this file is not completed. The scenario is: Azure Batch call azcopy.exe command as above, when utilizing Azure Batch “Terminate” to stop the job of azcopy.exe copy ongoing process.

How can we reproduce the problem in the simplest way?

reproduce it as below steps 1) run azcopy.exe copy "local vhd file" "azure blob url+SAS" 2) end the ongoing azcopy.exe in Windows Task Manager 3) check azure blob container, you can see the incomplete vhd file was lost there.

Have you found a mitigation/solution?

In fact, when sending "Ctrl+C signal" to azcopy.exe, the incomplete vhd file should be delete normally.

[ashruti-msft]

In Azure Storage, when you initiate an azcopy copy operation to copy a VHD file (or any other blob), the copy operation is managed by Azure Storage itself, not by azcopy directly. azcopy is simply a tool that facilitates this process. If the azcopy tool is terminated or closed during a copy operation, the operation initiated in Azure Storage will continue running independently. Azure Storage handles the copy operation asynchronously on its servers once it receives the request from azcopy. Therefore, terminating azcopy will not delete the VHD file or cancel the copy operation in Azure Storage.

[momodedev]

Thanks for your quick response Ashruti, One more question, other blob such as migration a 64GB binary file from AWS S3 to Azure Storage Blob Container, it is different, at the begin of azcopy.exe process, it does not create any file in destination-Azure Storage Blob Container, it is OK. Otherwise, once the azcopy.exe process was terminated forcibly, the incomplete VHD file was lost in Azure Storage Blob Container, it is not the expectation as the normal data migration. it will cause data integrity problem: when copying the file again, this VHD file will be skipped because it is the existing file with the same meta data as the original file in AWS S3 (in fact, it is incomplete).

[ashruti-msft]

To make sure that copy doesn't skip if there is already a file existing on destination you can use the--overwriteflag.

[momodedev]

Thanks for your help Ashruti. "To use the --overwrite=true" causes time cost, it is not acceptable by client, because there are huge amount files to migrate. Why does azcopy migrate such VHD files in different way contrate to other type files? Other type files will not be created at the beginning of copy, but creating them at the last of copy when copy is completed, however VHD files will be created at the begin of copy, is it right? Could you help me about this?

[ashruti-msft]

@momodedev can you share your debug logs for the run mentioned above?